Skip to content
All projects
ResearchResearch paper · University of Helsinki2026Completed

Failure Modes in Multi-Agent LLM Systems

A chaos-engineering study of how multi-agent coding pipelines break.

Code is private. Happy to walk through it on request.
Screenshot of Failure Modes in Multi-Agent LLM Systems
experiment runs across 6 coding tasks
1,480
experiment runs across 6 coding tasks
conversation traces automatically annotated
1,200
conversation traces automatically annotated
pass rate under a false assumption (p = 0.049)
−5.0 pp
pass rate under a false assumption (p = 0.049)

Overview

Seminar paper at the University of Helsinki. I injected controlled faults into ChatDev, a multi-agent coding framework, to measure how failure patterns from the MAST taxonomy affect the quality of the generated code. Everything ran locally with qwen2.5-coder:14b through Ollama.

Question

Multi-agent LLM systems fail in ways that a single model does not: agents withhold information, act on wrong assumptions or never stop. The MAST taxonomy names these failure modes. This study asks how much each one actually hurts the result.

Method

  • Five conditions: a baseline, three kinds of information withholding, and a false assumption injected into the prompt.
  • Each generated program is scored against HumanEval unit tests.
  • Every trace is labelled automatically to check that each injection triggered the failure it was meant to trigger, and no other.

Findings

  • False context is more harmful than missing context. Withholding information had almost no effect, because downstream agents could recover it from other channels.
  • The false assumption lowered the pass rate in an unexpected way. Agents ignored the false instruction itself, but it changed their high-level plan. The resulting multi-file designs failed at assembly: NameErrors rose from 22% to 34% of failures.
  • Removing the loop-exit signal made the pipeline run 3.6× longer, with no fallback.
  • Seeing a failure pattern in the conversation does not mean it caused harm. What matters is whether the missing content can be recovered.

Screens

Per-task pass rates: most effects are concentrated in specific tasks.
Per-task pass rates: most effects are concentrated in specific tasks.

Related projects

Next projectKushmaBazar