Failure Modes in Multi-Agent LLM Systems
A chaos-engineering study of how multi-agent coding pipelines break.

- experiment runs across 6 coding tasks
- 1,480
- experiment runs across 6 coding tasks
- conversation traces automatically annotated
- 1,200
- conversation traces automatically annotated
- pass rate under a false assumption (p = 0.049)
- −5.0 pp
- pass rate under a false assumption (p = 0.049)
Overview
Seminar paper at the University of Helsinki. I injected controlled faults into ChatDev, a multi-agent coding framework, to measure how failure patterns from the MAST taxonomy affect the quality of the generated code. Everything ran locally with qwen2.5-coder:14b through Ollama.
Question
Multi-agent LLM systems fail in ways that a single model does not: agents withhold information, act on wrong assumptions or never stop. The MAST taxonomy names these failure modes. This study asks how much each one actually hurts the result.
Method
- Five conditions: a baseline, three kinds of information withholding, and a false assumption injected into the prompt.
- Each generated program is scored against HumanEval unit tests.
- Every trace is labelled automatically to check that each injection triggered the failure it was meant to trigger, and no other.
Findings
- False context is more harmful than missing context. Withholding information had almost no effect, because downstream agents could recover it from other channels.
- The false assumption lowered the pass rate in an unexpected way. Agents ignored the false instruction itself, but it changed their high-level plan. The resulting multi-file designs failed at assembly: NameErrors rose from 22% to 34% of failures.
- Removing the loop-exit signal made the pipeline run 3.6× longer, with no fallback.
- Seeing a failure pattern in the conversation does not mean it caused harm. What matters is whether the missing content can be recovered.
Screens

Related projects
FinTax
Retrieval-first research on Finnish tax law: find the right provision before generating anything.
RAG Sales Analysis
Ask business questions about a sales dataset in plain English, fully offline.