Multi-Agent Coordination

You keep adding context to the session and the responses keep getting worse. The instinct is to write a better prompt. The real problem is that the task has outgrown one agent.

A single agent with a well-designed CLAUDE.md, the right MCP connections, and a compaction strategy will take you further than most teams expect. Parts 1 through 4 of this series built that foundation: better context hygiene, better tooling, better memory management. After all of that is in place, a different kind of ceiling appears.

This second ceiling is structural. The task is not simply too long for one context window — it is the wrong shape for one agent: too parallel to run sequentially, too specialised to handle with one general-purpose context, or too failure-sensitive to have no boundaries between phases. Multiple agents are the answer. The new problem is coordination: how do you give independent agents a shared goal when they have no shared conversational context?

The Single-Agent Ceiling

The wall you hit is not one ceiling but three, and they are worth distinguishing because each one points to a different solution.

Parallelism

A single agent works sequentially. Every subtask queues behind the one before it. If you need to research three topics, analyse four codebases, or validate a set of independent changes, a single agent processes them one by one, spending its context window on each in turn.

This is fine until the subtasks are genuinely independent and the sequential constraint is the bottleneck. At that point, the solution is not a better agent; it is more agents running concurrently, each handling one branch of the work.

Specialisation

A generalist agent optimised to handle everything is, in practice, optimised for nothing in particular. Anthropic’s documentation on building effective agents makes this explicit: routing inputs to specialised handlers exists precisely because “optimising for one kind of input can hurt performance on other inputs.” An agent that simultaneously holds the role of requirements analyst, implementer, and reviewer brings conflicting pressures into one context. When the same context window contains the implementation decisions and the review task, the review is anchored to those decisions. Independent context is not just role separation; it reduces correlated reasoning between phases.

Specialisation is the answer here: separate agents with separate system prompts, each scoped tightly to one kind of work.

Failure isolation

When a single agent handles everything, there is no boundary around a failure. If something goes wrong thirty minutes into a long session, you cannot tell which phase of the reasoning produced the error. You cannot retry just the failing step. You cannot discard a bad result without discarding the surrounding work.

Multiple agents create natural failure boundaries. A worker that produces a bad result can be re-run in isolation. The orchestrator that spawned it still has its own context intact. Diagnosing the failure is easier because the reasoning is already separated.

Three Coordination Patterns

Not all multi-agent designs are the same. Three patterns cover the majority of cases, and each one fits a different shape of work.

Orchestrator and workers

One agent plans and delegates; worker agents execute. The orchestrator decides at runtime what subtasks to create, constructs a task description for each worker, spawns the worker, and synthesises the results. Workers are disposable: one per task, discarded when done.

This is the most flexible pattern. The orchestrator does not need to know in advance what the subtasks are; it reads the input and decides. Anthropic’s cookbook distinguishes this from simple parallelisation precisely on this point: the key difference is flexibility — subtasks are not pre-defined, but determined by the orchestrator based on the specific input.

The tradeoff is overhead. The orchestrator’s planning step costs a full API call. Worker results flow back through the orchestrator’s context, which grows with each completed subtask. For long-running workflows with many workers, the orchestrator’s context can become a bottleneck in its own right.

Best for: tasks where the subtasks cannot be predicted in advance: research across an unknown number of sources, code generation where the file list depends on what the analyser finds, validation across a variable set of candidates.

Pipeline

Phases in a fixed order, each agent producing a structured output that the next consumes. The phases are known before the workflow starts. There is no dynamic decomposition step.

The pipeline pattern has lower overhead than orchestrator/worker because there is no orchestrator making runtime decisions; each agent runs once, writes its output, and hands off. The coordination layer is a state file, not a planning agent.

Best for: workflows where the shape of work is stable. The next post in this series applies this exact pattern to a software development pipeline: Requirement, Plan, Dev, Test, Review, in that order, every time.

Peer coordination

Multiple agents with equivalent authority, each investigating a different direction in parallel, coordinating through a shared state file rather than a hierarchy. No orchestrator; no fixed phase order. Each peer reads the coordination manifest, claims a task, does the work, and writes its result back.

This is the hardest pattern to reason about because there is no central planner maintaining a coherent view of the whole. Agents can conflict, duplicate work, or reach inconsistent conclusions. Use it sparingly, and only when the tasks are genuinely independent and the results are aggregated rather than sequenced.

Best for: parallel research where breadth matters more than order: investigating multiple approaches simultaneously, exploring a problem space from different angles.

A note on cost

Before picking between these patterns, account for what multiple agents cost. Each worker receives a copy of the context it needs, which may duplicate content already in the orchestrator’s window. Anthropic’s own multi-agent research system used roughly 15× the tokens of a comparable single-agent interaction in those research workloads — a useful calibration, not a universal estimate. The pipeline pattern has the lowest duplication: each agent sees only its direct inputs. Orchestrator/worker has the highest potential cost: if workers return results inline, the orchestrator accumulates them as they complete. Returning file paths instead of content (recommended above) mitigates this significantly, but does not eliminate the orchestrator’s own planning and dispatch overhead. Use multiple agents when the value of isolation or parallelism justifies the cost, not by default.

Shared State Without Shared Conversational Context

The hardest part of multi-agent design is not spawning agents. It is giving them a shared goal when they have no shared conversational context. Each agent starts with an empty conversation history. It does not see what other agents discussed. It can only know what it is explicitly provided: through a file, a task description, or a structured handoff.

This is not a limitation to work around. It is the design constraint you build from. Files are shared: they persist across sessions and are readable by any agent. What is not shared is conversational context. Every piece of information an agent needs from a prior agent must be in a file it can read, not in a session it cannot access.

Structured output files as the coordination layer

The primary handoff mechanism is a file. Each agent reads the previous agent’s output file; it does not receive a summary of the previous agent’s reasoning. The file is the contract between phases.

This is Anthropic’s pattern from their own multi-agent research system: “Subagents call tools to store their work in external systems, then pass lightweight references back to the coordinator.” The coordinator passes the file path, not the file contents. This keeps each agent’s context lean; inlining large outputs into the orchestrator’s context consumes its window with content it does not need for its own reasoning.

Files survive compaction completely. Part 4 of this series covered compaction in detail. The key point here is that files live outside the message array: they are not subject to the summariser’s choices, they do not degrade across compaction events, and they are readable by any agent in any session.

The coordination manifest

For workflows involving multiple agents or multiple runs, a COORDINATION.md at the root of the working directory tracks which phases are complete, which are blocked, and where to find each output artefact. Any agent can read it; only the agent responsible for a given phase writes to it.

# COORDINATION.md

## Status
Current phase: dev
Previous phases: requirement (complete), plan (complete)

## Artefacts
- SPEC.md: complete — /work/SPEC.md
- PLAN.md: complete — /work/PLAN.md
- DEVIATIONS.md: pending

## Blocking
None

The manifest is not a prose description of what happened. It is a structured state file the next agent reads to understand where it is picking up. Every field has a purpose; nothing is narrative.

Avoiding conflicting writes in peer coordination

In sequential pipelines, only one agent writes to the coordination manifest at a time. In peer coordination, multiple agents might attempt to update it simultaneously. A lock file sounds like a solution, but it has a race: two agents can both check for the lock’s absence and find it absent before either creates it. A lock file is only safe if acquisition is atomic.

The simpler design is to avoid concurrent writes to a shared file entirely. Each peer writes only to its own result file. A coordinator — the orchestrator at the next phase, or a dedicated merge step — reads all peer outputs and combines them. If you genuinely need mutual exclusion, use an operation that is atomic by specification (mkdir on most systems, or open with O_CREAT|O_EXCL) rather than a check-then-create sequence.

This is not needed for sequential pipelines. It applies only when peer agents run in parallel with a shared output target.

The API Mechanics

The patterns above are architecture. Here is how they actually work in the Claude API.

Workers as model invocations

There is no native “subagent” primitive in the Messages API. A “worker” is a separate call to messages.create. The Messages API is stateless: each call is independent, there is no session that persists between calls, and there is no automatic compaction. Conversation state and context accumulation are managed by the application. State passes between agents through files or, for small outputs, through the orchestrator’s context window.

A minimal worker pattern in Python:

import anthropic

client = anthropic.Anthropic()

def run_worker(task_description: str, system_prompt: str) -> str:
    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=4096,
        system=system_prompt,
        messages=[{"role": "user", "content": task_description}]
    )
    return response.content[0].text

# Orchestrator calls workers and receives their outputs
worker_result = run_worker(
    task_description="Analyse the attached spec and produce PLAN.md.",
    system_prompt="You are a software architect. You receive a specification and produce an implementation plan..."
)

run_worker as written is a single stateless invocation: one request, one response. If a worker needs to reason over multiple turns or use tools, you build a conversation loop: accumulate a messages array, call messages.create, append the assistant response, call again. A basic messages.create call does not maintain conversational state or compact history on its own, but applications can opt into API-level compaction when implementing multi-turn loops. Claude Code and managed agents handle context management within their respective runtimes. The orchestrator sees only what the worker returns; the reasoning that produced it is not retained.

Claude Code subagents

In Claude Code, the Agent tool spawns subagent sessions. Each subagent:

• Gets an independent context window (up to the model’s context limit)
• Has independent compaction events
• Returns a result to the parent agent when the subagent task completes

CLAUDE.md loads into custom subagents automatically at session start — re-read fresh, not inherited from the parent. The built-in Explore and Plan agents are exceptions; they skip CLAUDE.md by default. For any custom subagent you define, conventions in CLAUDE.md are present without any extra handling from the orchestrator.

Claude Code also ships an experimental agent teams mode (enable with CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1): multiple independent Claude Code sessions coordinating via a shared task list and direct inter-agent messaging, with one session acting as team lead. This is peer-style coordination with lead-managed orchestration rather than hand-rolled file conventions — the team lead adds structure to what would otherwise be a fully decentralised model. It is experimental and not recommended for production workflows yet.

System prompts as job descriptions

In the Messages API, specialisation is through system prompts. Each agent type, planner, worker, or reviewer, gets a distinct system prompt scoped tightly to its role. The system prompt is the job description: what the agent does, what format its output takes, and what it explicitly cannot do.

The orchestrator constructs the system prompt and the initial message for each worker at runtime. This is the mechanism that makes the orchestrator/worker pattern flexible: the same orchestrator can spawn a research worker, a code worker, and a review worker by passing different system prompts to otherwise identical API calls.

Lightweight references, not inline content

A worker that produces a substantial output should return a reference to where the output lives, not inline the contents into the orchestrator’s context. This applies to research summaries, code diffs, and test reports: anything large enough to crowd the orchestrator’s window with content it does not need for its own reasoning.

This requires the worker to have a mechanism for actually writing the file. The stateless run_worker above does not: messages.create returns text, and without tool-use or a separate file-writing step, prompting the worker to “write to /work/research.md” produces text that describes the action rather than performing it. In practice, file-backed handoffs work in one of two ways: the worker has a write-file tool available in a tool-use loop, or the application code receives the worker’s structured output and writes it to disk before passing the file path downstream. The latter is the simpler starting point.

Compaction Across Agents

In multi-turn agent sessions that use compaction, each agent’s context is managed independently. This is the most important thing to understand about memory in a multi-agent system.

When the orchestrator’s session compacts, the compaction summary captures what the orchestrator knows: the tasks it delegated, the file paths it received back, the state of the coordination manifest. It does not capture any worker’s reasoning, because the orchestrator never had that reasoning in its context in the first place.

When a worker’s session compacts, the worker’s compaction summary captures its own in-progress state. The orchestrator sees none of this; it only ever sees the worker’s final output.

What this means for design

Do not rely on the orchestrator “remembering” how a worker arrived at its result. The orchestrator knows the result; the path to the result is permanently gone once the worker session ends. If the reasoning matters and the orchestrator might need to verify, revisit, or explain it, that reasoning must be written to a file before the worker session closes.

Worker output files must be self-contained. The next agent to read a worker’s output file cannot ask “what was the worker thinking when it wrote this?” It can only read the file. Design the file accordingly: include not just the conclusions but enough context to understand them without the session.

If a worker compacts mid-task and continues incorrectly, the orchestrator sees only the final output. There is no session-level signal that compaction fired and the worker’s reasoning diverged. Regardless of what execution logs reveal, output validation is the final safeguard: check that the worker’s output file meets the expected format and passes the expected checks before accepting it.

The pause_after_compaction hook covered in Part 4 is available within each agent’s own session. It lets you intercept compaction within a session and augment the summary before continuing. It does not give the orchestrator visibility into worker compaction; each session’s compaction is private to that session.

The Trust Model

When the orchestrator receives a worker’s result, it faces a choice: accept it or verify it. Three stances, each appropriate for different tasks:

Trust

Take the worker output as correct and continue. No additional checks. Fast, minimal overhead. The right choice when the task is narrow and well-scoped, the output format is validated (the file exists, conforms to the expected schema, is non-empty), and the consequence of a wrong result is inexpensive and reversible.

Example: a research worker that summarises a single article. If it is wrong, re-running is cheap: the task is short, the output is self-contained, and a bad summary propagating one hop costs far less than mis-implementing a feature or committing broken code.

Spot-check

The orchestrator reads a specific field or section of the worker output and checks one targeted condition. Not a full review: a sanity check. Does the plan file contain the expected sections? Does the code diff touch the expected files? Does the spec list at least one acceptance criterion?

A spot-check costs one additional API call (the orchestrator reading and evaluating the file) but catches the most common failure modes: an empty output, a structurally malformed file, a worker that misunderstood the task and produced something completely off-target.

Verify

Pass the worker’s output to a dedicated review agent that evaluates it against explicit criteria before the orchestrator accepts it. The next post shows this in practice: a rubric-based reviewer that checks every acceptance criterion before marking the phase complete.

The right stance depends on two factors: the reversibility of the downstream work, and the cost of a silent failure. Code that will be committed to a repository warrants verification. A research summary that informs an exploratory query warrants trust. Everything in between is a spot-check judgment.

Failure Modes Unique to Multi-Agent

Silent worker failure

The worker completes its task and returns a result that looks valid: the file exists, it is non-empty, it has the right structure. But the content is wrong. The worker misunderstood the task, compacted mid-session and lost critical context, or generated plausible-looking output that does not actually satisfy the requirement.

The orchestrator has no automatic signal. Silent failure is the hardest failure mode in multi-agent systems precisely because nothing breaks visibly; the pipeline continues with bad data.

Prevention: structured output schemas with a required spot-check pass before the orchestrator accepts any worker result. Define what a valid output looks like and check it explicitly.

The telephone game

Each handoff is a summarisation. The orchestrator summarises the task for the worker; the worker summarises its findings in an output file; the next agent reads that file and summarises what it found. Across three or four hops, precision erodes. Specific file names become “the relevant files.” Exact line numbers become “in the authentication module.” Concrete decisions become “the team agreed on an approach.”

Prevention: pass the original artefact files, not summaries of summaries. If the spec is in SPEC.md, every downstream agent should read SPEC.md directly. The orchestrator should not paraphrase it. File paths are cheap to pass; precision is expensive to recover once it is lost.

Circular delegation

The orchestrator asks a worker to do X. The worker, unable to complete X alone, spawns a sub-worker and asks it to do X. The sub-worker spawns another. The task bounces between agents without making progress, each one passing the problem on rather than solving it.

Prevention: the system prompt for every worker should explicitly state that it does not spawn further subagents. If a worker needs to escalate, it should return an explicit “blocked” signal to the orchestrator, not delegate sideways.

Conflicting writes

Two peer agents read the coordination manifest, both conclude they need to update the same field, and write conflicting values. One overwrites the other. The manifest now reflects neither agent’s actual state.

Prevention: give each peer its own output file. A coordinator at the next phase reads all peer outputs and merges them. Where shared writes to a manifest are unavoidable, use an operation that is atomic by specification rather than a check-then-create sequence. If the coordination logic is getting complex, that is usually a sign the problem fits a pipeline better than a peer model.

What to Do Next

The decision comes down to two questions: does the work have a predictable shape, and can you define what a valid output looks like before you run it?

If both answers are yes, start with a pipeline. It is simpler than orchestrator/worker, easier to diagnose when something fails, and sufficient for most structured workflows. Start with two phases before scaling to five. A two-agent system (analyst + implementer, or planner + worker) is enough to expose the real failure modes. Adding more agents before you understand those failure modes makes diagnosis harder, not easier.

If the subtasks cannot be predicted before the work begins, use orchestrator/worker. Keep the orchestrator’s context lean: workers write to files and return file paths, not file contents. Add a spot-check pass after each worker before the orchestrator continues.

Either way, silent worker failure is the failure to design for first, not a coordination problem. A worker returns a file that exists and is non-empty, but does not satisfy the requirement. Define what a valid output looks like before you run the pipeline, and check it explicitly. Validation is easier to build early than to retrofit after a silent failure has propagated through three phases.

Build the review phase last, once the earlier phases are stable. A rubric that tries to cover everything before you know what can go wrong produces more noise than signal. Start with two or three blocking categories. Add rules when incidents reveal gaps.

For Claude Code: the Agent tool gives you subagents with minimal setup. Your CLAUDE.md conventions reach custom subagents automatically because each re-reads it at session start. (The built-in Explore and Plan agents are exceptions; they skip it by default.) Design effort goes into the coordination files: what each agent writes, what the next reads, and what COORDINATION.md tracks across the workflow.

The principles in this post are what make a multi-agent system designable. Three ceilings identify when you need multiple agents. Three patterns give you a starting point for the structure. The coordination layer of files, manifests, and lightweight references gives independent agents a shared goal without shared conversational context. The trust model tells you when to verify and when to proceed.

The next post applies these principles to one specific domain: a software development pipeline where each phase is handled by a separate agent, with a deterministic review at the end. The architecture follows directly from what is here; the details are in the implementation.