The Setup: Five Production Deployments, Zero Assumptions
Between Q3 2025 and Q1 2026, our engineering team shipped multi-agent orchestration systems across five production Kubernetes environments. Industries spanned: regulated healthcare 2 deployments, financial services compliance 1, logistics route optimization 1, and legal contract intelligence 1.
Every deployment ran on the same base stack: Kubernetes 1.29+, LangGraph for agent graph management, custom async task queues, and private LLM inference Mistral 7B–70B depending on latency requirements. None used shared cloud LLM APIs for sensitive workloads.
The headline metric: 4.2× throughput improvement over single-agent baseline at equivalent compute cost. But the path to that number destroyed three architectural assumptions we started with.
---
What We Started With And Why It Broke
Assumption 1: Stateless Agents Scale Linearly
We expected horizontally-scaled stateless agents to give us near-linear throughput gains. It didn't happen.
In Deployment 1 healthcare prior auth, we scaled from 4 → 16 agent replicas and got only 2.3× throughput — not 4×. The bottleneck wasn't CPU or memory. It was shared context retrieval: every agent was hammering the same vector store for patient history chunks, creating read contention that choked the entire pipeline.
Fix: Per-agent context caching with a 45-second TTL. Throughput jumped to 3.7× at 16 replicas.
Assumption 2: LangGraph Handles Fault Tolerance
LangGraph's graph execution model is excellent for defining agent workflows. It is not a fault-tolerance layer.
In Deployment 3 financial compliance, a downstream document parsing agent started throwing intermittent timeouts. LangGraph's default behavior: propagate the exception up the graph and fail the entire workflow. With 400+ concurrent document reviews in flight, this caused cascading failures that took down 23% of active workflows within 6 minutes.
Fix: Explicit retry envelopes at every agent boundary, dead-letter queues for failed tasks, and circuit breakers at the LangGraph node level using a thin wrapper:
python def resilientnodefunc, maxretries=3, backoffbase=2.0: async def wrapperstate: AgentState - AgentState: for attempt in rangemaxretries: try: return await funcstate except TransientError as e: if attempt == maxretries - 1: return state.markfailedstre await asyncio.sleepbackoffbase attempt return wrapper
Assumption 3: A Single Orchestrator Agent Is Fine
For workflows under 8 agent steps, a single orchestrator works. Beyond that, orchestrator becomes a bottleneck and single point of failure.
In Deployment 5 legal contract review, our orchestrator agent was managing 14 specialized sub-agents: clause extraction, risk classification, precedent lookup, redline generation, and more. At peak load, the orchestrator's context window was hitting 80K tokens of accumulated state, ca