Parallel Execution: Fan-Out, Fan-In, and Real Trade-offs

Recall Module 10’s Send mechanism, and Module 11’s reducers fixing the collision it exposed. Both modules taught the real mechanics correctly — and both, deliberately, stayed in the happy path. This module returns to that exact same pattern with a genuinely different lens: what happens when parallel execution meets real, production conditions — partial failure, rate limits, and real, multiplied cost.

The complete pipeline, assembled from what you already know

flowchart LR
    planner --> A[research: LLMs]
    planner --> B[research: RAG]
    planner --> C[research: Agents]
    A --> synthesis
    B --> synthesis
    C --> synthesis

from typing import TypedDict, Annotated
import operator
from langgraph.graph import StateGraph, START, END
from langgraph.types import Send
from langchain.chat_models import init_chat_model

class State(TypedDict):
    topics: list[str]
    findings: Annotated[list[str], operator.add]
    summary: str

class BranchState(TypedDict):
    topic: str
    findings: list[str]

model = init_chat_model("openai:gpt-4o-mini")

def plan(state: State) -> dict:
    return {"topics": ["LLMs", "RAG", "Agents"]}

def research_one(state: BranchState) -> dict:
    response = model.invoke(f"Give one key fact about {state['topic']}.")
    return {"findings": [response.content]}

def fan_out(state: State) -> list[Send]:
    return [Send("research_one", {"topic": t, "findings": []}) for t in state["topics"]]

def synthesize(state: State) -> dict:
    return {"summary": " | ".join(state["findings"])}

builder = StateGraph(State)
builder.add_node("plan", plan)
builder.add_node("research_one", research_one)
builder.add_node("synthesize", synthesize)
builder.add_edge(START, "plan")
builder.add_conditional_edges("plan", fan_out, ["research_one"])
builder.add_edge("research_one", "synthesize")
builder.add_edge("synthesize", END)

graph = builder.compile()
result = graph.invoke({"topics": [], "findings": [], "summary": ""})
print(result["summary"])

This is genuinely nothing new mechanically — Module 10’s fan_out, Module 11’s operator.add reducer, and a real, final synthesis step, assembled into one, complete, working pipeline. What’s new starts now.

The honest question this module actually exists to answer: what if one branch fails?

def research_one(state: BranchState) -> dict:
    if state["topic"] == "RAG":
        raise RuntimeError("Simulated: the research API is temporarily down.")
    response = model.invoke(f"Give one key fact about {state['topic']}.")
    return {"findings": [response.content]}

Run the full graph with this version, and the entire graph fails — one branch’s real, genuine failure takes down the whole fan-out, discarding the two branches that actually succeeded. This is a real, honest, and often unwelcome default. A genuinely resilient version needs to decide, deliberately, what “partial success” should mean for your specific application.

def research_one(state: BranchState) -> dict:
    try:
        response = model.invoke(f"Give one key fact about {state['topic']}.")
        return {"findings": [response.content]}
    except Exception as e:
        return {"findings": [f"[Failed to research {state['topic']}: {e}]"]}

Recall Module 12’s LangChain course lesson about tools handling their own likely failures gracefully — the exact same discipline applies here. Catching the failure inside the branch, rather than letting it propagate and crash the whole fan-out, means synthesis still runs, with two genuine findings and one honest, visible failure marker, rather than nothing at all.

The real, honest rate-limit problem

Firing Send across many branches genuinely means many real, near-simultaneous calls to your model provider. Real providers enforce real rate limits — a fan-out across fifty topics can genuinely trigger throttling that a single, sequential call would never hit.

model = init_chat_model("openai:gpt-4o-mini").with_retry(stop_after_attempt=3, wait_exponential_jitter=True)

Recall .with_retry() directly from your LangChain course’s own resilience module — it applies here exactly as it did there, giving each individual parallel branch a real, deliberate chance to recover from a transient rate-limit rejection, rather than treating every throttled request as a hard, immediate failure.

The real, honest cost multiplication

It’s worth being direct with real numbers, since parallelism can feel “free” simply because it’s fast. Recall Module 18 (LangChain course)‘s cost-per-token concerns — if one research branch costs roughly 200 tokens, a fan-out across ten topics costs roughly 2,000 tokens for that step alone, not 200. Parallelism reduces real, wall-clock latency; it does not reduce real, total cost — every branch is still a genuine, separately billed model call.

print(f"Estimated tokens: {len(state['topics'])} branches × ~200 tokens = {len(state['topics']) * 200} tokens")

A genuinely production-ready fan-out deliberately bounds how many parallel branches it allows — recall Module 13’s own retry_count discipline — rather than letting an unbounded input (like a user-supplied list of a hundred topics) silently produce a hundred real, billed parallel calls.

Why this connects to a genuine, documented production reality

This isn’t a hypothetical concern. Real, published research on production compound-AI systems — architectures composing multiple model calls, exactly like this module’s fan-out — has documented genuine “multi-model fan-out overhead” and “cascading coldstart propagation” as real, measured challenges distinct from what a single-model system ever faces. The same research reported that a properly engineered serving architecture, built specifically to handle these fan-out-specific challenges, achieved a real, measured 50%+ reduction in tail latency and 30-40% cost savings compared to a naive, unoptimized deployment — real, concrete proof that the failure and cost concerns this module raises are genuine, production-grade engineering problems, not academic caution.

Common mistakes worth avoiding

Letting one branch’s failure silently destroy an entire fan-out’s results. Recall this module’s own core example — without a deliberate try/except inside each branch, one real failure discards every other branch’s genuinely successful, real work.

Firing an unbounded number of parallel branches based on unvalidated input. A user-controlled topic list, passed directly into Send without a real, deliberate cap, can turn a reasonable feature into an accidental, expensive flood of simultaneous API calls — recall Module 13’s own retry-count discipline, applied here to branch count instead.

Assuming parallelism reduces total cost, not just wall-clock time. Recall this module’s own honest math — ten parallel branches cost genuinely more in total tokens than one branch, even though they finish in roughly the same real time as one branch would alone.

What you should take away from this module

• The full planner-fan-out-synthesis pipeline is genuinely just Module 10’s Send and Module 11’s reducers, assembled together — no new mechanism.
• Without deliberate try/except inside each branch, one real failure destroys an entire fan-out’s results, including genuinely successful ones.
• .with_retry(), from your LangChain course, applies directly to parallel branches facing real, transient rate-limit rejection.
• Parallelism reduces real latency, not real cost — every branch remains a fully, separately billed operation, worth bounding deliberately.

Where this goes next

The next module covers Error Handling, Retries, and Fallbacks at the level of the entire graph — building on this module’s own partial-failure lessons, now applied to node exceptions, tool failures, and full workflow-level fallback strategies.