Batching and Async for Real Throughput

You’ve used .batch() once, briefly, in Module 8, and .ainvoke() a couple of times since Module 4 — always just enough to prove the method existed. This module is where we actually measure the difference these methods make, and, just as importantly, understand when each one is genuinely the right tool, since they solve two related but honestly different problems.

The problem, made concrete with an actual clock

Let’s stop describing the problem abstractly and actually measure it. Imagine you need short descriptions for five different topics.

import time
from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4o-mini")
topics = ["RAG", "embeddings", "transformers", "fine-tuning", "vector databases"]

start = time.time()

results = []
for topic in topics:
    response = model.invoke(f"Describe {topic} in five words.")
    results.append(response.content)

elapsed = time.time() - start
print(f"Sequential: {elapsed:.2f} seconds for {len(topics)} calls")
for r in results:
    print("-", r)

Run this, and take note of the actual number printed. Each call in this loop waits, in full, for the previous one to completely finish before even starting — five separate round trips to the provider’s servers, one strictly after another, even though none of these five requests actually depend on each other’s results at all.

Example 1: .batch(), and the actual, measured difference

import time
from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4o-mini")
topics = ["RAG", "embeddings", "transformers", "fine-tuning", "vector databases"]

start = time.time()

prompts = [f"Describe {topic} in five words." for topic in topics]
results = model.batch(prompts)

elapsed = time.time() - start
print(f"Batched: {elapsed:.2f} seconds for {len(topics)} calls")
for r in results:
    print("-", r.content)

Run this and compare the two printed times directly. You should see the batched version complete noticeably faster than the sequential loop, because .batch() doesn’t send these five requests one after another — it sends them concurrently, and waits for all of them to come back together. The actual work is identical; the shape of how it’s requested is what changed, and that shape genuinely matters for real, measured speed.

Understanding what async actually means, properly this time

Back in Module 4, we promised a proper explanation of async once it actually mattered — and now it does. Let’s be precise about a distinction that’s easy to blur: .batch() and true async concurrency solve two genuinely different problems, even though both involve “doing more than one thing at once.”

.batch() is for exactly the situation you just measured: many similar calls to the same Runnable, where you’re willing to wait for the whole group to finish together. It’s a synchronous method — your program still pauses and waits — but internally, it’s smart about running those similar calls concurrently rather than one by one.

Async, using await and asyncio, is for a different, broader situation: your program needs to stay responsive and keep doing other work — handling other users, running other unrelated code — while waiting on a slow operation, rather than freezing entirely until it’s done. This matters enormously the moment your LangChain code lives inside something like a real web server, handling many different users’ requests at the same time, where one user’s slow AI call should never freeze the entire server for everyone else.

Example 2: running genuinely different tasks concurrently with asyncio.gather

Let’s build something .batch() genuinely can’t do on its own: run two different chains, doing two different jobs, at the same time.

import asyncio
from langchain.chat_models import init_chat_model
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

model = init_chat_model("openai:gpt-4o-mini")
parser = StrOutputParser()

summary_chain = ChatPromptTemplate.from_template("Summarize in one sentence: {text}") | model | parser
translate_chain = ChatPromptTemplate.from_template("Translate to French: {text}") | model | parser

text = "LangChain gives developers a shared set of building blocks for LLM applications."

async def main():
    summary_task = summary_chain.ainvoke({"text": text})
    translation_task = translate_chain.ainvoke({"text": text})

    summary, translation = await asyncio.gather(summary_task, translation_task)

    print("Summary:", summary)
    print("Translation:", translation)

asyncio.run(main())

Notice summary_chain and translate_chain are two genuinely different pipelines, doing two genuinely different jobs — this isn’t the “many similar inputs through one chain” shape .batch() is built for. asyncio.gather starts both .ainvoke() calls, lets them run concurrently, and waits for both to finish before continuing. This is the real, general-purpose tool for “run several different async operations at once,” and it’s what lets a real application do genuinely varied work concurrently, not just repeat one operation across a list of inputs.

Example 3: .abatch() — the async version of batch

If you’re already working inside async code — say, a web application’s request handler — and you need to process a whole list of similar inputs without blocking that handler while you wait, .abatch() is the tool built for exactly that combination.

import asyncio
from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4o-mini")

async def main():
    topics = ["RAG", "embeddings", "transformers"]
    prompts = [f"Describe {topic} in five words." for topic in topics]

    results = await model.abatch(prompts)
    for r in results:
        print(r.content)

asyncio.run(main())

This is genuinely the combination of both ideas you just learned: the “many similar inputs” efficiency of .batch(), running inside an async function so it never blocks whatever else your application might be doing at the same moment.

A practical rule for choosing between these

Given everything above, here’s the actual, practical decision to make in real code:

Use .batch() when you have many similar inputs to run through the same chain, and your code is otherwise ordinary, synchronous Python. Use asyncio with .ainvoke() and asyncio.gather() when you need to run genuinely different operations concurrently, or when your application — a web server being the most common real example — is already built on async from the ground up. Use .abatch() specifically when both of those situations apply at once: many similar inputs, inside an already-async application.

Common mistakes worth avoiding

Reaching for asyncio.gather on a list of similar calls, when .batch() would be simpler. If every task you’re running concurrently is really “the same chain, different input,” .batch() already handles this cleanly, with less code and less risk of a small mistake in manually managed async logic. Save asyncio.gather for genuinely different, varied operations.

Mixing .invoke() inside an async def function without await. It’s an easy typo to make: calling model.invoke(...) instead of await model.ainvoke(...) inside an async function. This won’t necessarily raise an obvious error — it will simply block the entire event loop while it runs, quietly defeating the entire purpose of writing async code in the first place.

Assuming .batch() guarantees results come back in a different order than requested. They don’t — .batch() always returns results in the exact same order as your input list, even though the underlying calls may have completed in a different order internally. It’s safe to zip your original inputs back up with .batch()’s results by position, exactly as you did in Module 8, Example 3.

What you should take away from this module

• A sequential loop of .invoke() calls genuinely wastes real time, waiting for each call to finish before starting the next — and you’ve now measured this difference directly, not just been told about it.
• .batch() is the right tool for many similar inputs through the same Runnable, run concurrently under the hood.
• asyncio.gather with .ainvoke() is the right tool for running genuinely different operations concurrently, and for keeping an already-async application, like a web server, responsive.
• .abatch() combines both: batch-style efficiency, inside an async context.
• Results from .batch() always come back in the same order as your original input list.

Where this goes next

The next module turns to Tools — one of the most important, and most code-heavy, topics in this entire course. You’ll go from an ordinary Python function to something a model can genuinely understand and ask to use, building the exact foundation every agent you construct for the rest of this course will depend on.