Batch Size

The Batch article closed by flagging exactly this gap: a batch is the group of examples processed together, but how many examples belong in that group is a real, consequential decision. That number is the batch size.

The simple definition

Batch size is the number of training examples included in each batch — a Hyperparameter, chosen by an engineer before training begins, that determines how many examples’ worth of gradient get averaged together before each weight update. A batch size of 32 means the model looks at 32 training examples, calculates one combined gradient across all of them, and updates its weights once. A batch size of 256 means that same update happens only after looking at 256 examples instead.

Counting batches and updates

Suppose the dataset contains 1,000 examples.

Batch size	Batches per epoch	Approximate updates per epoch
10	100	100
100	10	10
250	4	4
1,000	1	1

batches per epoch = dataset size / batch size

When the division is not exact, the final batch may contain fewer examples unless the training pipeline drops it.

The trade-off

Smaller batch
→ less memory per batch
→ more frequent, noisier gradient estimates

Larger batch
→ more memory per batch
→ fewer, smoother gradient estimates

Hardware utilization, model size, sequence length, optimizer, and learning rate all interact with batch size. Gradient accumulation can simulate a larger effective batch by combining gradients from several smaller batches before updating parameters.

One published InstructGPT fine-tuning example: batch size 8

For supervised fine-tuning of its 175B model, OpenAI’s InstructGPT paper reports a batch size of 8.

If a simplified training set contained 800 examples and every batch contained exactly 8 examples:

batches per epoch = 800 / 8 = 100

For 16 epochs:

approximate batch updates = 100 × 16 = 1,600

This simplified calculation assumes exactly 800 examples in every epoch and no incomplete or dropped batches.

Why this specific number is a genuine trade-off, not a technicality

Recall from the Batch article that using a smaller slice of data per update produces a noisier, less precise gradient estimate, while using the full dataset produces the most accurate possible gradient but is impractically slow. Batch size is the dial that sits between these two extremes, and moving it in either direction has real, opposite consequences:

• Smaller batch sizes produce noisier gradient estimates (since a small handful of examples represents the overall pattern less reliably), but allow far more frequent weight updates for the same amount of data processed, and require much less memory at any one time.
• Larger batch sizes produce smoother, more accurate gradient estimates (closer to what the “true” full-dataset gradient would say), but require more memory to hold that many examples’ calculations at once, and result in fewer, larger updates over the same amount of data.

flowchart LR
    A[Small batch size] --> B[Noisier gradients, more frequent updates, less memory]
    C[Large batch size] --> D[Smoother gradients, fewer updates, more memory]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a restaurant chef tasting a pot of soup to decide if it needs more salt. Tasting one small spoonful gives a quick read, but that one spoonful might not perfectly represent the whole pot if it hasn’t been stirred evenly. Tasting a much larger ladleful gives a more reliable read of the pot’s actual overall flavor, but takes longer to do each time, and you can only taste so much before you’ve eaten half the soup.

Where this breaks down: A chef’s judgment about taste is a rich, qualitative assessment. A batch’s gradient contribution is a precise mathematical average — there’s no “closer to the true flavor, but not quite” fuzziness, just a numerically more or less accurate estimate of the full-dataset gradient, calculated exactly the same mechanical way regardless of size.

The memory constraint that often decides this in practice

Beyond the statistical trade-off, there’s a very concrete, physical limitation that frequently ends up the deciding factor in real projects: batch size directly determines how much GPU memory a training run needs at any given moment, since every example in a batch has to be held in memory simultaneously while its gradient is calculated.

This connects directly to the parameter storage math from the Parameters article — a model with billions of parameters, combined with a large batch size, can require far more GPU memory than a single GPU has available, which is exactly why frontier labs run training across large clusters of GPUs or TPUs working together, as covered in the Training article, partly to accommodate batch sizes large enough to train efficiently at that scale.

Common batch size values, and why they tend to be round numbers

In practice, batch sizes are very often chosen from a small set of common values — 32, 64, 128, 256, and so on — powers of two. This isn’t arbitrary tradition; it reflects how GPU hardware processes data most efficiently, since GPUs are architecturally optimized to handle data in these specific power-of-two groupings. Choosing a batch size of, say, 100 instead of 128 wouldn’t break anything, but it can leave some hardware efficiency on the table — a small, practical detail that shapes real engineering decisions even though it has nothing to do with the statistical trade-offs discussed above.

A concrete example, layered

For a simple beginner example: training the one-weight house model on 1,000 houses with a batch size of 10 produces 100 weight updates per epoch, each based on a fairly noisy, small sample; the same dataset with a batch size of 500 produces only 2 updates per epoch, each based on a much more stable, representative sample, but far fewer total corrections across that epoch.

For a production example: OpenAI’s published GPT-3 training recipe used a final batch size of 256 sequences — but didn’t start there.

It used batch size warmup, beginning training with a much smaller batch size of just 16 and gradually increasing it up to the full 256 over the first 4 billion tokens processed, easing the model into larger, more stable batches rather than starting at full batch size from the very first step — a real, documented illustration of batch size itself being tuned dynamically over a training run, not just fixed once and left alone.

Check your understanding

If batch size doubles, do updates per epoch usually increase? No. They usually decrease.

Is the largest batch that fits memory always best? No. Training quality and optimization behavior also matter.

Common misconception

A frequent assumption: that a larger batch size is straightforwardly “better” since its gradient estimates are more accurate.

In practice, this isn’t reliably true — some research and practical experience suggests that the added noise from smaller batch sizes can actually help a model generalize better in certain situations, by preventing training from settling too comfortably and rigidly into a narrow solution that fits the training data closely but doesn’t generalize as well to new data (a subtle connection to the overfitting concerns raised throughout the Data Handling phase).

Batch size, like the learning rate covered in the previous article, is a genuine trade-off to be tuned for the specific situation — not a setting where bigger is automatically better.

Effective batch size and gradient accumulation

A GPU may fit only 8 examples at once. The program can calculate gradients for four groups of 8 before performing one optimizer update, creating an effective batch size of 32.

Effective batch size can also multiply across several GPUs. This helps hardware utilization, but a larger effective batch is not automatically more accurate; it changes gradient noise and may require a different learning rate.

Where this fits in what comes next

You now understand both halves of the batch concept: what a batch is, and how its size shapes training’s speed, memory needs, and gradient quality. The next article, Epoch, zooms out one more level — covering what it means to complete one full pass through every batch in the training set, and why training typically repeats that full pass many times over.

In one sentence

Batch size is the tunable number of examples per batch, trading off gradient accuracy, update frequency, and memory usage — and, like most hyperparameters covered in this phase, there’s no universally correct value, only one that’s well-matched to the specific model, hardware, and dataset at hand.

Related Terms

• Batch
• Hyperparameters
• Gradient Descent
• Learning Rate
• Epoch