Scaling LLMs | TechByteByByte

Before you continue: three tools for this module

Token: a piece of text processed by the model.
Parameter: a learned number controlling the model’s transformations.
Inference: using the trained model without updating its parameters.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Scaling LLMs solve inside a real language-model system?

Keep that central question about Scaling LLMs in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

model size + data + compute + training quality → capability and cost

1. What You Will Learn

Learning outcomes

Explain how model size, training data, and compute interact.
Interpret scaling laws as measured trends rather than guarantees.
Identify quality, cost, energy, data, and infrastructure limits.
Explain why better data and training methods can matter as much as raw size.

In one sentence

💡 Big picture

Scaling means increasing some combination of model size, training data, and computing power to build a more capable model.

2. Why This Module Exists

The problem this module solves

Bigger models often improve in predictable ways, but growth has real limits and costs.
Good data and training choices matter; simply adding parameters is not a magic recipe.

3. Intuition

training loss decreases predictably as you increase compute, model size, or data — but not linearly. Each additional 10x investment buys a progressively smaller improvement than the previous 10x. This is why massive compute investments in modern LLMs produce real but incremental gains, not unlimited runaway improvement.

4. Core Concept

Model size (N):        number of parameters (Module 12)
Dataset size (D):        number of training tokens (Module 8)
Compute (C):                total training FLOPs -- roughly
                          proportional to N x D (more parameters
                          AND more data both cost more compute)

Scaling laws:                empirically observed relationships
                          between loss and N, D, C -- loss
                          decreases as a POWER LAW as any of these
                          increase, with DIMINISHING RETURNS

5. How It Works — Step by Step

1. As COMPUTE increases, training LOSS decreases predictably --
   following an approximately power-law relationship (verified
   directly below)
2. This relationship shows DIMINISHING RETURNS: each additional
   10x compute produces a SMALLER loss improvement than the
   previous 10x
3. For a FIXED compute budget, there's a "compute-optimal"
   TRADE-OFF between model size and data size -- more parameters
   means fewer affordable training tokens for the same compute,
   and vice versa (verified directly below)
4. At SUFFICIENT scale, certain capabilities appear to emerge
   that smaller models (even trained proportionally) simply do
   not exhibit -- "emergent capabilities"

6. Mathematical Intuition

Read the mathematics as a story

model size + data + compute + training quality → capability and cost

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

A simplified, illustrative scaling law:

Loss(Compute) = a x Compute^(-alpha) + L_infinity

a, alpha: empirically fit constants (real scaling law papers derive these from extensive experiments).
L_infinity: an irreducible loss floor — the loss the relationship approaches but never quite reaches, no matter how much compute is added.
The -alpha exponent (a small positive number) is exactly what produces diminishing returns: as Compute grows, Compute^(-alpha) shrinks, but progressively more slowly.

The compute-optimal trade-off (illustrative, in the spirit of “Chinchilla-style” findings): Compute ≈ 6 × N × D — for a fixed compute budget, increasing model size N directly reduces how many training tokens D you can afford, and vice versa.

Analogy: The Bakery Sourdough Marathon & The Ingredient Budget Think of scaling parameters and training data in terms of budget constraints in a bakery:

The Kneading Marathon (Power-Law Loss): You are baking bread (training the model). If you knead for 1 hour (compute), the bread improves a lot. Kneading for 10 hours makes it slightly better, but kneading for 100 hours doesn’t make it 100x better — you hit hard diminishing returns.

**The $100 Oven-vs-Flour Budget (Compute-Optimal Allocation):** You have exactly \$ 100 to spend (fixed compute budget):

If you spend $95 on a massive, state-of-the-art commercial oven (large parameter size N), you only have $5 left to buy a tiny cup of flour (dataset size D). Your giant oven sits mostly empty and undertrained.

If you spend $95 on sacks of premium flour (D) but only have $5 left for a toy toaster oven (N), you cannot cook the bread properly.

The Chinchilla scaling law is the recipe for the perfect balance: spend roughly equal portions on oven capability and flour volume to get the best possible loaf of bread.

📊 Visual Chart: Compute-Optimal Sizing Trade-Off

Here is how dataset tokens and parameter limits trade off under a fixed compute budget:

graph TD
    classDef budget fill:#e74c3c,stroke:#333,stroke-width:1px,color:#fff;
    classDef option fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;

    FixedCompute["Fixed Training Compute Budget:<br>1e21 FLOPs (Constant C)"]:::budget

    FixedCompute --> Opt1["Option A: Small Model<br>1B params -> 166.7B tokens"]:::option
    FixedCompute --> Opt2["Option B: Balanced Model<br>5B params -> 33.3B tokens"]:::option
    FixedCompute --> Opt3["Option C: Large Model<br>10B params -> 16.7B tokens"]:::option
    FixedCompute --> Opt4["Option D: Over-sized Model<br>50B params -> 3.3B tokens (Undertrained)"]:::option

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

If a lab has a fixed compute budget, they face a genuine choice: train a larger model on fewer tokens, or a smaller model on more tokens.

Real scaling law research (which this module doesn’t derive from scratch, per this course’s depth calibration) found that many earlier large models were meaningfully undertrained relative to their size — a smaller model trained on proportionally more data would have used the same compute more effectively.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Scaling LLMs.
# Follow the inputs, transformations, and output in order.
def scaling_law_loss(compute, a=10.0, alpha=0.15, L_inf=1.5):
    return a * (compute ** -alpha) + L_inf

compute_budgets = [1e18, 1e19, 1e20, 1e21, 1e22, 1e23]
print("Compute (FLOPs) -> Predicted Loss:")
for c in compute_budgets:
    print(f"  {c:.0e} -> {scaling_law_loss(c):.4f}")

print("\nMarginal loss reduction per 10x compute increase:")
for i in range(1, len(compute_budgets)):
    prev_loss = scaling_law_loss(compute_budgets[i-1])
    curr_loss = scaling_law_loss(compute_budgets[i])
    print(f"  {compute_budgets[i-1]:.0e} -> {compute_budgets[i]:.0e}: improved by {prev_loss - curr_loss:.4f}")

# --- Compute-optimal allocation: model size vs. data size trade-off ---
def implied_tokens(compute, num_params):
    return compute / (6 * num_params)   # illustrative: compute ~ 6*N*D

fixed_compute = 1e21
param_options = [1e9, 5e9, 10e9, 50e9]
print(f"\nFixed compute budget: {fixed_compute:.0e} FLOPs")
for n_params in param_options:
    tokens = implied_tokens(fixed_compute, n_params)
    print(f"  {n_params/1e9:.0f}B params -> implied training tokens: {tokens/1e9:.1f}B tokens")

Expected Output:

Compute (FLOPs) -> Predicted Loss:
  1e+18 -> 1.5200
  1e+19 -> 1.5141
  1e+20 -> 1.5100
  1e+21 -> 1.5071
  1e+22 -> 1.5050
  1e+23 -> 1.5035

Marginal loss reduction per 10x compute increase:
  1e+18 -> 1e+19: improved by 0.0058
  1e+19 -> 1e+20: improved by 0.0041
  1e+20 -> 1e+21: improved by 0.0029
  1e+21 -> 1e+22: improved by 0.0021
  1e+22 -> 1e+23: improved by 0.0015

Fixed compute budget: 1e+21 FLOPs
  1B params -> implied training tokens: 166.7B tokens
  5B params -> implied training tokens: 33.3B tokens
  10B params -> implied training tokens: 16.7B tokens
  50B params -> implied training tokens: 3.3B tokens

9. How It Works

Loss genuinely, monotonically decreases as compute increases — but the marginal improvement shrinks with every step: 0.0058 → 0.0041 → 0.0029 → 0.0021 → 0.0015, each roughly 70% of the previous improvement — the concrete, numeric signature of diminishing returns.
The compute-optimal trade-off table shows directly: for the same fixed compute budget, a 1B parameter model can afford 166.7B training tokens, while a 50B parameter model can only afford 3.3B tokens — a genuine, real trade-off, not an abstract concept. Choosing a larger model at fixed compute genuinely means training on proportionally less data.

10. Emergent Capabilities

Emergent capability:   a capability that appears SUDDENLY, or much
                       more strongly, at sufficient scale -- not
                       present, or present only weakly, in smaller
                       models of the SAME architecture and training
                       procedure

Examples commonly discussed in the literature include certain multi-step reasoning or in-context learning capabilities that smaller models struggle with substantially, but larger models handle noticeably better — not as a simple linear extrapolation of smaller models’ performance, but as a more qualitative jump.

⚠️ Per this course’s depth calibration, this module doesn’t derive emergent capabilities from first principles or claim a settled theoretical explanation — the phenomenon is empirically observed and actively studied, not fully understood mechanistically.

11. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Scaling laws directly informed the “scale up” strategy behind essentially every major LLM’s development — labs use exactly this kind of compute-optimal reasoning (Section 8’s trade-off) to decide how to allocate a fixed training compute budget between model size and data quantity.

12. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: Moderate, indirectly. Emergent capabilities at sufficient scale are part of why larger models are often specifically chosen for agentic tasks requiring multi-step reasoning and tool use — smaller models may lack these capabilities even with the same architecture and similar training, a genuine, practical consideration in model selection for agent systems.

13. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming scaling produces unlimited, linear improvement.

Why it is incorrect: As verified directly, returns diminish predictably — loss keeps improving, but by progressively smaller amounts per unit of additional compute.

⚠️ Mistake

Incorrect idea: assuming bigger models are always the right choice for a fixed compute budget.

Why it is incorrect: As shown directly, more parameters at fixed compute means less training data — the trade-off matters, and “undertrained” large models are a real, documented phenomenon.

⚠️ Mistake

Incorrect idea: treating emergent capabilities as fully understood or guaranteed.

Why it is incorrect: As flagged directly, this remains an active area of study — not a deterministic guarantee that any specific capability will “emerge” at any specific scale.

14. Important Distinctions

Model Size (N)	Dataset Size (D)
Number of parameters (Module 12)	Number of training tokens (Module 8)
Both trade off against each other for a FIXED compute budget — verified directly

Diminishing Returns	Emergent Capabilities
Loss keeps improving with scale, but by progressively smaller amounts	Certain capabilities appear more suddenly/strongly at sufficient scale, not a smooth extrapolation

15. When to Use

Use compute-optimal reasoning (balancing model size against data quantity, Section 8) when planning a training run with a fixed compute budget — a genuine, practical planning consideration for organizations pretraining models.

16. When Not to Use

Don’t assume scaling alone solves every capability gap — for AI engineers working with existing models (rather than pretraining new ones), fine-tuning (Module 16) and prompting techniques often address task-specific gaps more directly and cheaply than seeking access to a larger base model.

17. Production Considerations

Larger models cost proportionally more to serve (Module 14, 24) — scaling’s benefits must be weighed against these real, ongoing inference costs, not just one-time training costs.
Diminishing returns apply to more than just loss — practical task performance improvements from scaling also tend to show diminishing returns for many tasks, a genuine consideration when evaluating whether a larger model justifies its added cost for your specific use case.

18. What You Should Remember

Scaling laws describe loss decreasing as a power law with compute, model size, or data — verified directly with genuine diminishing marginal returns at each step.
Compute-optimal allocation is a real trade-off between model size and training data quantity for a fixed compute budget — verified directly with real numbers.
Emergent capabilities appear at sufficient scale, but remain an active area of study — not something to assume deterministically.

19. Interview Questions

Beginner

Q: What do scaling laws describe in the context of LLMs?

Ans: The empirically observed relationship between a model’s training loss and the amount of compute, model size (parameters), or training data used — loss predictably decreases as any of these increase, following an approximately power-law pattern with diminishing returns.

Intermediate

Q: What does “diminishing returns” mean in the context of LLM scaling, and why does it matter practically?

Ans: It means each additional unit of compute (or model size, or data) produces a progressively smaller improvement in loss than the previous unit — verified directly, doubling compute repeatedly produced successively smaller loss improvements.

Practically, this means scaling up isn’t a free lunch — the cost of additional compute grows while the benefit shrinks, which is exactly why compute-optimal planning (balancing model size against data quantity) matters for efficient resource use.

Advanced

Q: Explain the practical trade-off between model size and dataset size for a fixed training compute budget.

Ans: Training compute is roughly proportional to the product of model size and dataset size (illustrated as Compute ≈ 6 × N × D).

For a FIXED compute budget, increasing the model size N directly reduces how many training tokens D can be afforded, and vice versa — verified directly: at a fixed compute budget, a 1B-parameter model could afford 166.7B training tokens, while a 50B-parameter model could only afford 3.3B tokens under the same budget.

This is precisely the trade-off “compute-optimal” training strategies address — finding the allocation between model size and data quantity that minimizes loss for a given compute investment, rather than simply maximizing model size.

Scenario

**Q: A team has a fixed training compute budget and is deciding between training a very large model on relatively little data, versus a smaller model on much more data.

What would you advise them to investigate?** A: I’d advise investigating compute-optimal scaling research findings specifically for their compute regime — historically, several early large models were found to be meaningfully undertrained relative to their size, meaning a smaller model trained on proportionally more data would have achieved better performance for the same compute investment.

The right allocation isn’t simply “bigger is always better” — it’s a genuine trade-off (verified directly in this module) between model size and data quantity, and the optimal balance depends on the specific compute budget available.

AI Engineering

Q: Why does understanding scaling laws matter for an AI engineer who will never personally pretrain a model, only use existing pretrained LLMs? A: It provides useful context for interpreting model releases and choosing between model options — understanding that larger models generally have more capacity but face diminishing returns, and that model quality depends on the training data/compute trade-off (not parameter count alone), helps in making informed decisions about which model size is genuinely appropriate for a given task’s complexity, rather than assuming “bigger is always better” or being surprised when a smaller, well-trained model performs comparably to a larger one for a specific application.

20. Next Step

Next: Module 14 — Inference — production-oriented: what actually happens when you send a prompt, prefill, decode, and the KV cache that makes serving practical.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026