Parameter Count

The Dense Model article closed on a promise: MoE introduced real ambiguity into a number that used to mean one simple thing. This article covers exactly that number, and exactly that ambiguity: parameter count.

The simple definition

Parameter count is the total number of learnable weights and biases in a model — the figure most commonly quoted when describing a model’s size, like “175 billion parameters” for GPT-3. Recall from the Parameters article’s foundational discussion — parameter count has always been the headline number the field uses to compare model scale, tracked constantly throughout this glossary’s discussion of GPT-3, Llama, and every other model.

Why this number used to be simple, and why it isn’t anymore

Recall from the Dense Model article: in a dense model, every parameter activates on every input, so “parameter count” and “compute per token” were always directly, predictably linked — a bigger parameter count reliably meant a proportionally bigger amount of actual computation per token. Recall from the Sparse Model and Mixture of Experts articles that this link breaks entirely for sparse architectures — a model’s total parameter count and its active parameter count (how many actually get used per token) can now be dramatically different numbers, and both are genuinely meaningful, describing different things.

flowchart LR
    A["Total Parameter Count: e.g. 671 billion"] --> B["All learned weights, whether or not they're used for a given token"]
    C["Active Parameter Count: e.g. 37 billion"] --> D["Only the parameters actually engaged for a specific token"]

Why both numbers genuinely matter, for different reasons

This is worth being precise about, since each number answers a different, real, practical question. Total parameter count determines how much memory a model needs just to be loaded — every parameter has to be stored somewhere, whether or not it’s used for a given token, echoing the memory discussion throughout the Parameters and KV Cache articles. Active parameter count determines how much actual computation happens per token — directly related to inference speed and cost, since GPUs, as covered throughout the Training article, are paid for by the actual calculations performed, not by parameters sitting unused in memory.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of a company’s total headcount — everyone on payroll — versus how many employees are actually working on any single specific project at a given moment. Both numbers describe something real and useful about the company: total headcount tells you overall organizational capacity and total payroll cost; the smaller, per-project number tells you how much work is actually happening on any one thing right now.

Where this breaks down: A company’s off-project employees are working on other, separate things. A sparse model’s inactive parameters, as covered in the Sparse Model article, aren’t doing anything at all for that specific token — they’re mathematically excluded entirely from that forward pass, not busy elsewhere, a genuine, structural difference from an employee simply assigned to a different project.

A concrete example, layered

For a simple beginner example: a small MoE model might be described as having “12 billion total parameters, 3 billion active” — meaning its full learned capacity spans 12 billion weights, but any single token only ever computes through about 3 billion of them. For a production example: DeepSeek-V3, covered throughout this phase, is publicly documented with roughly 671 billion total parameters and around 37 billion active parameters per token — two genuinely different, both meaningful numbers that a company evaluating the model needs to consider separately: the larger figure for memory and hosting requirements, the smaller figure for actual per-token inference cost and speed.

Why this distinction matters for real, practical comparisons

It’s worth being direct about a genuine, common source of confusion in real reporting and marketing. When a company announces a new model’s “parameter count,” it’s worth checking specifically whether that figure refers to total or active parameters, since the two numbers can differ by an order of magnitude for sparse architectures — comparing a dense model’s total parameter count directly against a sparse model’s active parameter count, without noting the difference, produces a genuinely misleading comparison of the two models’ real capability or cost.

Calculate what parameter count means for memory

If a dense model has 9 billion parameters, storing only its weights requires approximately:

FP32: 9 billion × 4 bytes   ≈ 36 GB
FP16: 9 billion × 2 bytes   ≈ 18 GB
INT8: 9 billion × 1 byte    ≈  9 GB
INT4: 9 billion × 0.5 bytes ≈ 4.5 GB

These are simplified weight-only calculations. Real inference also needs memory for the KV cache, activations, runtime buffers, and framework overhead. Training commonly requires much more memory for gradients and optimizer state.

For an MoE model, ask for both numbers: DeepSeek-V3 has 671B total parameters, describing storage and total capacity, and 37B active parameters per token, describing approximate active computation. Parameter count alone does not reveal data quality, architecture, benchmark quality, or speed. Source: DeepSeek-V3 Technical Report.

Common misconception

A frequent beginner assumption: that “parameter count” is always one single, unambiguous number, the way it effectively was for every dense model covered earlier in this glossary. As this article has explained, this stopped being true the moment sparse architectures became mainstream — any serious comparison of a modern model’s scale now needs to specify total versus active parameter count explicitly, rather than assuming the single figure everyone used to quote still tells the whole story.

Where this fits in what comes next

You now understand both meanings parameter count can carry in the sparse-model era. The next article, Model Architecture, zooms out to the broader concept encompassing everything covered in this phase so far — the overall structural design choices, beyond just parameter count, that define how a model actually works.

In one sentence

Parameter count is the headline number describing a model’s total learned capacity, but the rise of sparse architectures like Mixture of Experts split it into two genuinely different, equally important figures — total parameters (memory cost) and active parameters (per-token compute cost) — and any serious comparison between modern models needs to specify which one it’s actually talking about.

Related Terms

• Parameters
• Sparse Model
• Dense Model
• Mixture of Experts (MoE)
• Model Architecture