Model Parameters and Architecture

Before you continue: three tools for this module

Token: a piece of text processed by the model.
Parameter: a learned number controlling the model’s transformations.
Inference: using the trained model without updating its parameters.

You do not need to memorize these yet. Use this map when the terms reappear.

Begin with the central question

What hidden problem does Model Parameters and Architecture solve inside a real language-model system?

Keep that central question about Model Parameters and Architecture in mind. The definitions, numbers, diagrams, and code examples below answer it one piece at a time.

architecture defines operations → parameters store learned numerical behavior

1. What You Will Learn

Learning outcomes

Distinguish model architecture from learned parameters.
Calculate parameter counts for simplified layers and embeddings.
Explain why parameter count alone does not determine model quality.
Connect width, depth, attention design, memory, and compute cost.

In one sentence

💡 Big picture

Architecture is the model’s blueprint; parameters are the learned numbers filling that blueprint after training.

2. Why This Module Exists

The problem this module solves

Two models can follow a similar blueprint but learn different parameter values.
A larger parameter count can increase capacity, but it does not guarantee better data, reasoning, safety, or answers.

3. Intuition

a “parameter” is simply one learnable number — one entry in one of the weight matrices you already know completely from the Transformers course (Wq, Wk, Wv, Wo, the FFN’s W1/W2, the embedding table, the LM head). “70 billion parameters” means the sum of every single entry across every one of these matrices, across every stacked layer, totals 70 billion learnable numbers.

Analogy: The Skyscraper Floor Space Calculation Think of calculating a model’s parameter count like calculating the total floor space of a massive skyscraper:

The Lobby (Embeddings): The base floor is the embedding table. Its area is determined by vocabulary width and vector size (vocab_size * d_model). This lobby stays exactly the same size regardless of how many stories you build.

The Stacked Stories (Transformer Layers):

Attention columns: 4 major support walls per floor (4 * d_model^2).

Feed-forward shelves: Gated expansion zones (2 * d_model * d_ff).

The Compounding Area (Quadratic scaling): If you double the ceiling height (stack twice as many layers), your skyscraper’s total area doubles linearly. But if you try to make the building wider (doubling the width d_model), your total floor space quadruples because the attention columns (d_model^2) and FFN shelves (d_model * d_ff) scale quadratically.

📊 Visual Chart: Parameter Budget Allocation

Here is how parameters are distributed across layers and classification heads in standard configurations:

graph TD
    classDef block fill:#e67e22,stroke:#333,stroke-width:1px,color:#fff;
    classDef emb fill:#3498db,stroke:#333,stroke-width:1px,color:#fff;

    subgraph Sizing ["Total Parameter Sizing Budget"]
        direction LR
        Lobby["Embedding Table<br>(vocab_size x d_model)"]:::emb
        LMClassifier["LM Head Classifier<br>(vocab_size x d_model)"]:::emb

        subgraph LayerStack ["Stacked Transformer Layers (e.g. 24 or 96 blocks)"]
            direction TB
            Layer1["Layer 1 block: Attention + Gated FFN<br>(4 x d_model^2 + 2 x d_model x d_ff)"]:::block
            Layer2["Layer 2 block: Attention + Gated FFN"]:::block
            LayerN["Layer N block: Attention + Gated FFN"]:::block
        end
    end

4. Core Concept

Parameters:           the LEARNABLE numbers in a model -- every
                      entry in every weight matrix (and bias
                      vector), across every layer

Layers:                  how many Transformer blocks are stacked
                      (Module 10)

Hidden dimension          the width of each token's representation
(d_model):            vector throughout the model (Transformers
                      course)

Attention heads:            how many parallel attention
                        computations each block runs (Transformers
                        course)

Vocabulary size:              how many unique tokens the model
                          recognizes (Module 2)

Embedding dimension:             same as d_model -- the size of
                            each token's embedding vector

Feed-forward dimension              the FFN's internal expanded
(d_ff):                        dimension (Transformers course,
                              commonly 4x d_model)

5. How It Works — Step by Step (Parameter Counting)

1. EMBEDDING TABLE: vocab_size x d_model parameters
2. Per Transformer block:
   - ATTENTION: 4 matrices (Wq, Wk, Wv, Wo), each d_model x
     d_model -> 4 x d_model^2 parameters
   - FEED-FORWARD: two matrices, d_model x d_ff and d_ff x
     d_model -> 2 x d_model x d_ff parameters
   - LayerNorm: a small number of additional parameters (scale/
     shift vectors) -- negligible at scale
3. Multiply the PER-BLOCK total by NUM_LAYERS
4. LM HEAD: vocab_size x d_model parameters (sometimes TIED --
   sharing weights with the embedding table, Module 4)
5. TOTAL = embedding + (per-block total x num_layers) + LM head

6. Mathematical Intuition

Read the mathematics as a story

architecture defines operations → parameters store learned numerical behavior

First locate the input, operation, and output. Then treat the formula as a compact description of that journey rather than a collection of symbols to memorize.

attention_params_per_block = 4 x d_model^2

ffn_params_per_block = 2 x d_model x d_ff

total ≈ (vocab_size x d_model x 2)          [embedding + LM head]
      + num_layers x (4 x d_model^2 + 2 x d_model x d_ff)

Every variable here is something you already know from the Transformers course — this formula is simply that architecture’s shapes, summed.

7. Small Worked Example

Walk through the example

Name what each input represents.
Follow one transformation at a time.
Translate the result back into ordinary language.

The purpose is to reveal the mechanism, not merely display an answer.

Doubling num_layers roughly doubles the total parameter count coming from the Transformer blocks (since each block’s parameter count stays fixed, and you’re just stacking more of them) — but the embedding table and LM head’s parameter count stays exactly the same, since it only depends on vocab_size and d_model, not depth.

8. Python Example

What the code will demonstrate

The code builds a tiny version of the mechanism, prints values you can inspect, and connects them to the worked example. Predict the direction of the result before running it.

Python symbols used below

NumPy (np) stores numeric vectors and matrices.
np.array(...) creates a numeric collection.
Library calls perform the same conceptual steps shown above at a larger scale.

# Build a small, inspectable example of Model Parameters and Architecture.
# Follow the inputs, transformations, and output in order.
def count_parameters(vocab_size, d_model, num_layers, num_heads, d_ff):
    embedding_params = vocab_size * d_model
    lm_head_params = vocab_size * d_model

    attn_params_per_block = 4 * (d_model * d_model)       # Wq, Wk, Wv, Wo
    ffn_params_per_block = (d_model * d_ff) + (d_ff * d_model)
    ln_params_per_block = 4 * d_model                       # small, included for completeness

    per_block_total = attn_params_per_block + ffn_params_per_block + ln_params_per_block
    all_blocks_total = per_block_total * num_layers

    total = embedding_params + lm_head_params + all_blocks_total
    return {
        "embedding": embedding_params, "lm_head": lm_head_params,
        "per_block": per_block_total, "all_blocks": all_blocks_total, "total": total,
    }

configs = [
    ("Tiny",   {"vocab_size": 32000, "d_model": 512,  "num_layers": 6,  "num_heads": 8,  "d_ff": 2048}),
    ("Small",  {"vocab_size": 32000, "d_model": 1024, "num_layers": 12, "num_heads": 16, "d_ff": 4096}),
    ("Medium", {"vocab_size": 32000, "d_model": 2048, "num_layers": 24, "num_heads": 32, "d_ff": 8192}),
]

for name, cfg in configs:
    result = count_parameters(**cfg)
    print(f"{name} model: {cfg}")
    print(f"  Embedding table: {result['embedding']:>15,}")
    print(f"  LM head:         {result['lm_head']:>15,}")
    print(f"  All {cfg['num_layers']:>2} blocks:   {result['all_blocks']:>15,}")
    print(f"  TOTAL:           {result['total']:>15,}  ({result['total']/1e6:.1f}M)\n")

tiny = count_parameters(**configs[0][1])
print(f"Tiny model: Transformer blocks = {tiny['all_blocks']/tiny['total']*100:.1f}% of total,")
print(f"embeddings+LM head = {(tiny['embedding']+tiny['lm_head'])/tiny['total']*100:.1f}%")

Expected Output:

Tiny model: {'vocab_size': 32000, 'd_model': 512, 'num_layers': 6, 'num_heads': 8, 'd_ff': 2048}
  Embedding table:      16,384,000
  LM head:               16,384,000
  All  6 blocks:         18,886,656
  TOTAL:                 51,654,656  (51.7M)

Small model: {'vocab_size': 32000, 'd_model': 1024, 'num_layers': 12, 'num_heads': 16, 'd_ff': 4096}
  Embedding table:       32,768,000
  LM head:               32,768,000
  All 12 blocks:        151,044,096
  TOTAL:                216,580,096  (216.6M)

Medium model: {'vocab_size': 32000, 'd_model': 2048, 'num_layers': 24, 'num_heads': 32, 'd_ff': 8192}
  Embedding table:       65,536,000
  LM head:               65,536,000
  All 24 blocks:      1,208,156,160
  TOTAL:              1,339,228,160  (1339.2M)

Tiny model: Transformer blocks = 36.6% of total,
embeddings+LM head = 63.4%

9. How It Works

Doubling both d_model (512 → 1024) and num_layers (6 → 12) didn’t just double the total parameter count — it grew from 51.7M to 216.6M, roughly 4.2x — because attention and FFN parameter counts scale with d_model²-ish terms (specifically d_model × d_ff and d_model²), not linearly. Scaling width AND depth together compounds.
For the Tiny model, embeddings + LM head make up 63.4% of total parameters — a genuinely large share at small scale, exactly matching Module 4’s note about embedding matrix size. As models grow (Small, Medium), this share shrinks dramatically, since Transformer block parameters scale much faster with d_model and num_layers than the embedding table does (which only scales with d_model, linearly). This is precisely why very large real LLMs’ parameter counts are overwhelmingly dominated by their Transformer blocks, not their vocabulary/embedding size.

10. What Does “70B Parameter Model” Actually Mean?

It means: summing every entry across every attention matrix, every feed-forward matrix, the embedding table, and the LM head, across every stacked layer, totals approximately 70 billion individual learnable numbers — computed via exactly the formula verified above, just at a much larger scale (many more layers, much larger d_model and d_ff).

11. Why More Parameters ≠ Automatically Better Performance

More parameters generally mean more capacity to represent complex patterns — but capacity isn’t automatically realized capability. A model with more parameters trained on too little or low-quality data (Module 8’s data quality emphasis) can underperform a smaller, well-trained model. This directly connects to Module 13’s scaling laws, which formalize the real relationship between parameters, data, and compute.

12. How Is This Used in Modern AI?

Trace it through a real model call

user message → assembled context → LLM computation → decoded output → application checks

This topic affects one stage of that path; it is not the complete product. Hosted GPT- and Gemini-style applications also add instructions, safety systems, retrieval, tools, serving infrastructure, and evaluation around the model.

🤖 How Is This Used in Modern AI?

Parameter count is one of the primary levers determining a model’s serving cost, memory footprint, and latency (Module 14, 24) — understanding exactly where those parameters live (mostly in Transformer blocks at real scale, per this module’s verified result) is directly useful for reasoning about model size trade-offs when choosing between model variants.

13. How Is This Used in Agentic AI?

Separate the model from the runtime

goal + state + tool results → LLM proposal → runtime validation → execution or response

The LLM proposes text or a structured action. Ordinary application code controls permissions, tools, retries, memory, and execution.

Direct relevance to Agentic AI: Moderate, practically. When choosing which model size/variant to use for a given agent capability (e.g., a smaller model for fast, cheap routing decisions vs. a larger model for complex reasoning), understanding parameter count’s real cost implications — not just as an abstract number, but as a driver of memory and compute requirements — directly informs this trade-off.

14. Common Beginner Mistakes

⚠️ Mistake

Incorrect idea: assuming parameter count scales linearly with layer count alone.

Why it is incorrect: As verified directly, scaling both depth AND width together compounds the parameter count non-linearly.

⚠️ Mistake

Incorrect idea: assuming embeddings are a negligible fraction of parameters.

Why it is incorrect: As shown directly, at smaller model scales, they can be the MAJORITY of total parameters — this fraction shrinks as models scale up, but isn’t always negligible.

⚠️ Mistake

Incorrect idea: believing more parameters guarantees better performance.

Why it is incorrect: As emphasized directly, parameters represent capacity, not guaranteed capability — data quality and quantity (Module 8) and training procedure matter enormously too.

15. Important Distinctions

Parameters	Layers
The total count of learnable numbers	How many Transformer blocks are stacked — one factor determining parameter count

Parameter Count	Model Capability
A measure of representational CAPACITY	Actual performance — depends on capacity, data quality/quantity, and training procedure together

16. When to Use

Use larger parameter-count models when task complexity genuinely requires more representational capacity and the cost/latency trade-off (Module 14, 24) is acceptable for the application.

17. When Not to Use

Don’t default to the largest available model when a smaller one, given adequate training, performs comparably for your specific task — larger models cost more in memory, latency, and inference price (Module 14), often without a proportional accuracy benefit for simpler tasks.

18. Production Considerations

Parameter count directly determines memory requirements for serving a model — a genuine, hard infrastructure constraint (Module 24 covers quantization as one mitigation).
Model size selection is a real, practical trade-off between capability, cost, and latency — not simply “always choose the biggest.”

19. What You Should Remember

Parameters are the learnable numbers across every weight matrix in the model — computed directly, verified precisely from real architecture specs.
Scaling depth and width together compounds parameter count non-linearly — verified directly: doubling both roughly quadrupled the total.
Embedding/LM head parameters can dominate at small scale, but their share shrinks as models grow larger — verified directly across three model sizes.
More parameters ≠ automatically better performance — capacity is necessary but not sufficient; data and training matter too (Module 13 formalizes this).

20. Interview Questions

Beginner

Q: What does it mean when someone says “this is a 70B parameter model”? A: It means the total count of learnable numbers across every weight matrix in the model — every attention projection matrix, every feed-forward matrix, the embedding table, and the LM head, summed across all stacked layers — totals approximately 70 billion.

Intermediate

Q: Why does doubling both the number of layers and the hidden dimension of a model more than double its total parameter count?

Ans: Attention and feed-forward parameter counts scale with terms like d_model² and d_model × d_ff, not just linearly with d_model — doubling d_model alone roughly quadruples these terms.

Combined with doubling the number of layers (which linearly multiplies the per-block parameter count), the total effect compounds well beyond a simple doubling — verified directly in this module, where doubling both dimensions together produced roughly a 4.2x increase in total parameters.

Advanced

Q: Why can embedding and LM head parameters represent a much larger fraction of total parameters in a small model than in a large one?

Ans: The embedding table and LM head’s parameter count scales only with vocab_size × d_model — it doesn’t grow with the number of layers at all.

Transformer block parameters, by contrast, scale with both d_model (roughly quadratically, via attention and FFN terms) AND num_layers (linearly).

As a model’s depth and width both increase, the Transformer blocks’ parameter count grows much faster than the embedding table’s — verified directly: embeddings and LM head made up 63.4% of total parameters in the smallest model tested, but this proportion shrinks substantially as the model scales up, since block parameters begin to dominate.

Scenario

**Q: A team is choosing between a 7B and a 70B parameter model for a production application, and cost/latency is a real concern.

What factors, beyond raw parameter count, should inform this decision?** A: Beyond the raw parameter count (which directly affects memory and compute cost, Module 14/24), they should consider whether the specific task genuinely requires the larger model’s additional capacity — verified directly in this module, more parameters represent capacity, not automatically better performance.

If the smaller model, given adequate training or fine-tuning (Module 16) for their specific task, performs comparably, the significant cost and latency savings of the smaller model would likely make it the better practical choice — this kind of capability-vs-cost evaluation, rather than defaulting to the largest available model, is a genuine, standard production consideration.

AI Engineering

**Q: Why is understanding exactly where a model’s parameters live (embeddings vs.

Transformer blocks) practically useful for an AI engineer, beyond satisfying curiosity?** A: It directly informs reasoning about model size trade-offs — for instance, increasing vocabulary size (Module 2) has a real, quantifiable parameter cost concentrated in the embedding table and LM head, whereas increasing depth or width primarily grows the Transformer blocks’ parameter count.

Understanding these distinct scaling behaviors, verified directly in this module, helps in evaluating architectural trade-offs when comparing models or reasoning about why two models with similar total parameter counts but different architecture choices (deep vs. wide, large vs. small vocabulary) might have meaningfully different capability and cost profiles.

21. Next Step

Next: Module 13 — Scaling LLMs — why bigger models became possible, scaling laws, and emergent capabilities that appear only at sufficient scale.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026