How LLMs Actually Use Deep Learning

Begin with the central question

When an LLM writes one word, what chain of deep-learning operations produced it?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

text → tokens → vectors → Transformer layers → logits → next token → repeat

Before you continue: three tools for this module

Token ID: the integer assigned to one tokenizer entry.
Logit: one raw score for a possible next token.
Decoding or sampling: the rule used to choose a token from the model’s output distribution.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

The complete, integrated trace: text → tokenization → token IDs → token embeddings → positional information → Transformer blocks → hidden representations → logits → probability distribution → next token — using nothing but concepts you’ve already built and verified across this course. Then, precisely what differs between LLM training and inference.

One generation step follows this path:

text → tokens → token IDs → vectors → Transformer blocks
     → logits over vocabulary tokens → sampling rule → next token
                                                ↓
                                  append and repeat

Tokens are not necessarily whole words, and logits are raw scores rather than confidence. During training, many target positions can be processed together; during autoregressive generation, newly chosen tokens are produced sequentially.

Why the Pieces Must Be Connected into One Generation Loop

Modules 1-16 built every individual piece. This module exists to assemble them into the one thing you actually came here to understand: what genuinely happens, step by step, when you send a prompt to an LLM and it generates a response. Nothing new is introduced here — this is integration, not new material.

Autocomplete Repeated at Model Scale

an LLM is a very large, very deep function that takes in a sequence of tokens and outputs one thing: a probability distribution over “what token comes next.” Generating a full response is just this one operation, repeated — feed in everything so far, get a next-token probability distribution, pick one, add it to the sequence, repeat.

Analogy: The Autocomplete Megaphone Imagine you are using the predictive text autocomplete bar on your smartphone keyboard:

Inherent Autocomplete: If you type “The grass is”, the phone suggests “green” ( $70\%$ probability), “long” ( $20\%$ ), or “dry” ( $10\%$ ). It doesn’t “think” about botany; it has simply seen millions of sentences containing those phrases.

The Loop (Autoregressive Generation):

You select “green” from the suggestions.

The smartphone now feeds the new, extended phrase back into its engine: “The grass is green…”

Based on this updated input, it generates a new set of suggestions: “and” ( $80\%$ ), “but” ( $10\%$ ), “now” ( $5\%$ ).

You select “and”, and repeat the process.

An autoregressive LLM uses the same next-token loop at a much larger scale, with learned representations and many Transformer layers. Autocomplete is a useful analogy for the mechanism, but it does not describe the model’s full learned capabilities, training process, tool use, or surrounding application.

📊 Visual Flowchart: The Autoregressive Token Generation Loop

Here is the end-to-end feedback loop executed sequentially for every generated token. A token may be a whole word, part of a word, punctuation, or another learned text unit:

graph TD
    InputText["1. User Prompt:<br>'The sky is'"] --> Tokenizer["2. Tokenizer:<br>Convert to token IDs [342, 9811, 412]"]
    Tokenizer --> Embed["3. Embedding Lookup (Module 12) & Positional Encoding (Module 16)"]
    Embed --> Trans["4. Stack of Transformer Blocks:<br>LayerNorm -> Multi-Head Causal Attention -> MLP -> Residuals"]
    Trans --> Logits["5. Output Logits:<br>Raw score for each token in the model vocabulary"]

Logits --> Softmax["6. Softmax (Module 4):<br>Normalize logits to probabilities that sum to 1"]

Softmax --> Sampler{"7. Next-Token Sampler<br>(Temperature / Top-P)"}

Sampler -->|Selects token| NextWord["Chosen token: 'blue'"]

NextWord --> Append["8. Append to Prompt:<br>'The sky is blue'"]
    Append -->|Feed back as new input| InputText

4. Core Concept — The Complete Trace

Text
  ↓
Tokenization              (splitting text into sub-word units)
  ↓
Token IDs                  (each token mapped to an integer index)
  ↓
Token Embeddings            (Module 12 — each ID looked up in a
                            learned embedding table)
  ↓
+ Positional Information     (Module 16)
  ↓
Transformer Blocks            (Module 16 — many stacked blocks, each
                              containing multi-head self-attention,
                              Module 15, plus feed-forward + residuals
                              + normalization)
  ↓
Hidden Representations         (the output of the final block — a
                              vector per token position)
  ↓
Logits                          (the LAST position's hidden state,
                              projected to vocabulary size)
  ↓
Probability Distribution         (softmax over the logits, Module 4)
  ↓
Next Token                        (selected via some sampling
                              strategy)

5. How It Works — Step by Step

1. TOKENIZATION: raw text is split into sub-word tokens (this
   specific process is a large topic on its own, beyond this
   course's scope -- the key idea: text becomes a sequence of
   discrete units)
2. Each token is mapped to a TOKEN ID (an integer index into a
   fixed vocabulary)
3. Each token ID is looked up in a learned EMBEDDING TABLE
   (Module 12), producing a dense vector per token
4. POSITIONAL INFORMATION is added (Module 16)
5. This sequence of vectors flows through MANY STACKED
   TRANSFORMER BLOCKS (Module 16) -- each applying multi-head
   self-attention (Module 15) and a feed-forward network
   (Module 4), with residual connections and normalization
   (Module 11) throughout
6. The FINAL block produces one hidden representation vector
   per token position
7. To predict the NEXT token, the LAST position's hidden vector
   is projected (another weighted-sum operation, Module 2) into
   a vector the size of the entire VOCABULARY -- these raw scores
   are the LOGITS
8. SOFTMAX (Module 4) converts these logits into a genuine
   probability distribution over every possible next token
9. A token is SELECTED from this distribution (e.g., greedily
   picking the highest-probability token, or sampling -- a
   separate topic beyond this course's scope)
10. The selected token is appended to the sequence, and the
    ENTIRE process repeats to generate the token after that

6. Mathematical Intuition

First, use only small numbers

Imagine a tiny vocabulary with logits for cat, dog, and runs. Softmax might turn them into probabilities [0.6, 0.3, 0.1]; decoding selects one token, appends it, and runs another forward pass for the next position.

Read the mathematics as a story

An LLM is a trained neural network that repeatedly predicts a token. Its apparent conversation emerges from many numerical transformations plus decoding—not from a stored collection of ready-made sentences.

text → tokens → vectors → Transformer layers → logits → next token → repeat

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. Nothing new mathematically — this module’s entire value is in seeing Modules 2, 4, 5, 12, 15, and 16’s individual computations connected into one continuous pipeline, verified completely below.

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible.

Given the partial sentence “the cat sat,” an LLM’s forward pass produces a probability distribution over its entire vocabulary for what comes next — likely assigning meaningfully higher probability to plausible continuations (“on,” “down,” “quietly”) than to implausible ones (“purple,” “run,”, grammatically broken continuations), purely because of patterns learned during training (Module 6’s cross-entropy, applied at massive scale during pretraining).

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of How LLMs Actually Use Deep Learning.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

def softmax(x):
    exp_x = np.exp(x - np.max(x))
    return exp_x / np.sum(exp_x)

np.random.seed(5)

vocab = ["the", "cat", "sat", "mat", "dog", "ran"]
vocab_size = len(vocab)
d_model = 4

# Step 1-2: Tokenization -> Token IDs
input_text_tokens = ["the", "cat", "sat"]
token_ids = [vocab.index(t) for t in input_text_tokens]
print("Input tokens:", input_text_tokens)
print("Token IDs:", token_ids)

# Step 3: Token embeddings (a learned lookup table)
embedding_table = np.random.randn(vocab_size, d_model) * 0.5
token_embeddings = embedding_table[token_ids]
print("\nToken embeddings:\n", np.round(token_embeddings, 3))

# Step 4-6: Simplified Transformer processing (full mechanics
# already verified in Modules 15-16 — one linear transform here
# stands in for the full stack, to keep focus on the OVERALL
# pipeline shape)
W_transform = np.random.randn(d_model, d_model) * 0.3
hidden_states = token_embeddings @ W_transform
print("\nHidden states (after Transformer processing):\n", np.round(hidden_states, 3))

# Take the LAST position's hidden state -- this predicts the next token
last_hidden = hidden_states[-1]
print("\nLast position's hidden state:", np.round(last_hidden, 3))

# Step 7: Project to vocabulary size -> LOGITS
W_output = np.random.randn(vocab_size, d_model) * 0.5
logits = W_output @ last_hidden
print("\nLogits (raw score per vocabulary word):")
for word, logit in zip(vocab, logits):
    print(f"  {word:6s}: {logit:.4f}")

# Step 8: Softmax -> probability distribution
probs = softmax(logits)
print("\nProbability distribution over next token:")
for word, prob in zip(vocab, probs):
    print(f"  {word:6s}: {prob:.4f}")
print("Sum of probabilities:", probs.sum())

# Step 9: Select next token (greedy)
next_token_id = np.argmax(probs)
print(f"\nPredicted next token (greedy): '{vocab[next_token_id]}' (probability {probs[next_token_id]:.4f})")

Expected Output:

Input tokens: ['the', 'cat', 'sat']
Token IDs: [0, 1, 2]

Token embeddings:
 [[ 0.221 -0.165  1.215 -0.126]
 [ 0.055  0.791 -0.455 -0.296]
 [ 0.094 -0.165 -0.596 -0.102]]

Hidden states (after Transformer processing):
 [[ 0.056 -0.035 -0.051  0.039]
 [ 0.    -0.118  0.204 -0.328]
 [ 0.023  0.002 -0.026 -0.072]]

Last position's hidden state: [ 0.023  0.002 -0.026 -0.072]

Logits (raw score per vocabulary word):
  the   : -0.0181
  cat   : -0.0276
  sat   : -0.0299
  mat   : -0.0497
  dog   : -0.0114
  ran   : 0.0287

Probability distribution over next token:
  the   : 0.1666
  cat   : 0.1650
  sat   : 0.1646
  mat   : 0.1614
  dog   : 0.1677
  ran   : 0.1746
Sum of probabilities: 1.0

Predicted next token (greedy): 'ran' (probability 0.1746)

9. How It Works

This is Section 4’s complete pipeline diagram, run end to end with real numbers: 3 input tokens become 3 embedding vectors, flow through a (here, simplified) Transformer stage, and the last position’s hidden state alone is what gets projected into logits — this is a genuine, important detail: only the final position’s representation is used to predict the next token, even though every position’s hidden state was computed.
The probability distribution genuinely sums to exactly 1.0 — softmax’s defining property (Module 4), confirmed here over a real 6-word vocabulary instead of the toy 3-class example from that module.
This tiny untrained model’s probabilities are nearly uniform (~0.16 to ~0.17 each) — expected, since the weights are random, not learned. A real, trained LLM’s logits would show far sharper, more meaningful differences, reflecting genuine learned language patterns rather than near-random noise.

10. Training vs. Inference — For LLMs Specifically

DURING TRAINING:

tokens
  ↓
prediction (the exact pipeline above)
  ↓
LOSS               (compare predicted probabilities against the
                   ACTUAL next token that appeared in real training
                   text — cross-entropy, Module 6)
  ↓
backpropagation      (Module 7 — compute gradients for EVERY
                     parameter, across every layer)
  ↓
gradients
  ↓
optimizer              (Module 9 — typically AdamW — updates every
                       parameter)
  ↓
parameter update

DURING INFERENCE:

prompt
  ↓
forward pass            (the exact pipeline above)
  ↓
logits
  ↓
sampling/selection
  ↓
next token
  ↓
repeat (append the new token, run forward pass again)

⚠️ Inference does NOT update model weights. Every parameter in the model is completely fixed during inference — generating a response does not change the model in any way for future conversations. Only training (or fine-tuning, Module 16 of the ML course) updates parameters.

11. Python Example — Training vs. Inference, Contrasted

What the code will demonstrate

# TRAINING: we know the TRUE next token, compute cross-entropy loss
true_next_token_id = 3  # suppose the true next word was "mat"
loss = -np.log(probs[true_next_token_id])
print("TRAINING:")
print(f"  True next token: '{vocab[true_next_token_id]}'")
print(f"  Model's predicted probability for it: {probs[true_next_token_id]:.4f}")
print(f"  Cross-entropy loss: {loss:.4f}")
print("  -> This loss would now flow through backpropagation (Module 7)")
print("     to update EVERY parameter in the model (Module 8-9)")

# INFERENCE: no true label, no loss -- just sample/select from probs
print("\nINFERENCE:")
sampled_token_id = np.argmax(probs)
print(f"  No true label available -- just generate")
print(f"  Selected token (greedy): '{vocab[sampled_token_id]}'")
print("  -> NO loss computed, NO gradients, NO parameter updates")

Expected Output:

TRAINING:
  True next token: 'mat'
  Model's predicted probability for it: 0.1614
  Cross-entropy loss: 1.8237
  -> This loss would now flow through backpropagation (Module 7)
     to update EVERY parameter in the model (Module 8-9)

INFERENCE:
  No true label available -- just generate
  Selected token (greedy): 'ran'
  -> NO loss computed, NO gradients, NO parameter updates

This confirms Section 10’s diagram concretely: the exact same forward pass and logits are used for both training and inference — the only difference is what happens after: training computes a loss and backpropagates; inference simply selects a token and stops.

12. Real-World Example

When Claude (or any LLM) responds to your message, this exact pipeline runs once per generated token: your entire conversation (tokenized, embedded, positionally encoded) flows through the model’s Transformer blocks, the final position’s hidden state produces logits over the vocabulary, softmax converts these to probabilities, a token is selected, and the process repeats — each new token added to the context for the next forward pass — until a complete response has been generated.

13. How Is This Used in Modern AI?

Follow it from mechanism to product

An LLM combines tokenization, embeddings, repeated Transformer blocks, an output projection, and decoding. GPT and Gemini products add system instructions, safety layers, multimodal processing, tools, retrieval, and serving infrastructure around that learned model.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

This entire module is modern AI, at the mechanical level. Every interaction with an LLM is this exact pipeline, run repeatedly. Every LLM you’ve ever used learned everything it knows through the training half of Section 10, applied across an enormous training corpus.

Concept	AI application
The full pipeline	Literally what happens for every LLM API call
Training vs. inference	Directly explains why an LLM’s knowledge is “frozen” at inference time, and why fine-tuning (a separate training process) is needed to change its behavior durably
Next-token probability distribution	What “temperature” and sampling settings (in an API call) actually manipulate

14. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Very High. Every reasoning step an agent’s LLM performs — deciding which tool to call, interpreting a tool’s result, drafting a final response — is one or more runs of exactly this pipeline. An agent’s “context window” is literally the sequence of tokens (system prompt, conversation history, tool results) fed into Section 4’s pipeline at each step.

Understanding this is what makes an agent’s occasionally surprising behavior explicable: it’s a genuinely mechanical consequence of what tokens were in context and what probability distribution the model produced from them — not an unexplainable black box.

15. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: “LLMs are databases.”

Why it is incorrect: An LLM has no explicit, queryable storage of facts — everything it “knows” is encoded, distributedly, across billions of learned parameters (weights), discovered through training on next-token prediction. It’s a very different kind of information storage than a database’s explicit key-value lookups.

⚠️ Mistake

Incorrect idea: inference updates model weights.

Why it is incorrect: As Section 10 states explicitly — it does not. Every parameter is fixed during inference; generating a response never changes the model for future use.

⚠️ Mistake

Incorrect idea: the loss function itself changes the weights.

Why it is incorrect: Restated from Module 6, worth repeating here at LLM scale: the loss only measures wrongness; backpropagation computes gradients, and the optimizer applies the actual update — three genuinely distinct steps.

⚠️ Mistake

Incorrect idea: a larger model automatically has more intelligence.

Why it is incorrect: As noted in Module 2, parameter count measures capacity, not guaranteed capability — a large model under-trained on too little or poor-quality data can underperform a smaller, well-trained one.

16. Important Distinctions

LLM Training	LLM Inference
Forward pass → loss → backward pass → parameter update	Forward pass → output only
Parameters CHANGE	Parameters FIXED
Requires the true next token (from training data)	No true label needed — just generates
Happens rarely, expensively (pretraining/fine-tuning)	Happens constantly, comparatively cheaply

Logits	Probabilities
Raw, unbounded scores from the final projection	After softmax — bounded (0,1), sum to 1

17. Interview Questions

Beginner

Q: What does an LLM actually produce when it processes a prompt?

Ans: A probability distribution over its entire vocabulary, representing its prediction for what token should come next, given everything in the current context. Generating a full response means repeating this process — predict a next-token distribution, select a token, append it, repeat.

Intermediate

Q: Why does an LLM’s forward pass use only the LAST token position’s hidden state to predict the next token, even though every position has its own hidden state?

Ans: Because next-token prediction is specifically asking “what comes after everything so far” — the last position’s hidden state, having been computed via self-attention (Module 15) over the entire preceding sequence, is the representation that has “seen” all the relevant context needed to make that specific prediction.

The other positions’ hidden states remain useful during training (each position also gets its own next-token prediction, contributing to the overall training loss) but for single-step inference, only the final position’s prediction determines what’s generated next.

Advanced

Q: Trace precisely what changes, and what stays the same, between an LLM’s training forward pass and its inference forward pass.

Ans: The forward pass itself — tokenization through logits (Section 4) — is identical in both cases, using the exact same parameters and computation. What differs is what happens next: during training, the predicted probability distribution is compared against the actual next token from real training text, producing a loss (cross-entropy, Module 6), which flows backward through backpropagation (Module 7) to compute gradients, which the optimizer (Module 9) uses to update every parameter.

During inference, there’s no true label to compare against — the process simply selects a token from the predicted distribution and stops; no loss, no gradients, no parameter updates occur.

Scenario

Q: A user asks an LLM a factual question, and it responds confidently but incorrectly. Using this module’s concepts, explain what’s happening mechanically — is this a “training” or “inference” event, and what does that imply?

Ans: This is purely an inference event — the model’s parameters are completely fixed, and it’s simply producing whatever probability distribution its already-trained weights happen to generate for this specific prompt, then selecting a token from it. The “confidence” is just how peaked (via softmax) the token probabilities happen to be — it doesn’t reflect the model checking its answer against ground truth in any way, since no such checking mechanism exists at inference time.

This directly explains why LLMs can produce fluent, confident-sounding, yet factually wrong output: fluency and factual correctness are two different things the training process doesn’t guarantee will always align.

AI Engineering

Q: When you fine-tune an LLM, which half of Section 10’s diagram is actually running, and what does that tell you about what fine-tuning can and cannot do?

Ans: Fine-tuning runs the training half — forward pass, loss (computed against your fine-tuning dataset’s examples), backpropagation, and parameter updates — just starting from the model’s already-pretrained weights rather than random initialization, and typically with a lower learning rate (Module 9) and on a much smaller dataset than pretraining used.

This means fine-tuning genuinely can change the model’s behavior and style, since it’s a real training process updating real parameters — but it’s still bounded by the same training dynamics covered throughout this course (Module 6’s catastrophic forgetting risk, Module 11’s overfitting risk on a small fine-tuning set), not a fundamentally different or magic process.

18. What You Should Remember

The complete LLM pipeline: text → tokens → token embeddings + positional info → Transformer blocks → hidden states → logits (last position) → softmax → probability distribution → next token.
Inference never updates weights. Only training/fine-tuning does.
The same forward pass runs during both training and inference — the difference is entirely in what happens after: loss + backprop + update (training) versus simply selecting a token (inference).

19. How This Helps Me Build AI Systems

You have now traced, with real numbers you can inspect directly, the complete mechanism behind every LLM interaction you’ll ever build on top of. Nothing about “how an LLM works” should feel like an unexplainable black box anymore — it’s the concrete pipeline in Section 4, running repeatedly, built entirely from Modules 1-16’s individually-verified components.

Next: Module 18 — Deep Learning in Modern AI Engineering — the final integration: where these concepts live inside a complete AI/Agentic AI system architecture, and the bridge into the Transformers/LLM course proper.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026