Backpropagation – Deep Learning

Begin with the central question

How does one wrong answer tell millions of internal weights which way they should change?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

loss → local derivatives backward through layers → gradient for each parameter

Before you continue: three tools for this module

Derivative: how much an output changes when one input changes a tiny amount.
Gradient: a collection of derivatives, one for each trainable parameter.
Chain rule: multiply local rates of change to trace an effect through several connected calculations.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

This is the must-know module. You’ll understand exactly how a network figures out which direction to adjust every single weight, given only a loss value — by tracing gradients backward through the network using the chain rule. You’ll compute this by hand for a tiny network, and verify it numerically.

Backpropagation applies the chain rule backward through the recorded operations:

forward:  input → layer 1 → layer 2 → loss
backward: parameter gradients ← chain rule ← loss gradient

It computes how sensitive the loss is to each parameter. It does not itself update a weight; the optimizer uses those gradients to choose the update.

Why Every Parameter Needs Credit or Blame

Module 6 gave the network a single number: the loss. That number alone doesn’t say which weights caused the error, or how much each one should change. Backpropagation exists to answer exactly this: given the loss, how much did each individual parameter — potentially millions or billions of them — contribute to it, and in which direction should it move to reduce that loss?

Tracing an Error Back Through the Calculation

the prediction was wrong. Which weights should change, and by how much? Backpropagation answers this by working backward from the error, layer by layer, using the chain rule to figure out each parameter’s individual “share of the blame.”

🧠 Think of it this way: imagine a factory assembly line where the final product came out defective. To find the cause, you don’t just blame the last station — you trace backward: “how much did station 5’s work contribute to the defect? Given that, how much did station 4’s work — which fed into station 5 — contribute?” and so on, back to the start. Backpropagation is exactly this backward blame-tracing, done mathematically.

Analogy: The Corporate Blame Delegation Chain Imagine a product launch fails horribly, causing a massive financial loss (the output loss):

CEO (Output Node): The CEO receives the loss report. The CEO doesn’t write code or pack boxes, but calculates how much of the failure is due to the Operations VP ( $a_2$ ) and how much is due to the Marketing VP ( $a_3$ ) ( $\frac{\partial L}{\partial a_2}$ and $\frac{\partial L}{\partial a_3}$ ).

VP of Operations (Hidden Layer 2 Node): The VP receives a warning letter (gradient) from the CEO. The VP translates this blame by calculating how much of it was caused by the Factory Supervisor ( $a_1$ ) versus their own department’s internal schedules ( $w_2$ ) ( $\frac{\partial L}{\partial w_2} = \frac{\partial L}{\partial a_2} \cdot \frac{\partial a_2}{\partial w_2}$ ).

Factory Supervisor (Hidden Layer 1 Node): The supervisor receives the VP’s warning and passes the blame down to the individual raw material vendors (input weights $w_1$ ) ( $\frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial a_1} \cdot \frac{\partial a_1}{\partial z_1} \cdot \frac{\partial z_1}{\partial w_1}$ ).

This backward propagation of accountability, where each layer multiplies the incoming warning by its own local contribution scale, is exactly how the Chain Rule distributes error signals to every parameter in a neural network.

📊 Visual Flowchart: Backpropagation Gradient Flow (The Chain Rule)

Here is how error gradients propagate backward through the network graph to evaluate parameter adjustments:

graph RL
    Loss["1. Final Loss (L)"] -->|dF/da2| OutputNode["2. Output Layer: a2 = Sigmoid(z2)"]
    OutputNode -->|dL/dz2| OutSum["3. Output Summation Node: z2 = W2*a1 + b2"]

OutSum -->|dL/dW2| W2Grad["4. Weight Gradient:<br>dL/dW2 = dL/dz2 * a1"]
    OutSum -->|dL/db2| B2Grad["5. Bias Gradient:<br>dL/db2 = dL/dz2 * 1"]
    OutSum -->|dL/da1| HiddenNode["6. Hidden Layer: a1 = ReLU(z1)"]

HiddenNode -->|dL/dz1| HidSum["7. Hidden Summation Node: z1 = W1*x + b1"]

HidSum -->|dL/dW1| W1Grad["8. Weight Gradient:<br>dL/dW1 = dL/dz1 * x"]
    HidSum -->|dL/db1| B1Grad["9. Bias Gradient:<br>dL/db1 = dL/dz1 * 1"]

4. Core Concept

This distinction must be extremely clear:

Forward propagation  =  make a prediction        (Module 5)
Loss                 =  measure the error          (Module 6)
Backpropagation       =  determine how each parameter
                          CONTRIBUTED to that error
Gradient descent       =  actually UPDATE the parameters
                          using that information     (Module 8)

Backpropagation computes gradients — it does not, by itself, change any weights. That’s gradient descent’s job, covered next.

The chain rule, conceptually

If a small change in w1 affects z1, which affects a1, which affects z2, which affects the loss — the chain rule says you can compute w1’s total effect on the loss by multiplying together each individual step’s effect:

dLoss/dw1 = (dLoss/dz2) × (dz2/da1) × (da1/dz1) × (dz1/dw1)

Each factor is a local derivative — “how much does this one step change given a small change in its input” — and multiplying them together gives the total effect, all the way back to w1.

5. How It Works — Step by Step

1. FORWARD PASS: compute z1, a1, z2, y_pred, and the loss
   (exactly Modules 5-6, nothing new yet)

2. Start from the LOSS and work BACKWARD:
   a. How much does the loss change per unit change in y_pred?
      -> dLoss/dy_pred
   b. How much does y_pred change per unit change in z2?
      -> dy_pred/dz2  (this depends on the output activation, Module 4)
   c. Multiply: dLoss/dz2 = dLoss/dy_pred × dy_pred/dz2

3. Continue backward through EACH layer, computing:
   - the gradient with respect to that layer's WEIGHTS and BIAS
     (these are what get used to update the parameters)
   - the gradient with respect to that layer's INPUT (needed to
     keep propagating backward into the PREVIOUS layer)

4. Repeat until gradients have been computed for every parameter
   in the network

5. Hand these gradients to the optimizer (Module 8) to actually
   update the weights

6. Mathematical Intuition

First, use only small numbers

Suppose a weight influences a node by 2, and that node influences the loss by 3. The weight’s total influence on the loss is 2 × 3 = 6. Backpropagation repeats this simple multiplication through much larger computation graphs.

Read the mathematics as a story

Backpropagation applies the chain rule from the output toward the input. It assigns each parameter a gradient describing how a small change would affect the loss.

loss → local derivatives backward through layers → gradient for each parameter

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. A tiny, complete network worked entirely by hand: 1 input → 1 hidden neuron (ReLU) → 1 output neuron (linear), trained with MSE loss.

x = 2.0
w1 = 0.5, b1 = 0.1      (hidden neuron)
w2 = 0.8, b2 = -0.2     (output neuron)
y_true = 3.0

FORWARD:
  z1 = w1×x + b1 = 0.5×2.0 + 0.1 = 1.1
  a1 = ReLU(z1) = 1.1              (positive, unchanged)
  z2 = w2×a1 + b2 = 0.8×1.1 - 0.2 = 0.68
  y_pred = z2 = 0.68
  loss = (y_pred - y_true)² = (0.68 - 3.0)² = 5.3824

BACKWARD:
  dLoss/dy_pred = 2×(y_pred - y_true) = 2×(0.68-3.0) = -4.64

y_pred = z2 directly, so dy_pred/dz2 = 1
  dLoss/dz2 = -4.64 × 1 = -4.64

z2 = w2×a1 + b2:
    dLoss/dw2 = dLoss/dz2 × a1 = -4.64 × 1.1 = -5.104
    dLoss/db2 = dLoss/dz2 × 1  = -4.64
    dLoss/da1 = dLoss/dz2 × w2 = -4.64 × 0.8 = -3.712

a1 = ReLU(z1); derivative is 1 since z1=1.1 > 0
  dLoss/dz1 = dLoss/da1 × 1 = -3.712

z1 = w1×x + b1:
    dLoss/dw1 = dLoss/dz1 × x = -3.712 × 2.0 = -7.424
    dLoss/db1 = dLoss/dz1 × 1 = -3.712

Every variable: each dLoss/d(something) is “how much does the total loss change for a tiny change in that specific value” — a gradient. The chain rule lets each gradient be computed as the product of the local derivatives along the path back to it, which is exactly what the step-by-step computation above does.

Numerical verification (the standard way to sanity-check backprop without trusting the calculus alone): nudge w1 by a tiny amount ε, recompute the loss, and check that (loss(w1+ε) − loss(w1−ε)) / (2ε) matches the analytically-computed gradient.

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. If dLoss/dw1 = -7.424 (as computed above), this means increasing w1 slightly would decrease the loss (a negative gradient means “the loss goes down as this parameter goes up”). Gradient descent (Module 8) uses exactly this sign and magnitude to decide how to update w1 — it will push w1 in the direction that reduces the loss.

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Manual forward AND backward pass, verified against a numerical
# gradient check -- the standard way to confirm backprop is correct.

x = 2.0
w1, b1 = 0.5, 0.1
w2, b2 = 0.8, -0.2
y_true = 3.0

# --- Forward pass ---
z1 = w1 * x + b1
a1 = max(0, z1)          # ReLU
z2 = w2 * a1 + b2
y_pred = z2

loss = (y_pred - y_true) ** 2
print(f"z1={z1}, a1={a1}, z2={z2}, y_pred={y_pred}, loss={loss}")

# --- Backward pass (manual chain rule) ---
dL_dypred = 2 * (y_pred - y_true)
print(f"\ndL/dy_pred = {dL_dypred}")

dL_dz2 = dL_dypred * 1     # y_pred = z2 directly
print(f"dL/dz2 = {dL_dz2}")

dL_dw2 = dL_dz2 * a1
dL_db2 = dL_dz2 * 1
dL_da1 = dL_dz2 * w2
print(f"dL/dw2 = {dL_dw2}, dL/db2 = {dL_db2}, dL/da1 = {dL_da1}")

drelu_dz1 = 1.0 if z1 > 0 else 0.0
dL_dz1 = dL_da1 * drelu_dz1
print(f"dL/dz1 = {dL_dz1}")

dL_dw1 = dL_dz1 * x
dL_db1 = dL_dz1 * 1
print(f"dL/dw1 = {dL_dw1}, dL/db1 = {dL_db1}")

# --- Numerical gradient check ---
def forward(w1, b1, w2, b2, x, y_true):
    z1 = w1 * x + b1
    a1 = max(0, z1)
    z2 = w2 * a1 + b2
    return (z2 - y_true) ** 2

eps = 1e-6
numerical_dw1 = (forward(w1+eps, b1, w2, b2, x, y_true) -
                  forward(w1-eps, b1, w2, b2, x, y_true)) / (2*eps)
print(f"\nNumerical check dL/dw1 ~= {numerical_dw1} (analytical was {dL_dw1})")

Expected Output:

z1=1.1, a1=1.1, z2=0.6800000000000002, y_pred=0.6800000000000002, loss=5.3824

dL/dy_pred = -4.64
dL/dz2 = -4.64
dL/dw2 = -5.104, dL/db2 = -4.64, dL/da1 = -3.7119999999999997
dL/dz1 = -3.7119999999999997
dL/dw1 = -7.4239999999999995, dL/db1 = -3.7119999999999997

Numerical check dL/dw1 ~= -7.424000000533226 (analytical was -7.4239999999999995)

9. How It Works

Every value matches Section 6’s hand calculation exactly. The critical line is the last one: the numerical check (computed by nudging w1 slightly and measuring the actual change in loss, with no calculus involved) matches the analytical gradient (computed via the chain rule) to 6+ decimal places.

This is the standard way engineers confirm a backpropagation implementation is correct — and it confirms that the chain-rule “blame-tracing” story in Section 3 produces genuinely correct numbers, not just a plausible-sounding narrative.

🤖 How automatic differentiation frameworks do this: PyTorch and similar frameworks build a computational graph during the forward pass — recording every operation performed. Calling .backward() walks this graph in reverse, applying the chain rule automatically at every recorded operation, computing every parameter’s gradient without you writing the manual dL_dw1 = ... lines by hand. Conceptually:
loss.backward()
    ↓
walk the recorded computational graph backward,
applying the chain rule at each step,
storing the resulting gradient on every parameter
This is precisely what you just did manually above — PyTorch just automates the bookkeeping for networks far too large to trace by hand.

10. Real-World Example

Training an LLM with billions of parameters runs exactly this same process — forward pass, compute loss, backward pass computing a gradient for every single one of those billions of parameters — once per training batch, repeated for potentially trillions of tokens.

The mechanism is identical to Section 8’s 4-parameter example; only the scale (and the use of automatic differentiation instead of manual chain-rule bookkeeping) differs.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

GPT-style and Gemini-family models are trained by computing a loss and propagating gradients backward through many Transformer layers. An application using a hosted model normally performs inference only; backpropagation appears again if the model is fine-tuned or further trained.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

Backpropagation is literally the mechanism by which every neural network — including every LLM — learns anything at all. Without it, there would be no way to know how to adjust a network’s parameters given only a loss value.

Concept	AI application
Chain rule / backpropagation	The core training mechanism for every neural network, including LLM pretraining and fine-tuning
Computational graph	What PyTorch/TensorFlow build automatically to enable `.backward()`
Gradient	What gets computed here, and consumed by the optimizer (Module 8-9)

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Low, directly — most AI/agentic engineers never write backpropagation code by hand; frameworks and pretrained/fine-tuning APIs handle it.

But understanding it precisely explains why fine-tuning has real hyperparameters (learning rate, Module 9) that matter, and why fine-tuning can go wrong in specific, diagnosable ways (Module 10’s vanishing/exploding gradients) — genuinely useful when you’re the one configuring a fine-tuning job.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: backpropagation is the same thing as gradient descent.

Why it is incorrect: They are not. Backpropagation computes gradients — it tells you the direction and magnitude of each parameter’s effect on the loss. Gradient descent (Module 8) is the separate step that actually uses those gradients to update the parameters.

⚠️ Mistake

Incorrect idea: backpropagation requires understanding deep calculus to use in practice.

Why it is incorrect: The chain rule’s mechanics are exactly what you just traced by hand — repeated multiplication of local derivatives. Modern frameworks compute this automatically; you rarely derive it by hand for real networks, but understanding the mechanism (not the notation-heavy calculus) is what matters.

⚠️ Mistake

Incorrect idea: gradients tell you the “correct” new weight value.

Why it is incorrect: They only tell you the direction and relative steepness of the loss with respect to that parameter — the optimizer (Module 8-9) decides how big a step to actually take, using the learning rate.

14. Important Distinctions

Forward Propagation	Backpropagation
Input → output, computing a prediction	Loss → input, computing gradients
Happens during both training and inference	Happens ONLY during training

Backpropagation	Gradient Descent
Computes gradients (how much each parameter contributed to the error)	Uses gradients to actually update parameters
A calculation step	An update step

15. When to Use

Backpropagation isn’t optional or situational — it’s the standard, essentially universal mechanism for training any differentiable neural network. You’ll never manually choose whether to use it; the question in practice is whether your chosen framework’s automatic differentiation is computing it correctly, which is exactly what the numerical gradient check in Section 8 verifies.

16. When Not to Use

Not applicable — this is a core training mechanism, not a technique with meaningful alternatives for standard neural network training.

17. Interview Questions

Beginner

Q: What does backpropagation actually do?

Ans: It computes the gradient of the loss with respect to every parameter in the network — that is, how much (and in which direction) each individual weight and bias contributed to the total error — by working backward from the loss through the network, applying the chain rule at each step.

Intermediate

Q: What’s the difference between backpropagation and gradient descent?

Ans: Backpropagation computes gradients — it determines how much each parameter contributed to the loss. Gradient descent is the separate step that actually uses those gradients to update the parameters, moving each one in the direction that reduces the loss, scaled by the learning rate. Backpropagation calculates; gradient descent acts.

Advanced

Q: Explain, using the chain rule, how a gradient for a weight in an early layer of a network is computed, given the loss is only directly computed at the final layer.

Ans: The chain rule lets you compute an early layer’s weight gradient as the product of local derivatives along the path from that weight to the loss: how the loss changes with respect to the final output, times how that output changes with respect to the previous layer’s activation, times how that activation changes with respect to its pre-activation weighted sum, times how that weighted sum changes with respect to the weight itself.

Each factor is a small, locally-computable derivative; multiplying them together (as demonstrated numerically in Section 6, with the result confirmed via a numerical gradient check) gives the weight’s total effect on the final loss, however many layers away it sits.

Scenario

Q: You implement backpropagation manually for a custom layer and want to confirm it’s correct before trusting it in a larger training run. What would you do?

Ans: I’d perform a numerical gradient check — nudge each parameter by a tiny amount ε, recompute the loss, and estimate the gradient as (loss(param+ε) − loss(param−ε)) / (2ε). Comparing this numerical estimate against my analytically-computed (chain-rule) gradient, as demonstrated in Section 8, is the standard way to catch bugs in a custom backpropagation implementation before trusting it in a real training run.

AI Engineering

Q: When PyTorch code calls loss.backward(), what is actually happening underneath, in terms of what you learned in this module?

Ans: PyTorch has been recording every operation performed during the forward pass into a computational graph.

loss.backward() walks this graph in reverse, applying the chain rule at each recorded operation — exactly the manual process traced by hand in Section 6 — automatically computing and storing a gradient for every parameter involved in producing the loss, without the developer writing that chain-rule bookkeeping by hand.

18. What You Should Remember

Forward propagation = make a prediction. Loss = measure the error. Backpropagation = determine how each parameter contributed to that error. Gradient descent = update the parameters. Four distinct steps, not synonyms for each other.
Backpropagation uses the chain rule — multiplying local derivatives along the path from a parameter to the loss — to compute every parameter’s gradient.
A numerical gradient check (nudge and measure) is the standard way to verify a backpropagation implementation is correct.

19. How This Helps Me Build AI Systems

Every LLM you’ve ever used learned everything it knows through this exact mechanism — repeated, at massive scale, across billions of parameters and enormous amounts of training data. You’ve now computed it by hand and verified it numerically at small scale; nothing categorically different happens when it’s scaled up.

Next: Module 8 — Gradient Descent and the Training Loop — assembling forward pass, loss, backpropagation, and parameter updates into the complete loop every network is actually trained with.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026