TechByteByByte

Gradient Descent and the Training Loop

Assemble forward propagation, loss, backpropagation, and parameter updates into the complete training loop — batch, stochastic, and mini-batch gradient descent, epochs, iterations, and batches, all in one working example.

#Deep Learning#Neural Networks#AI#Gradient Descent#Training Loop

Begin with the central question

Once the network knows which way is downhill, how does it actually learn?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

batch → prediction → loss → gradients → parameter update → repeat

Before you continue: three tools for this module

  • Gradient: the direction in which loss increases fastest.
  • Learning rate: the size of the parameter-update step.
  • Batch: a small group of training examples processed together.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

How Modules 5-7 assemble into the actual loop every neural network is trained with: forward pass → loss → backpropagation → optimizer update → repeat. You’ll also cover epoch, batch, and iteration precisely, and the practical differences between batch, stochastic, and mini-batch gradient descent — verified with a real, working training loop.

Training repeats a feedback loop:

batch → forward pass → loss → backward pass → optimizer step
  ↑                                                ↓
  └──────────── repeat for more batches ──────────┘

An epoch means one pass through the training dataset, usually divided into batches. A lower training loss is not enough; validation checks whether the learned behaviour transfers to unseen examples.

Why Learning Requires a Repeated Feedback Loop

Module 7 computes gradients. A gradient alone doesn’t change anything — something has to actually use it to update the weights, repeatedly, until the network’s predictions genuinely improve. Gradient descent is that mechanism, and the training loop is the complete, repeated cycle that turns a randomly-initialized network into a trained one.

Practise, Measure, Correct, Repeat

imagine trying to find the lowest point in a hilly, foggy landscape, able to see only the ground immediately at your feet. A sensible strategy: feel which direction slopes downward, take a small step that way, then repeat from your new position. Gradient descent is exactly this — the gradient tells you the downhill direction; the learning rate controls your step size; repeating the process gradually walks the parameters toward lower loss.

Analogy: The Blind Hiker in the Foggy Valley Imagine you are a blindfolded hiker dropped onto a mountain peak (the high initial loss landscape) surrounded by dense fog. Your mission is to find the lowest base camp at the bottom of the valley (the minimum loss):

  • Gradient (The Slope): You stand in place and tap the ground with your walking stick in all directions. You feel which direction slopes downward the steepest. That slope direction is the negative gradient.
  • Learning Rate (Step Size): The size of the step you take. If your step size is too large (like jumping 10 yards blindly), you might leap right over the valley and crash into the opposite peak (overshooting / divergence). If your step size is too small (like sliding forward 1 millimeter), it will take you a million steps to reach the bottom (slow training).
  • Parameter Update (Taking the Step): You take one step in the downhill direction.
  • The Training Loop (Epochs): You stand at your new position, tap the ground again (calculate fresh gradients on the next batch of data), and take another step. You repeat this loop until the ground feels flat in all directions, indicating you have reached the valley floor (convergence).

📊 Visual Flowchart: The Complete Neural Network Training Loop

Here is the cyclic pipeline executed repeatedly during every epoch of network training:

graph TD
    Data["Get Training Batch (Inputs x, Labels y_true)"] --> Forward["1. Forward Propagation (Module 5)<br>y_pred = Model(x)"]
    Forward --> Loss["2. Calculate Loss (Module 6)<br>Loss = Cost(y_pred, y_true)"]
    Loss --> Backward["3. Backpropagation (Module 7)<br>Compute Gradients (dL/dW, dL/db)"]

Backward --> Update["4. Parameter Updates (Optimizer)<br>W = W - lr * (dL/dW)<br>b = b - lr * (dL/db)"]

Update --> CheckStop{"All batches / epochs<br>completed?"}

CheckStop -->|No| Data
    CheckStop -->|Yes| Finish["Training Complete: Save weights"]

4. Core Concept

The complete loop

Input batch
  ↓
Forward pass          (Module 5)
  ↓
Prediction
  ↓
Loss                   (Module 6)
  ↓
Backpropagation         (Module 7) — computes GRADIENTS
  ↓
Gradients
  ↓
Optimizer               (this module + Module 9) — UPDATES parameters
  ↓
Parameter update
  ↓
Repeat

Gradient vs. gradient descent vs. optimizer — explicitly

Gradient:          a NUMBER (per parameter) — how much and in which
                    direction the loss changes if that parameter
                    changes slightly. Computed BY backpropagation.

Gradient descent:   the GENERAL STRATEGY of repeatedly moving
                    parameters in the direction that reduces loss,
                    using gradients.

Optimizer:          the SPECIFIC ALGORITHM that implements gradient
                    descent (or a refinement of it — Module 9 covers
                    Momentum, Adam, AdamW) — it decides exactly HOW
                    to use the gradient to update each parameter.

The gradient descent update rule

new_parameter = old_parameter − (learning_rate × gradient)

Epoch, batch, iteration — precisely

TermDefinition
SampleOne single training example
BatchA group of samples processed together before one parameter update
Batch sizeHow many samples are in one batch
Iteration / stepOne single parameter update (processing one batch)
EpochOne complete pass through the entire training dataset

If you have 1,000 samples and a batch size of 100, one epoch consists of 10 iterations (1,000 ÷ 100).

5. How It Works — Step by Step

1. Split training data into batches
2. FOR each epoch:
3.   FOR each batch:
4.     Forward pass on this batch          -> predictions
5.     Compute loss                          -> a single number
6.     Backpropagation                        -> gradients for every parameter
7.     Optimizer updates every parameter      -> using the gradients
8.   (one epoch complete once every batch has been processed)
9. Repeat for many epochs, until loss stops meaningfully improving

Batch, Stochastic, and Mini-Batch Gradient Descent

Batch Gradient Descent:       compute the gradient using the ENTIRE
                               dataset before each update (accurate,
                               but slow per update, memory-heavy)

Stochastic Gradient Descent    compute the gradient using just ONE
(SGD):                        random example per update (fast per
                               step, but noisy)

Mini-Batch Gradient Descent:  compute the gradient using a small
                               BATCH (e.g., 32-256 examples) per
                               update — the practical, near-universal
                               standard in real training

6. Mathematical Intuition

First, use only small numbers

If a weight is 5, its gradient is +2, and the learning rate is 0.1, gradient descent updates it to 5 − (0.1 × 2) = 4.8. The positive gradient said that increasing the weight raises loss, so training moved it downward.

Read the mathematics as a story

Gradient descent moves parameters opposite their gradients. The training loop repeats this measurement-and-correction cycle across batches until improvement slows or validation quality stops improving.

batch → prediction → loss → gradients → parameter update → repeat

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. The update rule, worked for one parameter: if a weight w = 0.5 and its gradient is dw = -7.424 (Module 7’s example), with a learning rate of 0.01:

new_w = 0.5 − (0.01 × −7.424)
      = 0.5 + 0.07424
      = 0.57424

Every variable: w is the current parameter value; dw is its gradient (negative here, meaning increasing w would decrease the loss); learning_rate scales how big a step to take. The minus sign in the update rule is what makes this “descent” — moving against the gradient’s direction, toward lower loss.

7. Simple Example

Walk through the example

Read the example in three passes:

  1. Identify the input numbers and what each number represents.
  2. Follow one operation at a time instead of jumping directly to the answer.
  3. Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. Training a tiny linear model y = w×x + b to fit noisy data approximating y = 3x + 2: starting from w=0, b=0, each training step computes predictions across a batch, measures loss, computes gradients via backpropagation, and nudges w and b slightly closer to 3 and 2.

Repeated across enough epochs, w and b converge close to the true underlying relationship — demonstrated concretely below.

8. Python Example

Three Python symbols used below

  • NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
  • np.array(...) creates a numeric vector or matrix.
  • @ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of Gradient Descent and the Training Loop.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

np.random.seed(0)

# y = w*x + b, true relationship: y = 3x + 2 (plus noise)
X = np.linspace(0, 10, 20)
y_true = 3 * X + 2 + np.random.randn(20) * 0.5

w, b = 0.0, 0.0
learning_rate = 0.01
epochs = 200

loss_history = []
for epoch in range(epochs):
    # Forward pass
    y_pred = w * X + b
    loss = np.mean((y_pred - y_true) ** 2)
    loss_history.append(loss)

# Backward pass (gradients of MSE w.r.t. w and b)
    dw = np.mean(2 * (y_pred - y_true) * X)
    db = np.mean(2 * (y_pred - y_true))

# Parameter update (gradient descent step)
    w -= learning_rate * dw
    b -= learning_rate * db

print(f"Final w={w:.4f}, b={b:.4f}  (true: w=3, b=2)")
print(f"Loss at epoch 1: {loss_history[0]:.4f}")
print(f"Loss at epoch 50: {loss_history[49]:.4f}")
print(f"Loss at epoch 200: {loss_history[199]:.4f}")

Expected Output:

Final w=3.0501, b=1.8427  (true: w=3, b=2)
Loss at epoch 1: 378.6928
Loss at epoch 50: 0.8517
Loss at epoch 200: 0.2945

Now comparing batch, mini-batch, and stochastic gradient descent on the exact same data:

# Build a tiny, inspectable example of Gradient Descent and the Training Loop.
# Follow the intermediate values; they reveal what the model is doing.
def train(batch_size, epochs=50, lr=0.01, seed=0):
    rng = np.random.RandomState(seed)
    w, b = 0.0, 0.0
    n = len(X)
    for epoch in range(epochs):
        indices = rng.permutation(n)
        for start in range(0, n, batch_size):
            batch_idx = indices[start:start+batch_size]
            xb, yb = X[batch_idx], y_true[batch_idx]
            y_pred = w * xb + b
            dw = np.mean(2 * (y_pred - yb) * xb)
            db = np.mean(2 * (y_pred - yb))
            w -= lr * dw
            b -= lr * db
    return w, b

w_batch, b_batch = train(batch_size=20)   # full batch (all 20 samples)
w_mini, b_mini = train(batch_size=4)      # mini-batch
w_sgd, b_sgd = train(batch_size=1)        # stochastic

print(f"\nBatch GD:      w={w_batch:.4f}, b={b_batch:.4f}")
print(f"Mini-batch GD: w={w_mini:.4f}, b={b_mini:.4f}")
print(f"SGD:           w={w_sgd:.4f}, b={b_sgd:.4f}")

Expected Output:

Batch GD:      w=3.1795, b=0.9642
Mini-batch GD: w=3.0441, b=1.9986
SGD:           w=2.9579, b=2.5498

9. How It Works

  • The full training loop steadily reduces loss from 378.69 (epoch 1) to 0.29 (epoch 200), and w/b converge close to the true 3/2 — concrete proof the loop (forward → loss → backward → update → repeat) genuinely learns the underlying pattern.
  • Comparing the three variants after the same 50 epochs: mini-batch (w=3.04, b=2.00) lands closest to the true values, batch GD (w=3.18, b=0.96) hasn’t yet fully converged on b (fewer total updates per epoch — only 1 update per epoch, since the whole dataset is one batch), and SGD (w=2.96, b=2.55) is noisier — the classic trade-offs described in Section 4, now visible in real numbers rather than just asserted.

10. Real-World Example

Training a modern LLM uses mini-batch gradient descent at enormous scale: batches of thousands of token sequences, processed in parallel across many GPUs, with one optimizer update (“step”) per batch.

A full “epoch” over an LLM’s training corpus may not even complete once — many LLMs are trained on a large corpus for less than one full pass, given how enormous the data is, unlike Section 8’s tiny 20-sample dataset trained for 200 full epochs.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

Large language models repeat the same training loop across enormous batches: forward pass, loss, backward pass, optimizer update. Production agent calls do not run this loop; they use the already-trained parameters during inference.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

This exact loop — forward, loss, backward, update, repeat — is the entire training mechanism behind every neural network you’ll ever hear about, including every LLM. “Training a model” and “running this loop, many times, on a lot of data” are the same statement.

ConceptAI application
Mini-batch gradient descentThe near-universal standard for training neural networks and LLMs
EpochSometimes not even fully completed once during LLM pretraining, given corpus size
Batch sizeA key setting when fine-tuning an LLM — affects both training stability and GPU memory usage
Iteration/stepWhat “training step” refers to in LLM training logs and progress bars

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Moderate, specifically when fine-tuning any model used inside an agent pipeline (a routing classifier, or the agent’s core LLM). Batch size and learning rate (Module 9) are genuine, practical settings you’ll configure — not abstract theory — and understanding this loop is what lets you reason about why a fine-tuning job’s loss curve looks the way it does.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: “gradient,” “gradient descent,” and “optimizer” are interchangeable terms.

Why it is incorrect: As Section 4 makes explicit: a gradient is a number; gradient descent is the general strategy; the optimizer is the specific algorithm implementing that strategy (plain gradient descent, or a refinement like Adam — Module 9).

⚠️ Mistake

Incorrect idea: more epochs always means a better model.

Why it is incorrect: Training for too many epochs risks overfitting (Module 11) — the loss curve needs to be watched, not blindly maximized in duration.

⚠️ Mistake

Incorrect idea: batch gradient descent is strictly “better” because it’s more accurate per step.

Why it is incorrect: As Section 9 shows, batch GD takes fewer total update steps for the same amount of data seen (only one update per epoch here) — mini-batch’s more frequent, slightly noisier updates often converge faster in practice, and are far more memory-practical at real scale.

14. Important Distinctions

GradientGradient DescentOptimizer
A number: loss’s sensitivity to one parameterThe general strategy of using gradients to reduce lossThe specific algorithm implementing that strategy
Computed by backpropagationA concept/approachSGD, Adam, AdamW, etc. (Module 9)
EpochIteration / StepBatch
One full pass through the ENTIRE datasetOne single parameter updateThe group of samples processed in one iteration

15. When to Use

Use mini-batch gradient descent as the default for essentially all real neural network training — it balances gradient accuracy, update frequency, and memory/compute practicality better than either pure batch or pure stochastic gradient descent.

16. When Not to Use

Pure batch gradient descent is rarely practical once a dataset is large enough to matter — as demonstrated, it also makes fewer total updates per epoch, which can slow convergence in wall-clock terms even though each update is more “accurate.” Pure SGD (batch size 1) is rarely used directly in modern practice either — its noisiness (visible in Section 9’s w_sgd/b_sgd numbers) usually isn’t worth its marginal simplicity.

17. Interview Questions

Beginner

Q: What is the difference between an epoch, a batch, and an iteration?

Ans: An epoch is one complete pass through the entire training dataset. A batch is a group of training samples processed together before one parameter update. An iteration (or step) is one single parameter update — processing one batch. If a dataset has 1,000 samples and batch size is 100, one epoch consists of 10 iterations.

Intermediate

Q: What’s the difference between a gradient and an optimizer?

Ans: A gradient is a number (computed by backpropagation, per parameter) representing how the loss changes with respect to that parameter. Gradient descent is the general strategy of using gradients to reduce loss.

An optimizer is the specific algorithm that implements this strategy — deciding exactly how to translate a gradient into a parameter update (plain gradient descent uses the raw gradient directly; more advanced optimizers like Adam, Module 9, use additional information).

Advanced

Q: Why does mini-batch gradient descent generally train faster in wall-clock time than full batch gradient descent, even though each individual update uses a less accurate (noisier) gradient estimate?

Ans: Batch gradient descent only updates parameters once per full pass through the dataset — for a large dataset, this means very few total updates over a given amount of training time.

Mini-batch gradient descent makes many more updates per epoch (one per batch, not one per full dataset), and even though each individual gradient estimate is noisier, the much higher update frequency generally leads to faster overall convergence — demonstrated concretely in Section 9, where mini-batch converged closer to the true parameters than full batch GD within the same 50 epochs.

Scenario

Q: You’re training a model and notice the loss decreases very slowly, even after many epochs. What would you investigate, connecting to this module’s concepts?

Ans: I’d first check the learning rate — too small a learning rate produces exactly this symptom (slow, steady but sluggish improvement). I’d also check batch size — a very large batch size means fewer total updates per epoch (as Section 9 demonstrated with full batch GD), which can slow convergence even with a reasonable learning rate.

I’d plot the loss curve directly rather than guessing, and consider Module 9’s more advanced optimizers (like Adam), which often converge meaningfully faster than plain gradient descent.

AI Engineering

Q: When fine-tuning an LLM, why does batch size matter beyond just training speed?

Ans: Batch size affects gradient estimate quality (larger batches give smoother, less noisy gradients — visible in Section 9’s more erratic SGD numbers versus mini-batch), GPU memory usage (larger batches need more memory to hold all the batch’s activations simultaneously), and training stability more broadly. It’s a genuine, practical fine-tuning hyperparameter to configure thoughtfully, not an arbitrary setting.

18. What You Should Remember

  • The full training loop: forward pass → loss → backpropagation → optimizer update → repeat.
  • Gradient (a number) ≠ gradient descent (a strategy) ≠ optimizer (the specific algorithm) — three distinct concepts.
  • Mini-batch gradient descent is the practical, near-universal standard — balancing accuracy, update frequency, and memory practicality.

19. How This Helps Me Build AI Systems

You’ve now watched a real model’s parameters converge from (0, 0) toward the true underlying relationship, purely by repeating this loop — and you understand precisely why batch size and epoch count are settings worth reasoning about carefully, not just leaving at a framework’s default, whenever you configure a fine-tuning job.

Next: Module 9 — Optimizers and Learning Rate — why plain gradient descent can be slow, and how Momentum, Adam, and AdamW improve on it.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed