Optimizers and Learning Rate

Begin with the central question

Why can two networks with the same data and architecture learn at very different speeds?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

gradients + optimizer state + learning rate → parameter update

Before you continue: three tools for this module

Optimizer: the rule that converts gradients into parameter updates.
Momentum: a running memory of recent gradient directions.
Adaptive step: a parameter-specific update size calculated from previous gradients.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

Why plain gradient descent (Module 8) can be slow or unstable in practice, and how Momentum, Adam, and AdamW address specific weaknesses — with a real, executed comparison, not just a conceptual description.

The optimizer turns gradients into parameter updates:

gradient = local direction and sensitivity
learning rate = basic step-size control
optimizer state = rules using history, momentum, or adaptive scaling

Adam is a common starting point, not a guaranteed winner. Learning rate, scheduling, batch size, model scale, and optimizer settings interact, so they must be evaluated together.

Why Raw Gradients Need an Update Strategy

Module 8’s plain gradient descent update — w -= learning_rate × gradient — works, but has real practical weaknesses: it can be slow in “narrow valley” loss landscapes, and it uses the same step size for every parameter regardless of how that parameter’s gradient has been behaving recently. Optimizers exist to refine the basic update rule to train faster and more reliably.

Choosing the Size and Direction of Each Step

plain gradient descent is like walking downhill by only ever looking at the ground right under your feet, taking the same size step every time. Momentum is like a ball rolling downhill — it builds up speed in a consistently downhill direction, and doesn’t immediately stop or reverse due to small bumps. Adam goes further: it gives each individual parameter its own adaptive step size, based on that parameter’s own recent gradient history.

Analogy: The Step Size Strider & The Heavy Sled with Momentum Imagine navigating a hilly landscape with various slopes:

Learning Rate (Step size):

Too Large (Giant Strides): You are taking huge 10-foot leaps. You leap right over the narrow, deep valley bottom and land on the opposite mountainside, eventually bouncing higher and higher (divergence / overshooting).

Too Small (Tiny Steps): You take 1-millimeter steps. It will take you weeks to move down a single slope, and you will get stuck in the first tiny puddle or footprint you step into (slow convergence / local minima).

Momentum (The Heavy Sled): Imagine sliding down a steep snowy hill on a heavy wooden sled. As you slide, the sled builds up velocity. When you hit a small upward bump or a flat, snow-filled dip (a local minimum or saddle point), the sled’s accumulated momentum carries you right over it, letting you continue sliding toward the true bottom of the mountain.

Adam (Adaptive step size per parameter): Imagine walking on terrain where one foot is on slippery ice and the other is in deep mud. Instead of forcing the same step size on both legs, you adaptively shorten the stride of the slippery leg to avoid falling, while taking long, strong strides with the muddy leg to pull yourself out. Adam dynamically scales the step size for each parameter independently based on how erratic or steady its historical gradients are.

📊 Visual Flowchart: Learning Rate Landscape Optimization Outcomes

Here is how the learning rate coefficient affects the model’s trajectory across the loss surface:

graph TD
    classDef optimal stroke:#2ecc71,stroke-width:2px;
    classDef high stroke:#e74c3c,stroke-width:2px;
    classDef low stroke:#f39c12,stroke-width:2px;

Start["Configure Learning Rate (lr)"] --> CheckRate{"Rate setting value?"}

CheckRate -->|Too High: lr = 1.0| High["1. Overshooting / Diverging"]:::high
    CheckRate -->|Too Low: lr = 1e-6| Low["2. Slow Convergence / Local Minima Stuck"]:::low
    CheckRate -->|Optimal: lr = 3e-4| Opt["3. Smooth Convergence to global minimum"]:::optimal

High --> EffectHigh["Loss oscillates wildly or explodes to NaN"]
    Low --> EffectLow["Training takes too long; fails to find global minimum"]
    Opt --> EffectOpt["Reaches optimal loss value in minimum steps"]

4. Core Concept

Optimizer	Core idea
SGD (plain)	`w -= lr × gradient` — the raw update from Module 8
Momentum	Accumulates a “velocity” from recent gradients, smoothing updates and building speed in a consistent direction
Adam	Combines momentum-like behavior with an adaptive, per-parameter learning rate
AdamW	Adam, with weight decay (Module 11) handled correctly and separately — the modern default for training Transformers

Momentum, precisely

velocity = β × velocity + (1 − β) × gradient
w = w − learning_rate × velocity

β (commonly 0.9) controls how much of the previous velocity carries forward — high β means more “smoothing” and more resistance to sudden direction changes.

Adam, precisely (conceptually)

Adam tracks two running statistics per parameter: a momentum-like average of recent gradients (m), and an average of recent squared gradients (v, tracking how large the gradient has typically been, regardless of sign). It then scales the update by m / sqrt(v) — giving parameters with consistently large gradients smaller effective steps, and parameters with consistently small gradients larger effective steps.

5. How It Works — Step by Step

1. Compute the gradient (backpropagation, Module 7) -- same for
   every optimizer
2. THE OPTIMIZER DECIDES HOW TO USE THAT GRADIENT:
   - Plain SGD: use it directly, scaled by the learning rate
   - Momentum: blend it with an accumulated velocity from past
     gradients, THEN scale by the learning rate
   - Adam: track both a momentum-like average AND a measure of
     recent gradient magnitude, and use both to compute an
     ADAPTIVE, per-parameter step
3. Update every parameter using the optimizer's computed step
4. Repeat

6. Mathematical Intuition

First, use only small numbers

A learning rate of 1 might jump past the lowest point, while 0.000001 may move so slowly that training appears stuck. An optimizer tries to make useful progress without letting noisy or unusually large gradients control every step.

Read the mathematics as a story

The gradient provides direction; the learning rate controls step size; the optimizer decides how to combine current and past gradient information into a practical update.

gradients + optimizer state + learning rate → parameter update

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. Momentum’s velocity update, worked for one step (starting from velocity=0, β=0.9, first gradient dw=-7.424):

velocity = 0.9 × 0 + 0.1 × (−7.424) = −0.7424
w update = w − learning_rate × (−0.7424)

If the next gradient is also negative (consistently downhill in the same direction), velocity accumulates further in that direction — larger effective steps than plain SGD would take on its own, exactly the “rolling ball builds speed” intuition.

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. Imagine a loss landscape shaped like a narrow, elongated valley — plain gradient descent tends to zig-zag back and forth across the valley’s narrow width while making slow progress along its length.

Momentum dampens the zig-zagging (since it partially cancels out across oscillating directions) while reinforcing progress along the consistent downhill direction — this specific scenario is where momentum’s benefit is most visible; on simpler, smoother loss landscapes the difference can be much less dramatic, as the comparison below shows honestly.

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of Optimizers and Learning Rate.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

np.random.seed(0)
X = np.linspace(0, 10, 20)
y_true = 3 * X + 2 + np.random.randn(20) * 0.5

def train_sgd(lr=0.01, epochs=100):
    w, b = 0.0, 0.0
    losses = []
    for _ in range(epochs):
        y_pred = w * X + b
        loss = np.mean((y_pred - y_true) ** 2)
        losses.append(loss)
        dw = np.mean(2 * (y_pred - y_true) * X)
        db = np.mean(2 * (y_pred - y_true))
        w -= lr * dw
        b -= lr * db
    return w, b, losses

def train_momentum(lr=0.01, epochs=100, beta=0.9):
    w, b = 0.0, 0.0
    vw, vb = 0.0, 0.0
    losses = []
    for _ in range(epochs):
        y_pred = w * X + b
        loss = np.mean((y_pred - y_true) ** 2)
        losses.append(loss)
        dw = np.mean(2 * (y_pred - y_true) * X)
        db = np.mean(2 * (y_pred - y_true))
        vw = beta * vw + (1 - beta) * dw
        vb = beta * vb + (1 - beta) * db
        w -= lr * vw
        b -= lr * vb
    return w, b, losses

def train_adam(lr=0.1, epochs=100, beta1=0.9, beta2=0.999, eps=1e-8):
    w, b = 0.0, 0.0
    mw, mb, vw, vb = 0.0, 0.0, 0.0, 0.0
    losses = []
    for t in range(1, epochs + 1):
        y_pred = w * X + b
        loss = np.mean((y_pred - y_true) ** 2)
        losses.append(loss)
        dw = np.mean(2 * (y_pred - y_true) * X)
        db = np.mean(2 * (y_pred - y_true))

        mw = beta1 * mw + (1 - beta1) * dw
        mb = beta1 * mb + (1 - beta1) * db
        vw = beta2 * vw + (1 - beta2) * dw**2
        vb = beta2 * vb + (1 - beta2) * db**2

        # Bias correction -- accounts for m/v starting at zero
        mw_hat = mw / (1 - beta1**t)
        mb_hat = mb / (1 - beta1**t)
        vw_hat = vw / (1 - beta2**t)
        vb_hat = vb / (1 - beta2**t)

        w -= lr * mw_hat / (np.sqrt(vw_hat) + eps)
        b -= lr * mb_hat / (np.sqrt(vb_hat) + eps)
    return w, b, losses

w_sgd, b_sgd, loss_sgd = train_sgd(lr=0.01, epochs=100)
w_mom, b_mom, loss_mom = train_momentum(lr=0.01, epochs=100)
w_adam, b_adam, loss_adam = train_adam(lr=0.1, epochs=100)

print(f"Plain SGD  (100 epochs): w={w_sgd:.4f}, b={b_sgd:.4f}, final_loss={loss_sgd[-1]:.4f}")
print(f"Momentum   (100 epochs): w={w_mom:.4f}, b={b_mom:.4f}, final_loss={loss_mom[-1]:.4f}")
print(f"Adam       (100 epochs): w={w_adam:.4f}, b={b_adam:.4f}, final_loss={loss_adam[-1]:.4f}")

print(f"\nLoss after 10 epochs -- SGD: {loss_sgd[9]:.4f}, Momentum: {loss_mom[9]:.4f}, Adam: {loss_adam[9]:.4f}")

Expected Output:

Plain SGD  (100 epochs): w=3.1246, b=1.3369, final_loss=0.5640
Momentum   (100 epochs): w=3.1285, b=1.3035, final_loss=0.5880
Adam       (100 epochs): w=2.8877, b=2.8060, final_loss=0.1835

Loss after 10 epochs -- SGD: 1.2205, Momentum: 72.7856, Adam: 181.7981

9. How It Works — an honest reading of these numbers

After 100 epochs, Adam reaches the lowest final loss (0.18, versus ~0.56-0.59 for SGD/Momentum) — Adam’s adaptive per-parameter step sizing genuinely helps here.
Interestingly, at epoch 10, both Momentum and Adam show higher loss than plain SGD. This is real, expected behavior, not a bug: Momentum’s accumulated velocity and Adam’s higher learning rate (0.1 vs. 0.01) cause larger, temporarily-overshooting steps early in training, before settling into faster overall convergence. This is precisely why comparing optimizers only by early-training behavior can be misleading — what matters is where each converges given enough steps.
On this particular smooth, simple loss landscape, Momentum’s improvement over plain SGD is modest (0.588 vs. 0.564 — actually slightly worse here). This is honest, not manufactured: momentum’s clearest advantage shows up on harder, more elongated (“valley-shaped”) loss landscapes than this simple 2-parameter linear regression problem — a useful, realistic caveat rather than a manufactured clean win.

10. Real-World Example

Virtually every modern Transformer-based model (including LLMs) is trained with AdamW, not plain SGD or even plain Adam — the “W” specifically fixes a subtle interaction between Adam’s adaptive updates and weight decay regularization (Module 11), which matters more at the scale and duration of real LLM training than it does in small examples like this module’s.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

Large-model training commonly uses adaptive optimizers from the Adam family together with learning-rate schedules, warm-up, gradient clipping, and distributed training. The exact configuration is a training decision, not something changed by an ordinary user prompt.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

The optimizer setting matters directly and practically the moment you fine-tune any model. AdamW with a carefully-chosen learning rate is the standard, default choice for fine-tuning LLMs — getting the learning rate wrong (too high) is one of the most common, real causes of unstable or degraded fine-tuning runs.

Concept	AI application
Adam / AdamW	The standard optimizer for training and fine-tuning Transformers and LLMs
Learning rate	A genuine, practical hyperparameter you configure when fine-tuning
Learning-rate schedules	Real fine-tuning jobs typically reduce the learning rate over training (a “schedule”), rather than using one fixed value throughout

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Moderate. Whenever you fine-tune a model for a specific agent capability — an intent classifier, or the agent’s core LLM itself — you’ll directly choose an optimizer (often AdamW for Transformer fine-tuning) and a learning rate. Understanding why these choices matter, demonstrated concretely above, is what separates blindly copying a tutorial’s hyperparameters from making an informed decision.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: Adam always trains faster than SGD, in every case.

Why it is incorrect: As Section 9 shows honestly, Adam’s early-training loss was actually higher here — the real benefit is in where training eventually converges, not necessarily every intermediate step.

⚠️ Mistake

Incorrect idea: a fancier optimizer fixes a bad learning rate.

Why it is incorrect: It doesn’t. All the optimizers in this module still have a learning rate hyperparameter (or an equivalent), and getting it badly wrong (Module 8) still causes the same instability or slow-convergence symptoms.

⚠️ Mistake

Incorrect idea: Momentum always beats plain SGD.

Why it is incorrect: Section 9’s results explicitly show this isn’t guaranteed on every loss landscape — momentum’s benefit is most pronounced on landscapes with the “narrow valley” shape described in Section 7.

14. Important Distinctions

SGD	Momentum
Uses only the current gradient	Blends current gradient with accumulated past gradients
Simple, predictable	Smoother updates, can build speed in a consistent direction

Momentum	Adam
One accumulated “velocity,” shared step size across parameters	Separate adaptive step size PER parameter
Simpler, fewer moving parts	More sophisticated, generally faster convergence, more commonly used in practice

Adam	AdamW
Combines weight decay directly into the adaptive update	Applies weight decay separately, more correctly
Older, still common	The modern default for training Transformers

15. When to Use

Use AdamW as the default choice for training or fine-tuning neural networks, especially Transformers/LLMs — it’s the current, near-universal standard for good reason. Use a learning-rate schedule (starting higher, decreasing over training) for longer training runs.

16. When Not to Use

Plain SGD (without momentum) is rarely the best choice for deep networks in practice, though it remains simple, predictable, and sometimes used deliberately for its different generalization properties in some research contexts — not a concern for most practical AI engineering work, where AdamW is the safe, standard default.

17. Interview Questions

Beginner

Q: Why might plain SGD be slow to train in practice?

Ans: Plain SGD uses only the current gradient with a fixed step size for every parameter — it can zig-zag inefficiently on loss landscapes shaped like narrow valleys, and doesn’t adapt its step size based on how a parameter’s gradient has behaved recently, which can make convergence slower than necessary.

Intermediate

Q: How does Momentum improve on plain gradient descent?

Ans: Momentum accumulates a running “velocity” from recent gradients rather than reacting only to the current one — this smooths out updates that oscillate in inconsistent directions, while building up speed in directions the gradient has consistently pointed toward, generally leading to faster, more stable convergence, especially on harder loss landscapes.

Advanced

Q: Why is AdamW preferred over plain Adam for training modern Transformers?

Ans: Plain Adam applies weight decay (a regularization technique, Module 11) in a way that interacts awkwardly with its adaptive per-parameter learning rates — the effective amount of regularization ends up inconsistent across parameters.

AdamW decouples weight decay from the adaptive gradient update, applying it more directly and consistently — an empirically meaningful improvement specifically at the scale and training duration of modern Transformer/LLM training, which is why AdamW has become the standard.

Scenario

Q: You compare Adam against SGD on a new problem, and after only 10 training steps, SGD has lower loss. Should you conclude SGD is the better optimizer for this problem?

Ans: Not necessarily — as demonstrated in Section 9, Adam’s early loss can genuinely be higher due to larger initial steps and bias-correction effects, even when it ultimately converges to a better final result. I’d compare both optimizers’ loss curves over a much longer training run, not just the first several steps, before drawing a conclusion about which is actually better for this specific problem.

AI Engineering

Q: You’re fine-tuning an LLM and need to choose an optimizer and learning rate. What would you default to, and why?

Ans: AdamW is the standard, well-established default for fine-tuning Transformer-based models — it’s what the vast majority of fine-tuning frameworks and tutorials use, for good empirical reasons specific to this model family.

For the learning rate, I’d start from whatever value is recommended for the specific fine-tuning method and model size (often found in the framework’s documentation or established community practice), since fine-tuning typically needs a much smaller learning rate than pretraining used, to avoid overwriting the model’s existing knowledge too aggressively (Module 10 covers this failure mode directly).

18. What You Should Remember

Momentum smooths updates and builds speed in a consistent direction — most valuable on harder, elongated loss landscapes.
Adam gives every parameter its own adaptive step size, generally converging faster and more reliably than plain SGD — though not necessarily faster in every single early step, as this module’s real numbers showed.
AdamW is the modern, near-universal default for training and fine-tuning Transformers and LLMs.

19. How This Helps Me Build AI Systems

The moment you fine-tune any model, you’ll choose an optimizer (almost always AdamW) and a learning rate — and you now understand, from a real executed comparison rather than just a description, what these choices actually do to training dynamics and why getting them wrong produces specific, diagnosable symptoms.

Next: Module 10 — Training Deep Networks — vanishing/exploding gradients, initialization, and gradient clipping: the practical problems that show up specifically once networks get genuinely deep.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026