The Gradient Knows the Direction. But How Far Should the Model Move?

Following one neural network as it takes a learning step

Before running the snippets: The NumPy update fragment continues the complete program in the backpropagation article; it is not standalone. The PyTorch fragments assume you have imported torch and defined model, inputs, targets, and loss_fn. For integer-class targets, torch.nn.CrossEntropyLoss() takes raw logits, not probabilities. The learning rate scales the gradient; it is not the actual distance moved. In the simple bowl example the stated stability thresholds apply specifically to that quadratic, not every neural network.

This is the third article in our connected journey:

Forward pass → Backpropagation → Gradient descent

We already made a prediction and calculated responsibility for the error. Now we turn that responsibility into an actual parameter update.

In the previous article, our tiny neural network tried to complete:

The sky is …

It assigned the correct word, blue, a probability of only 45.73%.

The loss was:

$$0.782$$

Backpropagation then traced that loss backward through the network. Every weight received a gradient—a number describing how a tiny change in that weight would affect the loss.

For one connection leading toward blue, we had:

current weight =  0.4000
gradient       = -0.6512

That gradient tells us something useful:

• the negative sign gives us a direction;
• the magnitude 0.6512 tells us the local sensitivity.

But it leaves one important question unanswered:

How much should we actually change the weight?

Should 0.4000 become 0.4001? Should it become 0.4651? Or should it jump all the way to 1.0512?

This is the problem solved by gradient descent and its modern relatives.

Backpropagation calculates the gradients. The optimizer uses them to update the parameters.

flowchart LR
    Loss["Loss"] --> Backprop["Backpropagation"]
    Backprop --> Gradient["Gradient"]
    Gradient --> Optimizer["Optimizer"]
    Optimizer --> Weights["Updated weights"]

This article follows that last part slowly: from a gradient to an actual step.

---

Imagine standing on a dark hillside

Imagine being placed somewhere on a hill at night.

You want to reach the lowest point, but you cannot see the entire landscape. You can only feel the ground immediately beneath your feet.

The slope tells you which direction goes upward. To move downward, you step in the opposite direction.

This gives us the central idea of gradient descent:

Measure the local uphill direction, then take a small step downhill.

In this analogy:

Hillside idea	Neural-network idea
Your current location	Current parameter values
Height	Loss
Local slope	Gradient
Step size	Learning rate
Lowest reachable area	Parameters with low loss

The analogy is useful, but a real neural network is stranger than an ordinary hill. It may have millions or billions of adjustable coordinates. We cannot draw that landscape, but the mathematical idea remains the same.

---

First, let us create a loss we can see

Before returning to our network, consider a model with only one parameter, $w$. Suppose its loss is:

$$L(w)=(w-3)^2$$

Try a few values:

$w$	$L(w)=(w-3)^2$
0	9
1	4
2	1
3	0
4	1
5	4
6	9

The loss is smallest when $w=3$.

The graph would look like a bowl. If we start at $w=0$, we want to move right. If we start at $w=6$, we want to move left.

How can one rule handle both cases?

The derivative gives the slope:

$$\frac{dL}{dw}=2(w-3)$$

At $w=0$:

$$\frac{dL}{dw}=2(0-3)=-6$$

The negative gradient says that moving toward larger $w$ should lower the loss.

At $w=6$:

$$\frac{dL}{dw}=2(6-3)=6$$

The positive gradient says that moving toward smaller $w$ should lower the loss.

One formula can therefore move downhill from either side:

$$w_{new}=w_{old}-\eta\frac{dL}{dw}$$

The Greek letter $\eta$, pronounced eta, represents the learning rate.

---

Why do we subtract the gradient?

The gradient points in the direction of steepest local increase.

We want to reduce the loss, so we move in the opposite direction:

$$\text{parameter}-\text{gradient step}$$

Suppose the gradient is positive:

gradient = +4

Subtracting it reduces the parameter:

$$w_{new}=w_{old}-\eta(4)$$

Suppose the gradient is negative:

gradient = -4

Subtracting a negative value increases the parameter:

$$w_{new}=w_{old}-\eta(-4)=w_{old}+4\eta$$

So the same subtraction rule automatically chooses the correct local direction.

Gradient ascent uses addition instead and tries to maximize an objective. Neural-network training usually minimizes a loss, so we use descent.

---

What does the learning rate actually do?

The gradient tells us the slope. It does not decide the whole step.

The learning rate scales it:

$$\text{step}=\text{learning rate}\times\text{gradient}$$

Return to our simple bowl. Start with:

$$w=0,\qquad L=9,\qquad \frac{dL}{dw}=-6$$

A learning rate of 0.1

$$w_{new}=0-(0.1\times-6)=0.6$$

The new loss is:

$$(0.6-3)^2=5.76$$

The loss moved from 9 to 5.76. We are heading in the right direction.

A learning rate of 0.01

$$w_{new}=0-(0.01\times-6)=0.06$$

The move is safe but tiny. Too many tiny steps can make training painfully slow.

A learning rate of 0.5

$$w_{new}=0-(0.5\times-6)=3$$

For this unusually simple loss, we land exactly at the minimum in one step. Real loss landscapes rarely offer such luck.

A learning rate of 1

$$w_{new}=0-(1\times-6)=6$$

We jumped over the minimum. At $w=6$, the gradient is +6, so the next step sends us back to 0. The parameter can bounce forever:

0 → 6 → 0 → 6 → 0 ...

A learning rate greater than 1

With $\eta=1.1$:

0 → 6.6 → -0.72 → 8.064 ...

Instead of converging, the jumps grow. Training diverges.

This gives us the learning-rate trade-off:

Learning rate	Possible behaviour
Very small	Stable but slow
Reasonable	Makes useful progress
Too large	Overshoots or oscillates
Far too large	Loss explodes or becomes NaN

There is no universally perfect learning rate. A useful value depends on the model, optimizer, batch size, data and stage of training.

---

Now return to our neural network

Backpropagation gave one output weight this gradient:

weight   =  0.4000
gradient = -0.6512

Using a learning rate of 0.1:

$$w_{new}=0.4-(0.1\times-0.6512)$$ $$\boxed{w_{new}=0.4651}$$

The weight increased because its gradient was negative.

For a connection leading toward the incorrect token green, we had:

weight   = 0.1000
gradient = 0.4630

Update it:

$$w_{new}=0.1-(0.1\times0.4630)$$ $$\boxed{w_{new}=0.0537}$$

That weight decreased.

One update strengthened a useful route toward blue and weakened a route toward green. The optimizer did not know the meanings of those words. It simply followed the gradients produced by the loss.

After all weights and biases were updated, another forward pass produced:

Measurement	Before update	After update
$P(blue)$	45.73%	56.44%
$P(green)$	38.58%	30.37%
$P(runs)$	15.69%	13.19%
Loss	0.782	0.572

For this training example, the step helped.

But do not conclude that a large learning rate is always better just because it might reduce this one example’s loss quickly. Training must improve performance across many varied examples. An aggressive update for one batch may damage what the model learned from earlier batches.

---

A neural network does not have one weight

Our bowl example had one horizontal direction: $w$.

The tiny network already has many parameters. A real model has vastly more. Its current state is therefore a point in a high-dimensional parameter space.

For three parameters, we could write:

$$\theta=[w_1,w_2,b_1]$$

For a large model, $\theta$ contains every trainable weight and bias.

The gradient has the same structure:

$$\nabla_\theta L= \left[ \frac{\partial L}{\partial w_1}, \frac{\partial L}{\partial w_2}, \frac{\partial L}{\partial b_1}, \ldots \right]$$

The symbol $\nabla$, pronounced nabla, means “collect the partial derivatives with respect to all these parameters.”

Gradient descent updates them together:

$$\theta_{new}=\theta_{old}-\eta\nabla_\theta L$$

This is the familiar one-weight formula applied across the entire parameter collection.

Each parameter can move by a different amount because each has a different gradient. The learning rate may be shared, but the gradients are not.

---

Does the optimizer find the global lowest point?

Not necessarily.

The simple bowl had one obvious minimum. Neural-network loss landscapes are high-dimensional and can contain:

• valleys;
• flat regions called plateaus;
• steep directions beside shallow ones;
• saddle points, which curve upward in some directions and downward in others;
• many parameter configurations with similarly low loss.

The gradient only describes the immediate neighborhood. It does not give the optimizer a map of the entire landscape.

That sounds limiting, yet gradient-based optimization works remarkably well in practice. Large neural networks often have many routes to useful low-loss solutions, and techniques such as momentum, adaptive learning rates, normalization and learning-rate schedules make the journey more reliable.

The practical goal is not usually to prove that we found the mathematically lowest possible training loss. It is to find parameters that perform well on unseen data.

---

One gradient from which data?

So far, our gradient came from one example:

The sky is → blue

But a training dataset may contain millions or trillions of token targets. How many should contribute to one update?

This creates three related approaches. Their names describe how much data is used to estimate a gradient before one update; they do not describe three different backpropagation algorithms.

Batch gradient descent

Use the entire training dataset to calculate one gradient, then update once.

all examples → one average gradient → one update

The direction is stable, but each update can be prohibitively expensive for a large dataset.

Stochastic gradient descent

Use one training example for each update.

one example → one noisy gradient → one update

Updates are cheap and frequent, but one example may point in a noisy direction.

Strictly, stochastic gradient descent means this single-example version. In everyday deep-learning discussion, however, people often call the optimizer “SGD” even when it operates on mini-batches. That overloaded name is a common source of beginner confusion.

Mini-batch gradient descent

Use a small group of examples for each update.

small batch → combined gradient → one update

This is the usual deep-learning choice. It offers a useful compromise:

• more stable than a single example;
• much cheaper than the whole dataset;
• efficient on GPUs because examples can be processed in parallel.

Suppose a batch has four examples. Each produces some pressure on the same weight:

Example	Gradient contribution
“The sky is → blue”	-0.651
“Grass is often → green”	+0.120
“A clear ocean looks → blue”	-0.330
“The athlete → runs”	+0.090

If we average them:

$$\frac{-0.651+0.120-0.330+0.090}{4}=-0.19275$$

The combined gradient still suggests increasing this weight, but less aggressively than the first example alone.

The numbers are illustrative, but the principle is real: gradients from the batch combine before the parameter update.

---

Why training loss does not fall smoothly

People often imagine training as:

loss: 5.0 → 4.0 → 3.0 → 2.0 → 1.0

Real mini-batch training may look more like:

loss: 5.0 → 4.2 → 4.6 → 3.8 → 3.9 → 3.1

One batch may be harder than another. Its gradient may conflict with previous updates. Random batch composition introduces noise.

A temporary increase does not automatically mean training has failed. Engineers often inspect a smoothed loss trend, validation loss and other evaluation metrics.

However, a loss that repeatedly explodes, becomes NaN or trends upward for a long period may indicate:

• an excessive learning rate;
• exploding gradients;
• incorrect data or labels;
• numerical instability;
• a bug in the loss or model;
• or inappropriate preprocessing.

---

Why not keep the same learning rate forever?

Early in training, the model may be far from a useful solution. Larger steps can make progress quickly.

Later, when the model is near a useful region, the same step size may bounce around instead of settling.

A learning-rate schedule changes the learning rate during training.

A common shape is:

small warm-up → larger working rate → gradual decay

Warm-up

Training begins with a small learning rate and increases it over an initial period. This can prevent unstable early updates when activations and optimizer statistics are not yet well behaved.

Decay

The rate gradually decreases so that later updates become more precise.

Schedules may use step reductions, exponential decay, cosine-shaped decay or other rules. The best choice depends on the training setup. The main intuition is more important than the name:

Take controlled steps early, make useful progress, then become more careful.

---

Why plain gradient descent is often not enough

Imagine descending a long, narrow valley. The slope is steep from side to side but shallow along the path toward the bottom.

Plain gradient descent may zigzag across the valley while making slow forward progress.

Modern optimizers modify the basic update to handle problems like this. They do not replace backpropagation. They use the gradients backpropagation provides.

SGD

The basic update is:

$$\theta_{t+1}=\theta_t-\eta g_t$$

where $g_t$ is the current gradient.

SGD is simple, memory-efficient and can generalize well, but it may require careful learning-rate tuning.

Momentum

Momentum keeps a running direction from earlier gradients.

Think of a ball rolling downhill. Repeated gradients in the same direction build speed. Alternating side-to-side gradients partly cancel.

One simplified form is:

$$v_t=\beta v_{t-1}+g_t$$ $$\theta_{t+1}=\theta_t-\eta v_t$$

Here, $v_t$ is the accumulated movement and $\beta$ controls how much history is retained.

Adam

Adam keeps moving averages of:

• the gradients, providing momentum-like direction;
• the squared gradients, estimating the recent scale of gradients.

It then gives parameters adaptive step sizes. A parameter with consistently large gradients can be scaled differently from one with small gradients.

Adam is widely used because it often trains effectively with less manual tuning than plain SGD, though it still has important hyperparameters.

AdamW

AdamW uses Adam-style adaptive updates while applying weight decay separately from the gradient-based update.

Weight decay gently discourages weights from growing unnecessarily large. The “W” does not mean “Adam with weights”—it refers to the decoupled weight-decay formulation.

AdamW is a common choice for training Transformer-based models.

Optimizer	Main idea	Typical trade-off
SGD	Follow current gradient	Simple but may need careful tuning
SGD + Momentum	Add memory of past direction	Faster through consistent directions
Adam	Momentum plus adaptive scaling	Convenient, but uses extra optimizer state
AdamW	Adam with decoupled weight decay	Common Transformer choice; still needs tuning

No optimizer is universally best. Architecture, data, compute budget and generalization goals matter.

---

Weight decay is not the learning rate

These controls are easy to mix up.

The learning rate answers:

How strongly should the optimizer respond to the update direction?

Weight decay answers:

How strongly should we discourage large parameter values?

They can operate in the same optimizer step, but they solve different problems.

Likewise, gradient clipping is different again. It limits gradients when their norm becomes dangerously large. It is a safety mechanism, not a replacement for choosing a sensible learning rate.

---

The update in Python

Once gradients have been calculated, plain gradient descent needs only a few lines:

learning_rate = 0.1

W1 = W1 - learning_rate * grad_W1
b1 = b1 - learning_rate * grad_b1
W2 = W2 - learning_rate * grad_W2
b2 = b2 - learning_rate * grad_b2

In PyTorch, an optimizer handles parameter updates:

optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

optimizer.zero_grad()  # Clear gradients left by the previous step
logits = model(inputs) # Forward pass
loss = loss_fn(logits, targets)
loss.backward()        # Backpropagation calculates gradients
optimizer.step()       # Optimizer updates parameters

The order matters.

PyTorch accumulates gradients by default. Without clearing them at the intended time, the new gradients are added to existing ones. Sometimes accumulation is deliberate—for example, to simulate a larger batch. Often, forgetting to clear them is simply a bug.

An AdamW setup might look like:

optimizer = torch.optim.AdamW(
    model.parameters(),
    lr=3e-4,
    weight_decay=0.01,
)

Those values are examples, not universal recommendations.

---

How engineers know whether the step size is working

Useful signals include:

• training loss over time;
• validation loss;
• gradient norms;
• parameter-update norms;
• the ratio of update size to parameter size;
• the presence of NaN or infinite values;
• task-specific evaluation metrics.

Patterns can provide clues:

Observation	Possible interpretation
Loss decreases extremely slowly	Learning rate may be too small
Loss swings violently	Learning rate may be too large
Loss becomes NaN	Numerical instability or exploding update
Training loss falls but validation worsens	Possible overfitting
Gradients are near zero in early layers	Possible vanishing gradients
Update is huge relative to parameter	Potential instability

These are diagnostic hints, not guaranteed conclusions. Training behaviour is the combined result of the model, data, loss, precision, optimizer and distributed setup.

---

Common misunderstandings

“Gradient descent and backpropagation are the same thing”

Backpropagation calculates gradients. Gradient descent uses gradients to update parameters.

“The gradient tells us the perfect new weight”

It gives local slope information. It does not reveal the globally best value.

“A larger gradient always means a larger final update”

Not necessarily with adaptive optimizers, gradient clipping, momentum, weight decay or parameter-specific rules.

“The learning rate is how much the weight changes”

The learning rate scales the update. The actual change also depends on the gradient and optimizer state.

“A negative gradient means the weight should become negative”

No. It means increasing that parameter locally reduces the loss. The updated parameter may remain positive, become larger, or cross zero depending on the step.

“Every training step must lower every example’s loss”

Mini-batch updates optimize a changing sample of the dataset. Improving one batch can temporarily worsen another.

“AdamW means learning-rate tuning no longer matters”

Adaptive optimizers reduce some tuning difficulty; they do not eliminate it.

---

The one thing to remember

Backpropagation gave our model a direction:

$$g=-0.6512$$

Gradient descent combined it with a chosen step size:

$$\eta=0.1$$

Then it updated the weight:

$$w_{new}=0.4-(0.1\times-0.6512)=0.4651$$

That one equation contains three different ideas:

current weight → where the model is now
gradient       → local direction and sensitivity
learning rate  → how strongly the model responds

Across millions or billions of parameters, the optimizer repeats this idea after every training batch. Modern methods add memory, adaptive scaling and weight decay, but they still build upon the same foundation.

A neural network does not leap directly from wrong to correct.

It measures the slope, takes a step, checks the new loss and repeats.

That repeated downhill search is gradient-based optimization.

Together, the three articles now describe one complete learning step:

input → prediction → loss → gradients → parameter update

The natural next topic is what happens when we repeat that step across many examples: batches, iterations and epochs.

---

Sources and further reading

• PyTorch: Optimizing Model Parameters
• PyTorch: SGD
• PyTorch: Adam
• PyTorch: AdamW
• Kingma and Ba: Adam—A Method for Stochastic Optimization
• Loshchilov and Hutter: Decoupled Weight Decay Regularization

Continue the series

1. Forward Pass: Before a Neural Network Can Learn, It Must Make a Prediction
2. Backpropagation: The Model Was Wrong. Which Weight Should We Blame?
3. Gradient Descent: The Gradient Knows the Direction. But How Far Should the Model Move?