Gradient

The Loss Function article ended with a precise, calculable number — the loss — representing exactly how wrong a model’s prediction was. It also ended with a genuine gap: knowing the loss is high doesn’t, by itself, tell you which direction to adjust any given weight to make it lower. That missing piece — the thing that actually answers “which way, and how much” — is called a gradient.

The simple definition

A gradient tells you, for a given parameter, which direction to change it in order to increase or decrease the loss, and how strongly that parameter currently affects the loss. It’s a single number, calculated separately for every weight and every bias in the model, that essentially says: “if you nudge this particular parameter up slightly, the loss will go up (or down) by roughly this much.”

The direction and size of change

Suppose a model has one weight w, and its loss is:

loss = (w - 3)²

The best weight is 3, where loss becomes zero.

At w = 5:

gradient = 2 × (w - 3)
         = 2 × (5 - 3)
         = 4

The positive gradient says increasing the weight would increase loss. To reduce loss, move the weight in the opposite direction.

At w = 2:

gradient = 2 × (2 - 3) = -2

The negative gradient says the downhill direction is toward a larger weight.

gradient sign → direction
gradient size → steepness or sensitivity

Many parameters mean many gradient values

A model with one million parameters normally has a corresponding gradient value for each trainable parameter during a training step.

loss
 ↓ backpropagation
[gradient for weight₁, gradient for weight₂, ..., gradient for bias]

Building the intuition with a hillside

The most common and genuinely useful way to picture this: imagine standing on a hillside in thick fog, unable to see anything beyond your own feet, trying to reach the lowest point in the valley below. You can’t see the whole landscape, but you can feel which way the ground slopes right where you’re standing. That local slope — how steeply the ground rises or falls under your feet, and in which direction — is exactly what a gradient represents, mathematically, for the loss function described in the previous article.

flowchart LR
    A[Current weight value] --> B[Calculate gradient: slope of the loss at this point]
    B --> C{Which direction reduces loss?}
    C --> D[Nudge weight in that direction]

The “landscape” here isn’t a physical hillside, of course — it’s the loss function, plotted across every possible value a given weight could take. At any specific point on that landscape (the weight’s current value), the gradient tells you the slope right there: how much the loss would change if you nudged that weight slightly. A steep slope means a small change to the weight would swing the loss a lot; a nearly flat slope means the weight barely matters to the loss right now.

ANALOGY vs. TECHNICAL REALITY

Analogy: Picture a hiker in fog, feeling the ground’s slope underfoot to decide which direction to step next, always choosing to step downhill toward the valley floor.

Where this breaks down: A hiker feels one slope, under two feet, in a landscape with two directions (north-south, east-west). A real model calculates a separate gradient for every single one of its weights and biases simultaneously — potentially billions of them at once, each in its own separate “direction” of an almost incomprehensibly multi-dimensional landscape. There’s no physical intuition that fully captures a billion-dimensional hillside; the hiker analogy works for the idea of “follow the local slope downhill,” but the actual scale and dimensionality of what’s being calculated has no true physical equivalent.

What a gradient actually looks like, worked through simply

Recall the one-weight house model from the Parameters and Weights articles: predicted_price = square_footage × w. Suppose, for a specific house, the model’s current weight produces a prediction that’s too high, and the loss function (from the previous article) is being used to measure that error.

The gradient for w, at this specific point in training, is a number that says something like: “increasing w right now would increase the loss; decreasing w would decrease it, and the effect is fairly strong.” That gradient value itself doesn’t change the weight — it’s a piece of information about the weight, calculated fresh at each training step, based on the model’s current predictions and the actual labels.

Critically: the gradient isn’t the same for every weight, and it isn’t fixed — it’s recalculated at every single training step, because as weights change, the “shape of the hillside” as experienced from that new position changes too.

Why the gradient’s sign and size both matter

A gradient carries two pieces of information at once, and both are used:

• Sign (positive or negative) — tells you which direction increases the loss, so the weight should move the opposite way to reduce it.
• Magnitude (how large the number is) — tells you how sensitive the loss currently is to this particular weight. A large-magnitude gradient means this weight has a strong current effect on the loss; a small-magnitude gradient (close to zero) means this weight barely matters right now, or the model may already be close to a good value for it.

This is exactly the information the next article, Gradient Descent, uses to actually perform the update — the gradient tells you the direction and relative urgency; gradient descent is the procedure that turns that information into an actual change to the weight’s value.

Where gradients actually come from, computationally

It’s worth being honest about the scale involved here without diving into the full mechanics (covered properly in the upcoming Backpropagation article): calculating the gradient for every single weight and bias in a large language model, for every single training step, is an enormous computational undertaking — this is a large part of why training requires the massive GPU or TPU clusters described in the Training article.

A model with hundreds of billions of parameters needs hundreds of billions of individual gradient calculations, repeated at every training step, across potentially trillions of tokens of training data. The specific, clever algorithm that makes this computationally feasible at all — rather than absurdly, impossibly slow — is backpropagation, and it’s the very next major mechanic this glossary phase will cover.

Where this approach has real limitations

Following the gradient downhill is a genuinely powerful, general-purpose strategy — it works without needing to understand the loss landscape’s full shape in advance, which is essential given how impossibly complex that landscape is for a model with billions of parameters. But it comes with real limitations worth knowing.

The gradient only ever describes the local slope, right at the model’s current position — it has no visibility into the broader landscape beyond that immediate point.

This means the process can, in principle, settle into a spot that’s the lowest point in its immediate neighborhood but not the truest lowest point overall — a local minimum — without any way to know, from the gradient alone, whether a much better solution exists somewhere else in that landscape entirely.

In practice, this concern turns out to matter less for the enormous, high-dimensional landscapes of modern deep learning than early researchers once feared, but it remains a real, acknowledged limitation of gradient-based learning, not a solved problem.

A concrete example, layered

For a simple beginner example: in the one-weight house model, if the model consistently underpredicts prices across many training examples, the gradient for w will consistently point in the direction of “increase w” — and repeated small increases, guided by this signal, will gradually correct the underprediction.

For a production example: OpenAI’s published GPT-3 architecture has 175 billion parameters spread across 96 transformer layers — meaning every single training step involves calculating a separate gradient for each of those 175 billion numbers, across a training run that, as covered in the Training article, reportedly used a supercomputer-scale cluster of roughly 10,000 GPUs running continuously to make this feasible at all.

Check your understanding

Is a gradient the parameter update itself? No. The optimizer combines the gradient with the learning rate and possibly other state.

Why move opposite the gradient? The gradient points toward increasing loss; the opposite direction reduces it locally.

Common misconception

A common early mix-up: assuming the gradient is the update — that calculating the gradient directly changes the weight. It doesn’t. The gradient is purely diagnostic information: a measurement of the current slope, nothing more. Something has to actually use that information to decide how big a step to take and apply it — that’s the job of gradient descent, along with the learning rate hyperparameter introduced in the Hyperparameters article, both covered next. Gradient and gradient descent are closely related but distinct: one measures the slope, the other is the procedure that walks downhill using that measurement.

From one slope to a gradient vector

For one parameter, the derivative is one slope. For a model with parameters w1, w2, and b, the gradient is a collection such as [0.4, -0.1, 0.03]. Each position says how the loss would change if that corresponding parameter moved slightly.

Very small gradients can make early layers learn slowly; this is called a vanishing-gradient problem. Very large gradients can cause unstable jumps; this is called an exploding-gradient problem. Architecture choices, normalization, initialization, and gradient clipping can help.

Where this fits in what comes next

You now understand what a gradient is: the slope of the loss function at a weight’s current value, telling you which direction and how strongly to consider changing it. The next article, Gradient Descent, covers the actual algorithm that repeatedly uses this information to walk the loss function downhill, step by step, until it settles somewhere good — turning the diagnostic signal described in this article into the real, repeated weight updates that constitute training.

In one sentence

A gradient is the measured slope of the loss function at a weight’s current value — telling you which direction and how strongly to nudge that weight — and it’s the essential diagnostic signal that everything else in the training process depends on to actually know which way is “downhill.”

Related Terms

• Loss Function
• Weights
• Training
• Gradient Descent
• Hyperparameters