The Gradient article ended with a clean, important distinction: the gradient measures the slope, but something else has to actually use that measurement to change a weight’s value. That something is gradient descent — the algorithm that turns “here’s the slope” into “here’s the new, better weight.”
The simple definition
Gradient descent is the algorithm that repeatedly nudges a model’s weights and biases in the direction the gradient says will reduce the loss, a small step at a time, until the loss stops meaningfully improving. If the gradient is the compass reading — “downhill is that way, and fairly steeply” — gradient descent is the act of actually taking a step in that direction, checking the new compass reading, taking another step, and repeating this thousands or millions of times.
This is, quite literally, the specific mathematical procedure the Algorithm article referenced when it named gradient descent as the family behind most modern Deep Learning systems.
Three numerical updates
Use the loss:
loss = (w - 3)²
gradient = 2(w - 3)Start with w = 5 and learning rate 0.25.
new weight = old weight - learning rate × gradientUpdate 1:
gradient = 2(5 - 3) = 4
new w = 5 - 0.25 × 4 = 4Update 2:
gradient = 2(4 - 3) = 2
new w = 4 - 0.25 × 2 = 3.5Update 3:
gradient = 2(3.5 - 3) = 1
new w = 3.5 - 0.25 × 1 = 3.25The weight moves toward 3, and the steps become smaller as the gradient becomes smaller.
flowchart LR
A[Calculate prediction] --> B[Calculate loss]
B --> C[Calculate gradient]
C --> D[Update parameters downhill]
D --> AWhy this deserves to be called out as its own distinct thing
It’s worth being precise here, because “gradient” and “gradient descent” get used almost interchangeably in casual conversation, and confusing them genuinely obscures what’s happening. The gradient, on its own, is inert — a snapshot measurement that changes nothing by itself.
Gradient descent is the active process: take the gradient, decide how big a step to take based on it, apply that step to update the weight, then immediately recalculate the gradient at the new position and repeat. Without this repeated, iterative loop, a single gradient calculation would be a useless piece of trivia rather than the engine of an entire training run.
flowchart LR
A[Current weights] --> B[Calculate gradient]
B --> C[Take a small step in the downhill direction]
C --> D[Updated weights]
D --> BWalking through one update, concretely
Return to the one-weight house model, with its current weight w = 100, and suppose the gradient calculated at this point says: “increasing w would raise the loss; the slope is fairly steep.” Gradient descent’s actual update rule looks like this:
new_w = old_w - (learning_rate × gradient)If the gradient is positive (meaning: increasing w increases loss), subtracting a positive amount moves w down — exactly the direction that reduces loss. If the gradient were negative instead, subtracting a negative number would move w up. This is the entire mechanism, applied over and over: always step in the direction opposite the gradient’s sign, because that’s the direction that decreases loss. The learning_rate term controls exactly how big that step is — a hyperparameter, first introduced in the Hyperparameters article, that the very next article in this phase covers in full.
ANALOGY vs. TECHNICAL REALITY
Analogy: Return to the fogged-in hiker from the Gradient article. Gradient descent is the hiker actually walking: feel the slope, take one step downhill, stop, feel the slope again from the new spot, take another step — repeating this, step after step, until the ground underfoot feels essentially flat, meaning they’ve reached (or gotten very close to) the valley floor.
Where this breaks down: A hiker takes physical steps of roughly consistent size and can look around, use landmarks, or backtrack if a path seems wrong. Gradient descent has no such awareness — its step size is set entirely by the learning rate hyperparameter, chosen in advance, and it recalculates the gradient and steps again mechanically, for every single one of a model’s potentially billions of parameters simultaneously, with zero ability to “look around” beyond what the current gradient tells it.
Why this needs to happen so many times
A single gradient descent step rarely, if ever, gets a weight to its ideal value in one move — the step size is deliberately kept small (governed by the learning rate), because taking one enormous leap based on a single local slope reading would very likely overshoot past the actual best value, especially since the slope itself changes as the weight changes.
This is exactly why training, as described throughout the Training article, involves so many repeated steps across so many epochs: each individual gradient descent step makes only a small, incremental improvement, and it’s the accumulation of many such steps — recalculating the gradient fresh each time — that gradually walks the loss down to a genuinely good value.
Real-world variations engineers actually use
Plain gradient descent, calculating the gradient using the entire training dataset before taking even one step, turns out to be impractically slow for real datasets — recalculating gradients across millions or billions of training examples before making a single tiny adjustment wastes an enormous amount of computation.
In practice, essentially all real training uses stochastic gradient descent (SGD) or one of its refinements — calculating the gradient using just a small Batch of examples at a time (the next-but-one article in this phase covers this properly), updating weights far more frequently, using noisier but much faster gradient estimates.
Modern training runs typically go a step further still, using refined variants like Adam, an optimizer (the subject of an upcoming article in this phase) that adjusts the effective step size for each individual parameter automatically, based on that parameter’s recent gradient history — a meaningful practical improvement over the plain, fixed-step version described so far in this article.
Where this genuinely struggles
Gradient descent, for all its power and ubiquity, isn’t a magic fix — it inherits every limitation of the gradient it depends on, discussed in the previous article, including the risk of settling into a local minimum rather than the truest best solution available.
It can also behave badly if the step size is poorly chosen: too large, and updates can overshoot and bounce around without ever settling, exactly as the Training article warned when discussing learning rate; too small, and training can crawl forward so slowly it becomes impractical.
And because it requires recalculating gradients repeatedly across enormous numbers of parameters, it’s the single largest source of the computational cost — and the GPU/TPU infrastructure — described in the Training article’s discussion of what makes training expensive at frontier-lab scale.
A concrete example, layered
For a simple beginner example: training the one-weight house model might start with w at a random value, take a gradient descent step, check the new loss, take another step, and after perhaps a few hundred such steps, w settles near its ideal value and further steps barely change it — a clear sign training has converged.
For a production example: OpenAI’s published GPT-3 recipe ran gradient descent (in its stochastic, batch-based form) across roughly 300 billion training tokens, using a batch size of 256 sequences and a 2,048-token context window — meaning roughly half a million separate training steps over the full run, each one calculating a fresh gradient across all 175 billion parameters simultaneously and nudging every one of them before moving to the next batch.
Check your understanding
Does gradient descent guarantee the globally best solution? No. Real loss landscapes can be complicated.
Why repeat the update? One local step rarely reaches useful parameters for a complex model.
Common misconception
A frequent early mix-up, worth restating plainly since it echoes the Gradient article’s closing point: gradient descent is not the same thing as the gradient itself. The gradient is a measurement; gradient descent is the repeated act of using that measurement to actually change the weights. A second, related misconception: assuming gradient descent guarantees finding the single best possible set of weights.
It doesn’t guarantee global optimality — as the limitations section above explained, it can settle into a locally good, but not necessarily globally best, solution, and in practice, “good enough to work extremely well” is the realistic, achievable goal, not mathematical perfection.
Three meanings commonly hidden behind the name
| Method | Examples used for one gradient estimate |
|---|---|
| Full-batch gradient descent | The entire training dataset |
| Stochastic gradient descent | One example |
| Mini-batch gradient descent | A manageable group of examples |
Modern deep learning usually uses mini-batches, even when people shorten the name to “gradient descent” or “SGD.” Backpropagation calculates the gradients; the optimizer uses them to update parameters.
Where this fits in what comes next
You now understand the complete loop this phase has been building toward: a loss function measures wrongness, a gradient measures the slope of that wrongness with respect to each weight, and gradient descent repeatedly uses that slope to nudge weights toward better values. The next article, Learning Rate, zooms into the one setting mentioned throughout this article that controls exactly how big each of those steps is — arguably the single most consequential hyperparameter in the entire training process.
In one sentence
Gradient descent is the repeated, mechanical process of using the gradient’s direction and strength to nudge every weight a small step toward lower loss, over and over, and it’s the actual algorithmic engine underneath the word “training” everywhere else in this glossary.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed