Learning Rate

The Gradient Descent article’s update formula had one term this article has been promising a full look at: new_w = old_w - (learning_rate × gradient). The gradient tells you which direction to step. The learning rate tells you how big that step actually is.

The simple definition

The learning rate is a number, chosen before training begins, that controls how large each weight update is during gradient descent. It’s a Hyperparameter — set by an engineer, not learned by the model — and it’s widely considered one of the single most consequential choices in the entire training process, precisely because it directly scales every single weight adjustment the model ever makes.

The multiplier controlling each step

The update rule is:

new parameter = old parameter - learning rate × gradient

If the weight is 5 and gradient is 4:

Learning rate	Update	New weight
0.01	`0.01 × 4 = 0.04`	4.96
0.25	`0.25 × 4 = 1`	4.00
1.00	`1 × 4 = 4`	1.00

The gradient supplies direction and sensitivity. The learning rate controls how strongly the optimizer responds.

Too small, suitable, and too large

Too small → slow progress and wasted compute
Suitable  → stable progress toward lower loss
Too large → overshooting, oscillation, or divergence

Many training runs use a schedule:

warm up carefully → train with larger steps → reduce steps near the end

The best value depends on the model, optimizer, batch size, data, and training stage. It is normally chosen using experiments and validation results.

One published InstructGPT fine-tuning example: 0.00000503

OpenAI’s InstructGPT paper reports an initial supervised fine-tuning learning rate of 5.03 × 10⁻⁶ for its 175B model.

5.03 × 10⁻⁶
= 0.00000503

The paper reports a cosine schedule that reduces the learning rate to 10% of its initial value by the end:

initial: 0.00000503
final:   approximately 0.000000503

The schedule changes the step multiplier over training. It does not mean every weight changes by exactly those amounts; each update also depends on gradients and Adam’s optimizer state.

Building the intuition, continuing the hillside picture

Return once more to the fogged-in hiker from the Gradient article, now actually walking (Gradient Descent). The learning rate is the hiker’s stride length. A hiker taking huge, confident strides down a foggy hillside might cover ground fast, but risks overshooting the valley floor entirely, stumbling past it, or even ending up higher than where they started if the terrain curves unexpectedly.

A hiker taking tiny, cautious steps will almost certainly get closer to the true bottom eventually, but might take an impractically long time to get there — possibly running out of daylight (or, for a model, running out of allotted training time and compute budget) before ever arriving.

flowchart LR
    A[Learning rate too high] --> B[Overshoots, bounces around, may never settle]
    C[Learning rate too low] --> D[Crawls forward, extremely slow, may get stuck early]
    E[Learning rate well-tuned] --> F[Steady, efficient progress toward low loss]

What actually goes wrong at each extreme

This is worth making concrete, because both failure modes look different in practice and an engineer needs to recognize each one:

Too high a learning rate. Each update is so large that the model’s weights swing past the good values entirely, then swing back past them again the other way — sometimes settling into a worse position than before. In the most extreme cases, this can cause the loss to actually increase over time instead of decrease, a visible sign of a badly miscalibrated learning rate, sometimes described as training that “diverges” rather than converges.
Too low a learning rate. Each update is so small that the model barely changes at all, step after step. Training technically works, in the sense that loss does slowly decrease, but it can take an impractically large number of steps — and given how expensive training compute is, as covered in the Training article, an unnecessarily low learning rate can mean paying for weeks of GPU time that a better-tuned learning rate could have accomplished in days.

Why a single fixed number often isn’t the best approach

In practice, using one constant learning rate for an entire training run is rarely optimal, and real training setups usually adjust it over time using a learning rate schedule.

A common pattern: start with a relatively higher learning rate, when the weights are still far from good values and large steps make sense, then gradually reduce it as training progresses, taking smaller, more careful steps as the model gets closer to a good solution — much like a hiker taking big strides down the open hillside early on, then shortening their steps as they near the valley floor and want to avoid overshooting a good spot.

Another common technique, called warmup, does the opposite at the very start: begin with a very small learning rate for the first portion of training and gradually increase it, which helps avoid unstable, erratic updates while the model’s weights are still at their initial random values.

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of adjusting a shower’s temperature knob. Turn it in large increments, and you’ll swing wildly between too cold and too hot, never quite landing on comfortable. Turn it in tiny, cautious increments, and you’ll eventually find the right temperature, but it takes many more small adjustments to get there.

Where this breaks down: A person adjusting a shower knob can feel the water and consciously judge “getting warmer, almost there” — a form of real-time feedback and reasoning. Gradient descent has no such awareness beyond the current gradient; the learning rate is fixed in advance (or follows a predetermined schedule) rather than being intelligently adjusted moment-to-moment based on genuine judgment about how close the model is to done.

Learning rate vs. gradient vs. step: keeping the three straight

These three ideas travel together so constantly that they’re worth separating explicitly, since blurring them is an easy mistake even after reading the last two articles closely. The gradient, covered in its own article, is a measurement — the slope of the loss at the model’s current position, telling you direction and relative urgency, nothing more.

The learning rate, covered in this article, is a fixed setting — chosen in advance, controlling how large a step to take. The actual step — the real change applied to a weight — is what you get when you combine the two: step = learning_rate × gradient.

None of these three is optional or redundant with the others: without the gradient, you wouldn’t know which direction to go; without the learning rate, you’d have no way to decide how far to go in that direction; without actually taking the step, nothing about the weight would ever change.

Keeping these three roles distinct — measurement, setting, action — is what makes the whole gradient descent process, described across the last three articles, easy to reason about rather than a single blurred idea.

How engineers actually find a good learning rate

Since there’s no universal correct value — the right learning rate depends on the specific model architecture, dataset, and loss function involved — finding one is fundamentally an experimental process, exactly the kind of hyperparameter tuning described in the Hyperparameters article.

A common practical starting point is trying a small set of standard values (like 0.1, 0.01, 0.001) and observing how the loss behaves on the validation set over the first portion of training, then narrowing in from there.

Frontier AI labs invest significant effort in tuning learning rate schedules specifically because, at the scale of a training run costing tens of millions of dollars (as covered in the Training article), a poorly chosen learning rate isn’t just a minor inefficiency — it can mean the difference between a successful run and a wasted one.

A concrete example, layered

For a simple beginner example: training the one-weight house model with a learning rate of 0.5 might see w swing wildly between values that are too high and too low for several steps before settling down, while a learning rate of 0.0001 would see w creep toward its ideal value so slowly that a beginner watching the loss might wrongly conclude training “isn’t working” at all, when it’s actually just working very slowly.

For a production example: OpenAI’s published GPT-3 training recipe used a learning rate of 6×10⁻⁴, with a gradual linear warmup over the first 375 million tokens processed, followed by a slow cosine-shaped decay down to a minimum learning rate of 6×10⁻⁵ over the remaining training run — a concrete, real illustration of the warmup-then-decay schedule described above, rather than one single fixed value used throughout.

Check your understanding

Is the learning rate learned like a weight? Usually no. It is a hyperparameter or scheduled setting.

Does doubling the learning rate always halve training time? No. It may make training unstable or worsen the final solution.

Common misconception

A frequent beginner assumption: that a smaller learning rate is always “safer” and therefore always the better default choice, since it avoids the dramatic instability of an overly large one.

This isn’t quite right — an unnecessarily small learning rate carries its own real cost, in wasted training time and compute, and can sometimes get a model stuck in a mediocre solution simply because it never took steps large enough to escape a poor starting region of the loss landscape (echoing the local minimum concern from the Gradient article).

The learning rate isn’t a “smaller is always better” dial — it’s a genuine trade-off between speed and stability that has to be tuned to the specific situation, not defaulted to an extreme in either direction.

A learning-rate schedule over time

Training may begin with a warmup that raises the learning rate gradually, continue near a useful peak, and then decay it so later updates become gentler. The quoted InstructGPT value belongs to a particular published fine-tuning setup; it is not one universal GPT learning rate.

The update combines three distinct ideas: the gradient supplies direction and relative size, the learning rate scales the move, and the optimizer may further transform it using momentum or adaptive statistics.

Where this fits in what comes next

You now understand the setting that controls the size of every gradient descent step. The next two articles, Batch and Batch Size, cover a related but distinct practical question this article has referenced but not yet explained: exactly how many training examples get used to calculate each gradient before an update happens at all.

In one sentence

The learning rate controls how large each step of gradient descent is, and getting it right — often through deliberate schedules rather than one fixed value — is one of the highest-leverage, most consequential decisions in the entire training process.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026