How AI Works · 20 min read
The Model Scored 99% in Practice—and Failed the Real Test
A slow, beginner-first explanation of underfitting, overfitting and generalization, with training curves, examples, diagnosis, fixes and modern AI connections.
Understanding underfitting, overfitting and what learning really means
Imagine two students preparing for a mathematics exam.
The first student studies for ten minutes. He remembers one formula, skips most of the chapters and cannot even solve the practice questions.
The second student does something more impressive. She memorizes every practice question—including the position of each answer on the page. She scores 99% when the same questions are shown again.
Then the real exam arrives.
The questions test the same concepts but use different numbers and wording. Both students perform poorly.
They failed for opposite reasons:
- the first student did not learn enough;
- the second student learned the practice paper too specifically.
Machine-learning models can fail in exactly these two ways.
The first failure is called underfitting.
The second is called overfitting.
Between them is the goal of machine learning: generalization—learning a pattern that continues to work on new examples.
flowchart LR
Under["Underfitting: too little learned"] --> Good["Generalization: useful pattern"]
Good --> Over["Overfitting: training details memorized"]This article builds the idea slowly. We will not begin with definitions and expect them to make sense. We will watch three models learn, compare what they do on familiar and unfamiliar data, and then diagnose what went wrong.
What does it mean for a model to learn?
Suppose we want a model to estimate a house price.
We give it examples:
| Size in square feet | Age in years | Location score | Price |
|---|---|---|---|
| 700 | 12 | 6 | ₹52 lakh |
| 950 | 5 | 8 | ₹78 lakh |
| 1,200 | 8 | 7 | ₹88 lakh |
| 1,500 | 2 | 9 | ₹1.25 crore |
The first three columns are features—information the model can use. The price is the target it should predict.
During training, the model makes predictions, measures its errors and adjusts its parameters. But the purpose is not merely to reproduce these four rows.
We want it to estimate the price of a fifth house it has never seen.
That distinction is the heart of machine learning:
A useful model does not only perform well on examples it studied. It discovers a pattern that transfers to new examples from the same kind of problem.
This ability is called generalization.
Training data is the practice paper
We cannot judge the student using only the exact questions she studied. In the same way, we should not judge a model using only its training data.
We usually divide available data into separate groups:
| Dataset | Purpose | Can training learn from it? |
|---|---|---|
| Training set | Adjust model parameters | Yes |
| Validation set | Compare choices and monitor generalization | Not through ordinary weight updates |
| Test set | Final, less-biased evaluation | No |
flowchart TD
Data["Available labelled data"] --> Train["Training set"]
Data --> Validation["Validation set"]
Data --> Test["Test set"]
Train --> Learn["Learn parameters"]
Validation --> Tune["Choose model and settings"]
Test --> Report["Final evaluation"]The exact percentages vary. A small project might use 70% / 15% / 15%;
another may use cross-validation; a time-series system must usually split by
time. The important rule is separation of purpose, not one compulsory ratio.
Now we can ask two different questions:
- How well does the model fit the training examples?
- How well does it perform on unseen validation or test examples?
Underfitting and overfitting become visible only when we compare both.
Model One learns almost nothing
Suppose our first house-price model always predicts:
Every house costs ₹70 lakh.
It ignores size, age and location.
For a 700-square-foot house, ₹70 lakh may be too high. For a new 1,600-square- foot house in a strong location, it may be far too low.
Its errors are large on the training data. They are also large on new data.
| Performance | Result |
|---|---|
| Training error | High |
| Validation error | High |
This is underfitting.
The model has failed to capture enough of the real relationship between the features and the target.
Common reasons include:
- the model is too simple for the pattern;
- useful features are missing;
- training stopped too early;
- regularization is too strong;
- the learning rate or optimization setup prevents learning;
- the input representation loses important information;
- the data or labels contain serious problems.
Underfitting is sometimes described as high bias. Here, bias does not mean social unfairness. It means the model makes strong simplifying assumptions that systematically miss the pattern.
Our “every house costs ₹70 lakh” rule has very high bias.
Model Two learns the useful pattern
The second model learns broad relationships:
- larger houses usually cost more;
- newer houses may receive a premium;
- location strongly affects price;
- those factors interact.
It does not predict every training price perfectly. Real housing data contains noise: negotiation, interior quality, urgency of sale and facts not present in our columns.
But it performs reasonably well on both familiar and unfamiliar houses.
| Performance | Result |
|---|---|
| Training error | Low |
| Validation error | Low and reasonably close to training error |
This is useful generalization.
The model learned enough structure to make good predictions without treating every accidental detail as a universal rule.
Model Three remembers the neighbourhood too perfectly
The third model is extremely flexible.
Instead of learning only broad price relationships, it starts learning details such as:
- one 947-square-foot training house sold for exactly ₹77.8 lakh;
- a particular property ID was associated with an expensive transaction;
- houses photographed on a Tuesday happened to sell for more in this dataset;
- one rare postcode always appeared with premium homes;
- small errors and noise in the training labels.
It may reproduce nearly every training target.
Then a genuinely new house arrives. Its combination of size, location and age does not exactly match anything memorized. The model’s rules do not transfer.
| Performance | Result |
|---|---|
| Training error | Extremely low |
| Validation error | Much higher |
This is overfitting.
The model has learned the training data too specifically—including patterns that are accidental, noisy or irrelevant outside that sample.
Overfitting is not simply “the model learned too much.” The model learned the wrong level of detail.
A small visual example: fitting five points
Imagine five observations that roughly follow an upward trend.
A model can respond in three ways.
Too simple
It draws an almost flat line. The line misses most points and ignores the upward relationship.
training fit: poor
new-data fit: poor
result: underfittingAppropriately flexible
It draws a smooth line through the general centre of the points. It does not touch every observation, but it captures the trend.
training fit: good
new-data fit: good
result: generalizationToo flexible
It draws a complicated curve that twists through all five points. The training error becomes zero, but the curve behaves wildly between the observations.
training fit: perfect
new-data fit: poor
result: overfittingThe smooth model deliberately accepts a little training error. That is not a failure. It may be the price of avoiding a fragile rule.
The relationship becomes clearer when we place training performance and unseen-data performance on separate axes:
quadrantChart
title "Training fit is not the same as generalization"
x-axis "Poor training performance" --> "Strong training performance"
y-axis "Poor unseen-data performance" --> "Strong unseen-data performance"
quadrant-1 "Useful generalization"
quadrant-2 "Unusual: inspect the pipeline"
quadrant-3 "Underfitting"
quadrant-4 "Overfitting"
"Too-simple model": [0.25, 0.22]
"Balanced model": [0.78, 0.82]
"Memorizing model": [0.96, 0.30]The memorizing model sits far to the right because it performs extremely well on training data, but it remains low on the vertical axis because that performance does not transfer.
Training loss alone can mislead you
Suppose we train a neural network for several epochs.
An epoch means one pass through the training dataset.
At first, both training and validation loss fall:
| Epoch | Training loss | Validation loss |
|---|---|---|
| 1 | 1.20 | 1.28 |
| 2 | 0.90 | 0.97 |
| 3 | 0.68 | 0.76 |
| 4 | 0.51 | 0.64 |
| 5 | 0.39 | 0.67 |
| 6 | 0.28 | 0.75 |
| 7 | 0.19 | 0.89 |
Until epoch 4, the model improves on both datasets.
After epoch 4, training loss continues to fall—but validation loss begins to rise. The model is getting better at the practice paper while getting worse at the real exam.
That separation is a classic sign of overfitting.
xychart-beta
title "Training loss keeps falling"
x-axis "Epoch" [1, 2, 3, 4, 5, 6, 7]
y-axis "Loss" 0 --> 1.4
line [1.20, 0.90, 0.68, 0.51, 0.39, 0.28, 0.19]xychart-beta
title "Validation loss turns upward after epoch 4"
x-axis "Epoch" [1, 2, 3, 4, 5, 6, 7]
y-axis "Loss" 0 --> 1.4
line [1.28, 0.97, 0.76, 0.64, 0.67, 0.75, 0.89]Because both charts use the same axes, we can compare their shapes directly. Training loss keeps falling. Validation loss falls until epoch 4 and then turns upward.
The difference between training and validation performance is often called the generalization gap.
For a loss metric, one simple version is:
At epoch 4:
At epoch 7:
The much larger gap is a warning. But no universal gap value separates “safe” from “overfit.” Its meaning depends on the task, metric and data.
The same model can pass through all three states
“Underfitting model” and “overfitting model” can sound like two permanent kinds of software. Often, they are two stages of the same model during training.
flowchart LR
Early["Early: both losses high"] --> Useful["Useful: both losses lower"]
Useful --> Late["Late: validation worsens"]At the beginning, the parameters may be nearly random. The model underfits because it has not learned the broad relationship yet.
After useful training, both training and validation performance improve.
If training continues, the model may increasingly fit peculiarities of the training sample. Training loss keeps falling while validation loss rises.
Therefore, the diagnosis depends on more than architecture:
current parameters
+ amount of training
+ amount and variety of data
+ regularization
+ evaluation distributionChange one of these, and the same architecture may behave differently.
Underfitting has a different curve
An underfit model usually performs poorly even on the training set.
For example:
| Epoch | Training loss | Validation loss |
|---|---|---|
| 1 | 1.40 | 1.46 |
| 2 | 1.30 | 1.37 |
| 3 | 1.25 | 1.33 |
| 4 | 1.23 | 1.31 |
The two losses may be close, but both are bad.
This is why “a small training-validation gap” does not automatically mean a good model. A student who scores 25% on both practice and final exams is consistent, not successful.
The four common patterns are:
| Training performance | Validation performance | Likely interpretation |
|---|---|---|
| Poor | Poor | Underfitting or broken pipeline |
| Good | Poor | Overfitting or distribution mismatch |
| Good | Good | Useful generalization |
| Poor | Surprisingly good | Recheck metrics, split and pipeline |
Real diagnosis requires more than one number, but this table is a strong first step.
Bias and variance—without making them mysterious
Underfitting and overfitting are often explained using bias and variance.
Bias asks: does the model systematically miss the pattern?
Imagine training the “every house costs ₹70 lakh” model on several different samples. It will make similar oversimplified mistakes each time.
That is high bias.
Variance asks: does the model change too much when the training sample changes?
Imagine training the highly flexible model on five slightly different samples. Each version may build a very different twisty rule around its particular examples.
That is high variance.
| Model behaviour | Bias | Variance |
|---|---|---|
| Too simple | High | Often low |
| Balanced for the task | Manageable | Manageable |
| Too sensitive to training sample | Often low on training data | High |
This is called the bias–variance trade-off. Increasing flexibility can reduce bias while increasing sensitivity to noise. Reducing flexibility can control variance while making the model too rigid.
Modern deep learning complicates the simple textbook picture—large models can sometimes generalize surprisingly well—but the diagnostic intuition remains useful.
Why overfitting happens
Overfitting is not caused by model size alone. It appears when the learning capacity, data and training process are out of balance.
Too little representative data
A flexible model can memorize a small dataset. More importantly, a narrow dataset may not contain the variation the production system will encounter.
A fraud detector trained only on weekday card payments may struggle on weekend UPI behaviour, even if the dataset contains many rows.
Too much model capacity for the available signal
A large network or very deep decision tree can represent complicated patterns. That capacity is helpful when real complexity exists, but it also makes fitting noise possible.
Training for too long
The model may first learn broad patterns, then gradually fit exceptions and noise. This is what our validation-loss table showed.
Noisy labels
If two house prices are recorded incorrectly, a sufficiently flexible model may try to reproduce those mistakes.
Leakage
Data leakage gives the model information that would not be available at real prediction time.
For example, a loan-default model accidentally receives a “collection status” field created only after the customer has already defaulted. Validation scores may look spectacular, but the model cannot use that future information when a new application arrives.
Leakage is not ordinary overfitting, but it often produces the same suspicious pattern: excellent offline results and disappointing production performance.
Repeatedly tuning against the validation set
The validation set is not used for gradient updates, but humans can still overfit to it. If we try hundreds of configurations and keep the one with the best validation score, our choices gradually adapt to that validation set.
That is why a separate untouched test set matters.
Why underfitting happens
Underfitting is also broader than “the model is too small.”
Missing information
If house price depends heavily on location but the model only receives size, even a powerful algorithm cannot recover the missing location.
Inadequate representation
Raw inputs may hide the relationship. A linear model given only a timestamp may struggle with daily seasonality unless time is represented appropriately.
Excessive regularization
Regularization deliberately constrains learning. Too much can prevent the model from fitting genuine patterns.
Insufficient training
A neural network may still have high training loss simply because optimization has not progressed far enough.
Optimization problems
A poor learning rate, vanishing gradients or an implementation bug can look like underfitting. The model has capacity, but the training process cannot use it effectively.
A mismatched model family
A straight line cannot naturally represent a strongly curved relationship. More data does not remove that structural limitation.
How to reduce underfitting
The right fix depends on the cause.
Increase useful capacity
Use a more expressive model, add layers or units, allow a deeper tree, or add nonlinear features—only when the problem requires it.
Improve the features or representation
Give the model information that actually explains the target. For text, this may mean better embeddings; for time series, seasonality features; for house prices, location and property condition.
Train longer or improve optimization
If training loss is still falling steadily, more training may help. Also inspect the learning rate, gradient flow and optimizer settings.
Reduce excessive regularization
Weaken weight decay, reduce dropout or relax another constraint carefully.
Check the pipeline
Before redesigning the model, verify labels, preprocessing, loss calculation and metric code. A broken pipeline can imitate underfitting perfectly.
How to reduce overfitting
Overfitting has no single magic cure. We want the model to focus on stable signal rather than accidental training detail.
Get more representative data
More varied, correctly labelled examples make memorization harder and reveal which patterns repeat. Quality and coverage matter more than merely increasing row count.
Use data augmentation
Create valid variations without changing the target—for example, crops and small transformations for images or carefully designed perturbations for audio. Augmentation must preserve meaning; careless text augmentation can change the label.
Reduce model capacity
Use a smaller network, shallower tree, fewer features or simpler hypothesis class when the available signal does not justify the complexity.
Add regularization
Regularization changes training so that overly complicated solutions become less attractive.
With L2 regularization, the objective may become:
The first term measures prediction error. The second penalizes large weights. controls the strength of the penalty.
This does not mean every large weight is wrong. It encourages the optimizer to prefer a less extreme solution when two solutions fit the data similarly.
L1 regularization uses absolute values:
It can encourage some weights to become exactly or nearly zero.
Use dropout
During neural-network training, dropout randomly disables a fraction of activations. The network cannot rely too heavily on one narrow path and must build more distributed representations.
Dropout behaves differently during inference: all required units participate, with the framework handling the appropriate scaling convention.
Stop at the right time
Early stopping monitors validation performance. If training loss continues to improve while validation performance stops improving, we keep the checkpoint from the best validation point rather than the final epoch.
In our example, that would likely be epoch 4—not epoch 7.
Use cross-validation when data is limited
Cross-validation rotates which subset is held out. It gives a more stable view of performance than trusting one lucky split, though it costs more computation.
Remove leakage and duplicates
Near-duplicate examples appearing in both training and validation sets can make generalization look better than it is. Group related records before splitting.
Regularization is a dial, not a switch
Suppose we gradually increase regularization:
too little → model may memorize
appropriate → model focuses on stable patterns
too much → model cannot learn enoughA technique used to fight overfitting can create underfitting when applied too strongly.
The same is true of model size and training duration. These are not moral categories where “smaller” or “more training” is always better. They are controls that must match the problem.
A production example: fraud detection
Imagine a payment company trains a model to identify fraudulent transactions.
An underfit fraud model
It uses only transaction amount:
if amount > ₹50,000 → suspicious
otherwise → safeIt misses smaller coordinated fraud and wrongly blocks many legitimate large payments. Training and validation performance are both weak.
An overfit fraud model
It learns that several fraudulent training transactions occurred from one specific device ID at 2:13 AM. It treats that exact combination as the rule.
Criminals use new devices and times. The model’s excellent offline score does not survive changing behaviour.
A better-generalizing model
It learns broader behavioural signals:
- unusual amount relative to this customer’s history;
- rapid transactions across distant locations;
- new device combined with unusual recipient behaviour;
- patterns across linked accounts;
- velocity and timing changes.
Production adds another challenge: data drift. Fraud patterns, customer behaviour and payment products change. A model that generalized last year may degrade today even if it was not originally overfit.
That is why deployed systems monitor performance, input distributions and business outcomes—not only the score from an old test set.
What overfitting looks like in LLM fine-tuning
The same idea appears when adapting a language model.
Suppose we fine-tune an LLM on 500 customer-support examples.
An underfit result may:
- ignore the desired answer format;
- fail to learn product terminology;
- respond much like the original base model;
- perform poorly on both training-like and new support questions.
An overfit result may:
- reproduce training answers almost word for word;
- work only when the user’s wording resembles the fine-tuning set;
- insert memorized customer details;
- lose useful general abilities;
- become overconfident about narrow patterns.
Useful evaluation should therefore contain unseen prompts, paraphrases, edge cases and adversarial variations—not copies of training examples.
Fine-tuning is not the only place this matters. An embedding model can overfit to a narrow benchmark. A reranker can learn annotation quirks. A prompt can be manually tuned until it performs beautifully on a small fixed evaluation set and fails on real traffic.
RAG does not automatically solve overfitting. Retrieval can provide fresh evidence and reduce the need to store changing facts in model weights, but the retriever, reranker, prompts and evaluation process can still be tuned too specifically.
Overfitting is not the same as memorization—but they are related
Memorization means retaining specific training examples or details. Overfitting means poor generalization caused by fitting the training sample too specifically.
A model can memorize some rare examples while still generalizing well overall. It can also overfit through unstable rules without reproducing entire examples verbatim.
So we should not treat the words as exact synonyms.
For generative models, memorization also raises privacy and copyright concerns. Testing only average task accuracy may miss those risks.
Your validation set must resemble the real future
A random split is not always safe.
Time-dependent data
If predicting next month’s sales, randomly mixing future rows into training can make evaluation unrealistic. Train on earlier periods and validate on later ones.
Multiple records from one user
If the same customer’s near-identical transactions appear on both sides, the model may recognize the customer rather than generalize. Split by customer when that matches production use.
Images from the same source
Several frames from one video are almost duplicates. Randomly splitting frames can place near-identical images in training and validation.
Distribution mismatch
If production serves Marathi and English users but validation contains only English, a good validation score says little about the missing population.
The right question is not merely “did we hold out 20%?” It is:
Does the held-out data represent the genuinely unseen situations the model will face?
Build all three behaviours in Python
Let us create a small experiment instead of only describing the curves.
We will treat x as a normalized house-size value:
x = 0.0 → approximately 600 square feet
x = 1.0 → approximately 1,800 square feetThe teaching dataset contains only 18 training houses. Prices generally rise with size, while location, condition and market noise create variation.
We will fit three polynomial models:
| Model | Degree | What it can draw |
|---|---|---|
| Too simple | 1 | One straight line |
| Balanced | 3 | A smooth curve |
| Too flexible | 10 | A highly twisting curve |
A polynomial degree is the highest power of x the model can use. A
degree-1 model uses a rule such as:
A degree-3 model can use:
Higher degree means more flexibility. It does not automatically mean better predictions.
import numpy as np
import matplotlib.pyplot as plt
rng = np.random.default_rng(7)
def true_price(x):
"""Teaching relationship; output is approximately in ₹ lakh."""
return 50 + 30 * x + 10 * np.sin(2 * np.pi * x)
# Eighteen houses used for learning.
x_train = np.sort(rng.uniform(0, 1, 18))
y_train = true_price(x_train) + rng.normal(0, 2.5, 18)
# A separate set of houses not used to fit the models.
x_validation = np.sort(rng.uniform(0, 1, 120))
y_validation = true_price(x_validation) + rng.normal(0, 2.5, 120)
def mean_squared_error(actual, predicted):
return np.mean((actual - predicted) ** 2)
models = {
"Underfitting — degree 1": 1,
"Useful fit — degree 3": 3,
"Overfitting — degree 10": 10,
}
x_plot = np.linspace(0, 1, 400)
figure, axes = plt.subplots(1, 3, figsize=(15, 4), sharey=True)
for axis, (name, degree) in zip(axes, models.items()):
# Learn polynomial coefficients only from the training houses.
coefficients = np.polyfit(x_train, y_train, degree)
train_predictions = np.polyval(coefficients, x_train)
validation_predictions = np.polyval(coefficients, x_validation)
plot_predictions = np.polyval(coefficients, x_plot)
train_mse = mean_squared_error(y_train, train_predictions)
validation_mse = mean_squared_error(
y_validation,
validation_predictions,
)
print(
f"{name:27s} "
f"train MSE={train_mse:7.2f} "
f"validation MSE={validation_mse:7.2f}"
)
axis.scatter(x_train, y_train, color="navy", label="Training houses")
axis.plot(x_plot, true_price(x_plot), "--", color="gray", label="Broad pattern")
axis.plot(x_plot, plot_predictions, color="crimson", label="Model")
axis.set_title(name)
axis.set_xlabel("Normalized house size")
axis.grid(alpha=0.2)
axes[0].set_ylabel("Price in ₹ lakh")
axes[0].legend()
plt.tight_layout()
plt.show()The expected errors are approximately:
Underfitting — degree 1 train MSE= 29.60 validation MSE= 30.25
Useful fit — degree 3 train MSE= 4.97 validation MSE= 8.94
Overfitting — degree 10 train MSE= 1.87 validation MSE= 459.14Let us interpret them carefully.
Degree 1: underfitting
The straight line cannot reproduce the curved relationship. Training MSE is high, and validation MSE is also high.
Degree 3: useful generalization
The model captures the broad curve. Its training error is low, and its validation error remains reasonably low.
Validation MSE is not identical to training MSE because validation contains new houses and fresh noise. A healthy model is not required to produce identical numbers on both sets.
Degree 10: overfitting
This model achieves the lowest training error: 1.87.
If we looked only at training data, we would declare it the winner.
But its validation error explodes to approximately 459.14. The curve bends
around the 18 training observations and behaves wildly where training evidence
is sparse.
This experiment captures the central lesson numerically:
The model with the smallest training error is not necessarily the model we should deploy.
The exact results depend on the generated sample, random seed and library versions. The fixed seed makes this run reproducible, but the comparison—not the last decimal place—is what matters.
A practical diagnosis checklist
When a model performs poorly, work in this order.
1. Establish a simple baseline
Compare against a mean prediction, majority class, rule-based method or small model. Without a baseline, “good” and “bad” lack context.
2. Verify the data and metric
Inspect labels, duplicates, class imbalance, preprocessing and metric implementation. Confirm that the metric matches the business goal.
3. Compare training and validation performance
training poor + validation poor → investigate underfitting or pipeline issues
training strong + validation poor → investigate overfitting or mismatch4. Plot learning curves
Track both datasets across epochs or across increasing amounts of training data. One final score hides the direction of change.
5. Examine errors by slice
Average performance may hide failure for new users, rare classes, languages, locations, devices or time periods.
6. Change one meaningful variable at a time
Try more data, different capacity, regularization or improved features based on the diagnosis. Changing everything together makes the result hard to explain.
7. Protect the test set
Use it for final evaluation, not repeated daily tuning. Once your decisions adapt to it, it is functioning like another validation set.
Common misunderstandings
“A 99% training score means the model is excellent”
It only proves that the model fits data it has seen. Generalization requires evaluation on unseen, representative data.
“Overfitting happens only with small datasets”
Small or narrow data increases the risk, but leakage, duplicates, excessive tuning and distribution mismatch can cause misleading generalization at larger scales too.
“A large model is automatically overfit”
Capacity creates the ability to fit complexity and noise; it does not guarantee poor generalization. Data scale, optimization, regularization and architecture all matter.
“More data always fixes overfitting”
More representative, high-quality data often helps. More duplicated, biased or incorrect data may not.
“Validation loss must always be lower than training loss”
Not necessarily. Training may include dropout, augmentation or regularization terms that are absent during evaluation. Focus on correctly computed, comparable metrics and their trend.
“If training and validation scores are close, the model is good”
Both can be equally poor. Always compare them with a useful baseline and task requirement.
“Early stopping finds the perfect model”
It is one regularization and checkpoint-selection technique. It cannot repair leakage, unrepresentative data or a fundamentally wrong objective.
The one idea to remember
Underfitting and overfitting are not mainly about how impressive a model looks on its training data.
They are about what happens when the model leaves familiar territory.
Underfitting
The model did not capture enough real structure.
Training performance: poor
Unseen-data performance: poor
Useful generalization
The model captured patterns that repeat.
Training performance: good
Unseen-data performance: good
Overfitting
The model fitted the training sample too specifically.
Training performance: excellent
Unseen-data performance: poorThe purpose of training is not to produce the smallest possible training loss.
The purpose is to build a model that makes useful predictions after the practice examples are gone.
That is generalization—and it is the real test every machine-learning model must pass.
Sources and further reading
Continue reading
How AI Works
Attention: How an LLM Decides Which Words Matter Right Now
A slow, number-by-number explanation of attention—from context and Query, Key and Value vectors to dot products, masking, softmax, multi-head attention and KV cache.
◷ 21 min read
How AI Works
Where Does an LLM Store ‘Paris Is the Capital of France’?
Follow one page from training data into tokens, gradients and model weights—and then watch those learned parameters answer a simple question.
◷ 18 min read
How AI Works
The Model Was Wrong. Which Weight Should We Blame?
A slow, number-by-number journey through backpropagation—from a loss of 0.782 to gradients, updated weights and a better second prediction.
◷ 14 min read