Loss Functions – Deep Learning

Begin with the central question

How can a network improve if it cannot measure how wrong it is?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

prediction + correct answer → loss number → learning signal

Before you continue: three tools for this module

Target: the answer the training example says is correct.
Probability: a number from 0 to 1 representing confidence.
Natural logarithm (log): a mathematical function used here to punish confident wrong answers much more strongly than uncertain ones.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

How a network quantifies “how wrong was that prediction?” — MSE and MAE for regression, binary and categorical cross-entropy for classification — with every formula worked through a small numeric example.

Loss connects a prediction to learning:

prediction + target → loss function → one error number
                                         ↓
                              backpropagation computes gradients

The chosen loss defines what training rewards. MSE emphasizes large numerical errors; cross-entropy rewards probability assigned to the correct class. Loss is an optimization signal, while evaluation metrics express practical quality.

Why a Network Needs a Numerical Measure of Error

Module 5 showed forward propagation producing a prediction. But a prediction alone is useless for learning — the network needs a single number expressing exactly how wrong it was, in a form that later steps (backpropagation, Module 7) can actually use to improve. That number is the loss.

Turning “Wrong” into a Measurable Signal

the model doesn’t directly know that it is wrong. The loss function quantifies how wrong it is. Prediction tells us what the model thinks; loss tells us how far that is from reality — a single number the rest of training exists to minimize.

Analogy: The Tape Measure vs. The Confident Hot-or-Cold Game

Regression Loss (The Tape Measure): Think of throwing a dart at a board.

MAE (Mean Absolute Error): You use a standard ruler to measure the absolute physical distance in inches from where your dart landed to the bullseye ( $|y - \hat{y}|$ ). If you are 5 inches away, your penalty score is 5.

MSE (Mean Squared Error): You square the measurement ( $[y - \hat{y}]^2$ ). If you are 1 inch away, your penalty is 1. If you are 5 inches away, your penalty is 25! MSE heavily punishes wild, outlier throws to force the model to stay close to all targets.

Classification Loss (The Confident Hot-or-Cold Game): You play a game of search with a friend, but you are allowed to declare your confidence level.

Cross-Entropy: You guess “The key is hidden in the drawer (95% confidence)”. If the key is indeed in the drawer, your friend nods quietly (low loss, $L \approx 0.05$ ).

But if you confidently shout “The key is 100% in the trash!” and the key is actually in the drawer, your friend screams at you (extremely high exponential loss penalty, $L > 2.30$ ). Cross-entropy heavily penalizes overconfident wrong guesses.

📊 Visual Flowchart: Regression vs. Classification Loss Paths

Here is the split in execution pathways depending on whether your network outputs a continuous number or a probability distribution:

graph TD
    subgraph LossPathways ["Loss Function Selection Tree"]
        TaskType{"What is the target<br>output type?"}

TaskType -->|Continuous Numbers / Price / Temp| Regression["Regression Task"]
        TaskType -->|Probability / Categories / Tokens| Classification["Classification Task"]

Regression --> MSE["MSE (Mean Squared Error)<br>L = mean((y - y_pred)²)<br>Amplifies large errors"]
        Regression --> MAE["MAE (Mean Absolute Error)<br>L = mean(|y - y_pred|)<br>Robust to outliers"]

Classification --> BCE["Binary Cross-Entropy (BCE)<br>L = -mean(y*log(p) + (1-y)*log(1-p))<br>Used for 2 classes"]
        Classification --> CCE["Categorical Cross-Entropy (CCE)<br>L = -Σ(y_true * log(p))<br>Used for multi-class / LLM next-token"]
    end

4. Core Concept

Loss	Formula	Used for
MSE (Mean Squared Error)	`mean((prediction − true)²)`	Regression
MAE (Mean Absolute Error)	`mean(	prediction − true
Binary cross-entropy	`−mean(y·log(p) + (1−y)·log(1−p))`	Binary classification
Categorical cross-entropy	`−Σ(y_true_onehot · log(p))`	Multi-class classification

⚠️ Mistake

Incorrect idea: the loss function itself changes the weights.

Why it is incorrect: It doesn’t. The loss function only measures wrongness. Backpropagation (Module 7) computes gradients from the loss, and the optimizer (Module 9) uses those gradients to actually change weights. The loss is a measurement, not an update mechanism.

5. How It Works — Step by Step

1. The network computes a prediction (forward propagation, Module 5)
2. The loss function compares this prediction to the TRUE, known
   answer for that training example
3. The loss function outputs a SINGLE NUMBER: how wrong this
   prediction was
4. This number (specifically, its GRADIENT) is what
   backpropagation (Module 7) uses to figure out which direction
   to adjust every parameter

6. Mathematical Intuition

Read the mathematics as a story

A loss function converts the gap between prediction and target into one number. Training tries to reduce that number; the choice of loss defines what ‘better’ mathematically means.

prediction + correct answer → loss number → learning signal

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. MSE and MAE, worked on 3 predictions:

predictions = [3.2, 5.5, 2.8]
true_values = [3.0, 6.0, 2.5]

errors        = [0.2, -0.5, 0.3]
squared_errors = [0.04, 0.25, 0.09]
MSE = mean(squared_errors) = (0.04+0.25+0.09)/3 = 0.1267

abs_errors = [0.2, 0.5, 0.3]
MAE = mean(abs_errors) = (0.2+0.5+0.3)/3 = 0.3333

Binary cross-entropy, worked on one example (true=1, predicted probability=0.95):

loss = -(1 × log(0.95) + 0 × log(0.05))
     = -log(0.95)
     ≈ 0.0513

A small loss for a confident, correct prediction. Compare a confident, wrong prediction (true=1, predicted probability=0.10):

loss = -(1 × log(0.10) + 0 × log(0.90))
     = -log(0.10)
     ≈ 2.303

Every variable: y is the true label (0 or 1); p is the model’s predicted probability of class 1; log is the natural logarithm. Notice how sharply the loss increases for a confident wrong answer versus a confident correct one — cross-entropy specifically punishes confident mistakes far more than uncertain ones.

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. If a spam classifier predicts 0.95 probability of spam for a genuinely spam email, its binary cross-entropy loss for that example is small (~0.05).

If it predicts 0.10 for that same genuinely-spam email, its loss balloons to ~2.3 — over 40x larger, for what looks like a “smaller” numeric miss (0.85 away from truth either way isn’t the comparison — cross-entropy specifically penalizes confidence in the wrong direction non-linearly).

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of Loss Functions.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

# Regression: MSE and MAE
predictions = np.array([3.2, 5.5, 2.8])
true_values = np.array([3.0, 6.0, 2.5])

mse = np.mean((predictions - true_values) ** 2)
mae = np.mean(np.abs(predictions - true_values))
print("MSE:", mse)
print("MAE:", mae)

# Binary cross-entropy
def binary_cross_entropy(y_true, y_pred):
    epsilon = 1e-15   # avoids log(0)
    y_pred = np.clip(y_pred, epsilon, 1 - epsilon)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

y_true_binary = np.array([1, 0, 1])
y_pred_confident_correct = np.array([0.95, 0.05, 0.90])
y_pred_confident_wrong = np.array([0.10, 0.90, 0.15])

print("\nBCE (confident + correct):", binary_cross_entropy(y_true_binary, y_pred_confident_correct))
print("BCE (confident + WRONG):", binary_cross_entropy(y_true_binary, y_pred_confident_wrong))

# Categorical cross-entropy (single example, 3 classes)
def categorical_cross_entropy(y_true_onehot, y_pred_probs):
    epsilon = 1e-15
    y_pred_probs = np.clip(y_pred_probs, epsilon, 1)
    return -np.sum(y_true_onehot * np.log(y_pred_probs))

true_class_onehot = np.array([0, 1, 0])       # true class is index 1
predicted_probs = np.array([0.1, 0.7, 0.2])   # from a softmax output
cce = categorical_cross_entropy(true_class_onehot, predicted_probs)
print("\nCategorical cross-entropy:", cce)

Expected Output:

MSE: 0.12666666666666668
MAE: 0.3333333333333333

BCE (confident + correct): 0.06931570147764247
BCE (confident + WRONG): 2.167430056957991

Categorical cross-entropy: 0.35667494393873245

9. How It Works

MSE/MAE match Section 6’s hand calculations (0.1267, 0.3333).
Note MSE (0.127) is smaller than MAE (0.333) here mainly because MSE’s squaring shrinks already-small errors (all errors here are under
1. — with larger errors, squaring would instead amplify them relative to MAE.
BCE for the confident-correct batch (≈0.069) is over 30x smaller than the confident-wrong batch (≈2.17) — the exact “confident mistakes are punished heavily” behavior demonstrated by hand in Section 6, now averaged across a small batch of 3 examples.
categorical_cross_entropy reduces, when there’s only one true class (one-hot encoded), to simply −log(predicted probability of the true class) — here −log(0.7) ≈ 0.357, confirming the formula.

10. Real-World Example

Training an image classifier to distinguish 10 digit classes (0-9) uses categorical cross-entropy: for each training image, the network outputs 10 probabilities (via softmax, Module 4), and the loss compares that distribution against the one true digit label. A house-price regression model instead uses MSE or MAE, since the label is a continuous number, not a category.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

During LLM pretraining, token-level cross-entropy compares the predicted probability distribution with the actual next token. Product quality is not identical to training loss, so teams also evaluate factuality, safety, usefulness, and task success.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

LLM pretraining uses categorical cross-entropy at every single token position: at each point in a training sentence, the model outputs a probability distribution over its entire vocabulary (often 50,000+ possible tokens, via softmax), and cross-entropy compares that distribution against the actual next token that appeared in the real text. This is mechanically identical to Section 8’s categorical_ cross_entropy function — just computed across a vocabulary of tens of thousands of classes instead of 3, at every token position, across the entire training corpus.

Loss	AI application
Categorical cross-entropy	LLM next-token prediction (pretraining); any multi-class classifier
Binary cross-entropy	Content moderation/safety classifiers, binary intent classifiers
MSE	Reward models in RLHF (predicting a continuous quality score); regression-style scoring components

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Moderate. Any fine-tuned classifier inside an agent pipeline — an intent router, a safety filter — is trained using exactly binary or categorical cross-entropy, the same mechanism covered here at small scale.

When an agent’s underlying LLM was itself fine-tuned (Module 16 of the ML course), that fine-tuning process is still, at its core, minimizing cross-entropy loss on the fine-tuning dataset’s token sequences.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: the loss function changes the weights.

Why it is incorrect: As stated in Section 4 — it doesn’t. It only measures wrongness; backpropagation and the optimizer (Modules 7-9) do the actual updating.

⚠️ Mistake

Incorrect idea: MSE and cross-entropy are interchangeable.

Why it is incorrect: Using MSE for classification (or cross-entropy for regression) produces a poorly-behaved training signal — each is mathematically designed for its specific output type (continuous numbers vs. probability distributions).

⚠️ Mistake

Incorrect idea: a lower loss always means a “better” model in every practical sense.

Why it is incorrect: Loss is a proxy for what you actually care about. Your ML course’s evaluation metrics module covers cases where strong loss/accuracy numbers can still mean a practically poor model.

14. Important Distinctions

MSE	MAE
Squares errors — penalizes large errors much more	More robust to outliers
Smoother gradients for optimization	Less sensitive to a few extreme errors

Binary Cross-Entropy	Categorical Cross-Entropy
Two classes, one probability output (sigmoid)	Multiple classes, a full distribution (softmax)
`y` is a single 0/1 value	`y` is a one-hot vector

15. When to Use

Use MSE/MAE for regression tasks (predicting a continuous value). Use binary cross-entropy for two-class classification. Use categorical cross-entropy for multi-class classification — including, at enormous scale, next-token prediction in LLM training.

16. When Not to Use

Don’t use MSE for classification outputs (probabilities) — cross-entropy is specifically designed to work well with probability outputs and produces a much more useful training signal. Don’t use MAE as a default without considering whether you actually want large errors penalized more heavily (in which case MSE is the better choice).

17. Interview Questions

Beginner

Q: What is a loss function, and why does a network need one?

Ans: A loss function measures how wrong a network’s prediction is compared to the true answer, producing a single number. It’s essential because training is a search for parameter values that minimize this number — the network has no other way of “knowing” it was wrong.

Intermediate

Q: Why is cross-entropy used for classification instead of MSE?

Ans: Cross-entropy is specifically designed to work with probability outputs — it heavily penalizes confident wrong predictions (as shown numerically in Section 6, where a confident wrong prediction’s loss is over 40x a confident correct prediction’s loss) in a way that produces a much more useful training signal for classification than MSE, which isn’t designed around probability semantics.

Advanced

Q: Why does cross-entropy loss grow so sharply as a model’s predicted probability for the true class approaches zero?

Ans: Because of the logarithm in its formula: as predicted probability for the correct class approaches 0, −log(p) approaches infinity. This means the model is heavily penalized specifically for being confidently wrong, encouraging it to express appropriate uncertainty rather than being confidently incorrect — a deliberate mathematical property, not an incidental one.

Scenario

Q: You’re training a next-word prediction model and notice the loss decreases steadily during training but the model’s generated text still seems repetitive and low-quality. What might this indicate?

Ans: Loss decreasing means the model is getting better at matching the statistical patterns in the training data’s next-token distribution — but low loss doesn’t automatically guarantee subjectively good generated text (repetitiveness is a known separate issue, often related to how tokens are sampled at inference time — Module 17 — rather than the training loss itself).

I’d separately evaluate actual generated output quality, not rely on the loss number alone as a proxy for “good text.”

AI Engineering

Q: Concretely, what is cross-entropy loss comparing during LLM pretraining, at each step?

Ans: At each token position in a training sequence, the model outputs a probability distribution (via softmax) over its entire vocabulary, representing its prediction for the next token.

Cross-entropy compares this predicted distribution against the actual next token that appeared in the real training text (treated as the correct class, with probability 1), producing a loss value used to adjust the model’s parameters via backpropagation and gradient descent (Modules 7-8) — mechanically identical to this module’s categorical cross-entropy example, just computed over a vocabulary of tens of thousands of tokens instead of 3.

18. What You Should Remember

The loss function is how a network measures wrongness — it never directly changes weights itself.
MSE/MAE for regression; binary/categorical cross-entropy for classification.
Cross-entropy specifically, and heavily, penalizes confident wrong predictions — a deliberate design property from its logarithmic form.

19. How This Helps Me Build AI Systems

Cross-entropy is the exact loss function every LLM’s pretraining process minimizes, at every one of trillions of token positions. You’ve now computed it by hand, in code, at small scale — the mechanism scaling up to LLM pretraining is identical, not conceptually different.

Next: Module 7 — Backpropagation — the must-know module: how the loss value actually tells every parameter in the network which direction to change.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026