Forward Propagation | TechByteByByte

Begin with the central question

When data enters a trained network, what exact journey produces the answer?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

input → layer 1 calculation → layer 2 calculation → output

Before you continue: three tools for this module

Matrix: a rectangular grid of numbers.
Matrix multiplication: a compact way to calculate many weighted sums together.
Inference: using learned parameters to produce an answer without training them.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

The complete forward pass: how one input travels through every layer of a network, transformed by weighted sums and activations at each step, until it produces a final prediction. You’ll trace this by hand through a tiny network, then confirm every number in code.

A forward pass follows one direction:

input x → W₁x+b₁ → activation → W₂a₁+b₂ → output
                    hidden state              prediction

During inference, learned weights and biases stay fixed while data moves forward. During training, the same forward pass produces predictions needed to calculate loss before gradients are sent backward.

Why Every Prediction Needs a Forward Pass

Modules 2-4 gave you all the individual pieces: neurons, layers, and activation functions. Forward propagation is simply the name for what happens when you run these pieces in sequence on a real input — it’s the mechanism every single prediction from every neural network (feed- forward, CNN, RNN, or Transformer) is built from.

The Layer-by-Layer Assembly Line

Forward propagation is an assembly line. Raw material (your input) enters at one end. Each station (layer) does its fixed job — weighted sum, then activation — and passes its output to the next station. By the time the material reaches the end of the line, it’s been transformed into the finished product (the prediction).

Analogy: The Factory Assembly Line Think of forward propagation as a physical manufacturing assembly line building a car:

Input ( $x$ ): The raw steel chassis enters the factory.

Station 1 (Hidden Layer 1): The machinery welds the frame ( $W_1$ ), adds structural nuts ( $b_1$ ), and sprays an base anti-rust coating (activation $a_1$ ). The output is a primed chassis.

Station 2 (Hidden Layer 2): Prime chassis ( $a_1$ ) moves down the line. Robot arms mount the doors ( $W_2$ ), bolt the console ( $b_2$ ), and seal the windows (activation $a_2$ ).

Station 3 (Output Layer): The intermediate vehicle ( $a_2$ ) enters the final bay. Wheels are attached, and the inspection label is stuck on the glass (final activation leading to output prediction).

Each station depends only on the exact output of the station immediately preceding it. The raw steel chassis cannot skip straight to the wheel mounting bay.

📊 Visual Flowchart: Forward Propagation Dataflow

Here is how numerical activations propagate sequentially through the network layers:

graph TD
    Input["Input Vector x<br>[1.0, 2.0]"] --> L1Sum["1. Hidden Layer Summation (z1)<br>z1 = W1 * x + b1<br>[0.2, 0.85]"]
    L1Sum --> L1Act["2. Hidden Layer Activation (a1)<br>a1 = ReLU(z1)<br>[0.2, 0.85]"]

L1Act --> L2Sum["3. Output Layer Summation (z2)<br>z2 = W2 * a1 + b2<br>-0.17"]
    L2Sum --> L2Act["4. Output Layer Activation (a2)<br>a2 = Sigmoid(z2)<br>0.4576"]

L2Act --> Predict["5. Final Prediction (y_pred)<br>45.76% probability"]

4. Core Concept

Input
  ↓
Linear transformation:  z = W @ x + b
  ↓
Activation:              a = activation_function(z)
  ↓
(a becomes the input to the NEXT layer — repeat)
  ↓
...
  ↓
Output layer's final activation = the prediction

z (sometimes called a “logit” at the very final layer, before a classification activation) is the raw weighted sum — Module 2’s computation.
a is the activation — the value actually passed to the next layer (Module 4).
This entire chain, from raw input to final prediction, is forward propagation.

5. How It Works — Step by Step

1. Input vector x enters the network
2. Layer 1 computes z1 = W1 @ x + b1
3. Layer 1 applies its activation: a1 = activation(z1)
4. a1 becomes the input to Layer 2
5. Layer 2 computes z2 = W2 @ a1 + b2
6. Layer 2 applies ITS activation: a2 = activation(z2)
7. Repeat for every remaining layer
8. The FINAL layer's activation is the network's prediction

6. Mathematical Intuition

Read the mathematics as a story

Forward propagation is evaluation from left to right. Each layer uses the previous layer’s numbers; no weights are changed during this journey.

input → layer 1 calculation → layer 2 calculation → output

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. A tiny, complete network: 2 inputs → hidden layer (2 neurons, ReLU) → output layer (1 neuron, sigmoid) — worked entirely by hand, then verified.

x = [1.0, 2.0]

Hidden layer:
  W1 = [[0.3, -0.1],
        [0.5,  0.2]]
  b1 = [0.1, -0.05]

z1[0] = (0.3 × 1.0) + (-0.1 × 2.0) + 0.1 = 0.3 - 0.2 + 0.1 = 0.2
  z1[1] = (0.5 × 1.0) + (0.2 × 2.0) + (-0.05) = 0.5 + 0.4 - 0.05 = 0.85

a1 = ReLU(z1) = [ReLU(0.2), ReLU(0.85)] = [0.2, 0.85]
  (both positive, so ReLU passes them through unchanged)

Output layer:
  W2 = [0.6, -0.4]
  b2 = 0.05

z2 = (0.6 × 0.2) + (-0.4 × 0.85) + 0.05
     = 0.12 - 0.34 + 0.05
     = -0.17

a2 = sigmoid(-0.17) = 1 / (1 + e^0.17) ≈ 0.4576

Every variable: x is the raw input. W1/b1 and W2/b2 are each layer’s learned parameters (Module 2). z1/z2 are the raw weighted sums (“logits” at the final layer). a1 is the hidden layer’s activated output, which becomes the input to the output layer. a2 ≈ 0.4576 is the network’s final prediction — here, interpretable as roughly a 45.8% predicted probability of the positive class.

7. Simple Example

Walk through the example

Read the example in three passes:

Identify the input numbers and what each number represents.
Follow one operation at a time instead of jumping directly to the answer.
Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. If this tiny network were a spam classifier, x = [1.0, 2.0] might represent two normalized features (say, link count and urgency-word count).

The hidden layer combines them into two new internal representations (0.2 and 0.85) — not directly interpretable as “link count” or “urgency” anymore, just useful intermediate values the network found helpful. The output layer combines those into a final 0.4576 prediction — just under 50%, meaning this network currently leans slightly toward “not spam” for this input.

8. Python Example

Three Python symbols used below

NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
np.array(...) creates a numeric vector or matrix.
@ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters.

# Build a tiny, inspectable example of Forward Propagation.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

def relu(z): return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-z))

# The exact network from Section 6
x = np.array([1.0, 2.0])

W1 = np.array([
    [0.3, -0.1],
    [0.5,  0.2],
])
b1 = np.array([0.1, -0.05])

z1 = W1 @ x + b1
print("Hidden layer weighted sums (z1):", z1)
a1 = relu(z1)
print("Hidden layer activations (a1):", a1)

W2 = np.array([[0.6, -0.4]])
b2 = np.array([0.05])

z2 = W2 @ a1 + b2
print("Output weighted sum (z2, logit):", z2)
a2 = sigmoid(z2)
print("Final output (prediction):", a2)

Expected Output:

Hidden layer weighted sums (z1): [0.2  0.85]
Hidden layer activations (a1): [0.2  0.85]
Output weighted sum (z2, logit): [-0.17]
Final output (prediction): [0.45760206]

9. How It Works

Every number matches Section 6’s hand calculation exactly: z1 = [0.2, 0.85], unchanged by ReLU since both are positive; z2 = -0.17; and the final sigmoid output ≈ 0.4576.

This confirms, concretely, that “forward propagation” is nothing more than repeatedly applying z = W @ x + b followed by an activation function, layer after layer — you’ve now traced that entire chain by hand and in code, for a network with real (if small) weights.

10. Real-World Example

When you send a prompt to an LLM, the entire response-generation process is forward propagation — repeated, once per generated token — through a network with dozens of layers, each doing exactly this same z = W @ x + b → activation sequence, just at vastly larger scale (thousands of dimensions per vector, dozens of layers, and — for Transformers specifically — an additional attention computation per layer, Module 15).

The mechanism is identical to Section 8’s tiny example; only the scale and the specific layer architecture differ.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

Every prompt sent to an LLM triggers forward propagation. Token representations pass through the model’s layers, raw output scores are produced, and a decoding rule selects a next token; the cycle repeats for later tokens.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

“Running inference” on any neural network — including asking an LLM a question — is forward propagation. Nothing else happens during a single generation step: input flows forward through the network’s layers, producing an output. (Module 17 traces this exact process specifically for how an LLM turns tokens into a next-token prediction.)

Concept	Where it shows up
`z = W @ x + b`	Every linear layer in every neural network, including the projections inside attention (Module 15)
Layer-by-layer propagation	How data moves through an image model, a text model, and an LLM alike
Final layer’s output	For image models, class probabilities. For LLMs, next-token probabilities (via softmax, Module 4)

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Moderate, indirectly but constantly. Every single LLM call an agent makes — every reasoning step, every tool- selection decision the LLM itself makes — is one (or several, for multi-token output) forward propagation pass through the underlying model.

Understanding this mechanism is what separates “the agent just decided this, somehow” from a genuine grasp of what’s computationally happening at each step.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: forward propagation involves any learning or weight updates.

Why it is incorrect: It doesn’t — forward propagation only computes an output using the network’s current weights. Learning (updating weights) happens through backpropagation and gradient descent (Modules 7-8), which are entirely separate, subsequent steps.

⚠️ Mistake

Incorrect idea: “logit” and “probability” mean the same thing.

Why it is incorrect: A logit (z2 above) is the raw, unbounded weighted sum before a final activation. Only after applying sigmoid or softmax (Module 4) does it become an interpretable probability, bounded between 0 and 1.

⚠️ Mistake

Incorrect idea: each layer’s activation values carry forward unchanged to every later layer.

Why it is incorrect: Each layer only sees the immediately preceding layer’s output — Layer 2 never directly sees the raw input x, only a1. Information from earlier layers only persists to the extent it survives being encoded into each subsequent layer’s activations.

14. Important Distinctions

Logit	Probability
Raw weighted sum, before final activation	After sigmoid/softmax — bounded, interpretable
Can be any real number	Always between 0 and 1

Forward Propagation	Backpropagation
Input → output, computing a prediction	Output error → input, computing gradients (Module 7)
Happens during BOTH training and inference	Happens ONLY during training

15. When to Use

Forward propagation isn’t optional or situational — it’s simply what “running the network” means, whether you’re training or doing inference. Understanding it precisely matters most when debugging unexpected model output: tracing shapes and values layer by layer is the standard technique for finding where a computation is going wrong.

16. When Not to Use

Not applicable — this is a core mechanism, not a technique with alternatives.

17. Interview Questions

Beginner

Q: What is forward propagation?

Ans: The process of passing an input through a neural network’s layers, in order, computing each layer’s weighted sum and activation in sequence, until the final layer produces the network’s output (prediction).

Intermediate

Q: What’s the difference between a logit and a probability in the context of forward propagation?

Ans: A logit is a layer’s raw weighted sum output — an unbounded real number, with no inherent probabilistic meaning. It becomes a probability only after passing through a final activation function like sigmoid (bounding it to (0,1)) or softmax (converting a whole vector of logits into a distribution that sums to 1).

Advanced

Q: Why does each layer in forward propagation need both a linear transformation and an activation function, in that specific order?

Ans: The linear transformation (z = W @ x + b) computes a weighted combination of the previous layer’s outputs — this is where the layer’s learned parameters do their work. The activation function then introduces non-linearity (Module 4), which is what gives multi-layer networks genuine representational power beyond a single linear transformation (proven directly in Module 4).

Skipping the activation step, or applying it before the linear transformation, breaks this specific mechanism.

Scenario

Q: You’re debugging a network that always outputs the same prediction regardless of input. Using forward propagation, how would you investigate?

Ans: I’d trace each layer’s output values (z and a) for several different inputs, checking where the values stop varying.

If a hidden layer’s activations are identical (or all zero) across genuinely different inputs, that layer — or an earlier one — is likely the problem: possibly dead ReLU neurons (Module 4), weights that were never properly initialized or updated, or a bug feeding the same data repeatedly regardless of the actual input.

AI Engineering

Q: When an LLM generates a response, is that training or inference, and what specifically is happening at each generated token?

Ans: It’s inference. For each token the model generates, the current sequence of tokens flows forward through the model’s layers — exactly this module’s forward propagation, at LLM scale — producing a probability distribution over the entire vocabulary (via a final softmax, Module 4) for what the next token should be.

No weights are updated during this process; the model’s parameters are entirely fixed at inference time (Module 17 covers this in full).

18. What You Should Remember

Forward propagation = repeatedly applying z = W @ x + b then an activation function, layer after layer, from input to final output.
A logit is the raw pre-activation output; a probability is what you get after a final sigmoid/softmax.
Forward propagation happens during both training and inference — it only computes; it never updates weights.

19. How This Helps Me Build AI Systems

You’ve now traced a real prediction, number by number, from raw input to final output. Every LLM response you’ve ever received is this exact mechanism, run at a scale of billions of parameters and dozens of layers — but structurally, nothing new happens; it’s Section 8’s calculation, repeated and scaled.

Next: Module 6 — Loss Functions — how a network measures exactly how wrong its forward-propagated prediction was.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026