TechByteByByte

Activation Functions

Understand why activation functions are structurally necessary, with sigmoid, tanh, ReLU, Leaky ReLU, and softmax explained through intuition, formula, and numeric example — plus a direct proof of why linear layers alone can't work.

#Deep Learning#Neural Networks#AI#Activation Functions#ReLU#Softmax

Begin with the central question

Why would stacking layers be useless without a tiny function after each calculation?

That question is the reason this topic exists. Keep it in mind as each new term appears: every equation, diagram, and code example below is one part of the answer.

weighted sum → activation function → useful nonlinear signal

Before you continue: three tools for this module

  • Nonlinear: able to bend or change a pattern instead of producing only a straight-line relationship.
  • Logit: a raw score produced before conversion into a probability.
  • Derivative: a number describing how quickly an output changes when an input changes slightly.

You do not need to memorize these definitions yet. Use them as a small map whenever the terms appear below.

What You Will Understand

Why activation functions are not an optional add-on but a structural necessity, and the specific functions you’ll see everywhere: sigmoid, tanh, ReLU, Leaky ReLU, and softmax — plus a direct, run-in-code proof that stacking layers without them buys you nothing.

Activation functions change what stacked layers can represent:

linear → linear → linear       = one combined linear transformation
linear → ReLU → linear → ReLU  = piecewise nonlinear transformation

Different activations serve different roles. ReLU and its variants are common inside networks; sigmoid is useful for a binary output; softmax converts a vector of logits into a distribution across mutually exclusive classes.

Why Depth Needs Nonlinearity

Module 3 showed a single linear neuron (a perceptron) can’t solve XOR. The natural next idea is “stack more linear layers.” This module proves that idea, by itself, doesn’t work either — and shows exactly what ingredient actually fixes it.

Hinges That Let Layers Bend a Boundary

Without nonlinearity, no matter how many layers you stack, the entire network still only computes one giant weighted sum — mathematically indistinguishable from a single layer. Activation functions introduce a “bend” after each layer, and it’s specifically those bends, accumulated across layers, that let a network represent curved, complex decision boundaries.

Analogy: The Hinged Origami Paper or The Flexing Joint Imagine you have multiple straight segments of stiff metal wire:

  • Stacking Linear Layers (Welding straight wires): If you weld three straight wires end-to-end in a straight line, the resulting single wire is still completely straight. No matter how many segments you weld, you can only lay it flat on a table (a flat linear hyperplane). Stacking linear layers without activation functions collapses mathematically into one single flat linear layer.
  • Activation Functions (Flexing joints/hinges): An activation function behaves like a hinge or joint inserted between each segment. Now, when you connect the wires, you can bend, twist, and fold them into intricate 3D origami structures. These “bends” (nonlinearities) allow the network to wrap around complex, curved boundaries in the data space.

📊 Visual Flowchart: The Linear Collapse vs. Non-linear Activation Protection

Here is how inserting activation functions blocks the algebraic simplification that collapses multi-layer depth:

graph TD
    subgraph PathA ["Path A: Purely Linear Stacking (Collapses)"]
        InA["Input Vector (x)"] --> L1A["Layer 1: z1 = W1*x + b1"]
        L1A --> L2A["Layer 2: z2 = W2*z1 + b2"]
        L2A --> ResultA["Output: W2*(W1*x + b1) + b2<br>Simplifies to: W_comb*x + b_comb"]
        ResultA --> CollapseA["Result: Collapses into 1 single linear equation"]
    end

subgraph PathB ["Path B: Non-linear Activation Stacking (Preserves Depth)"]
        InB["Input Vector (x)"] --> L1B["Layer 1: z1 = W1*x + b1"]
        L1B --> Act1["Activation: a1 = f(z1)"]
        Act1 --> L2B["Layer 2: z2 = W2*a1 + b2"]
        L2B --> ResultB["Output: W2*f(W1*x + b1) + b2"]
        ResultB --> PreserveB["Result: CANNOT be simplified; retains full representation power"]
    end

4. Core Concept

The linearity problem — proven, not asserted

🧠 Without non-linear activation functions, stacking many linear layers still produces one overall linear transformation.

# Build a tiny, inspectable example of Activation Functions.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

np.random.seed(0)
x = np.array([1.0, 2.0, 3.0])

W1 = np.random.randn(4, 3)
b1 = np.random.randn(4)
W2 = np.random.randn(2, 4)
b2 = np.random.randn(2)

# Two-layer, NO activation function between them
layer1_out = W1 @ x + b1
layer2_out = W2 @ layer1_out + b2
print("Two-layer (no activation) output:", layer2_out)

# Equivalent single combined linear layer
W_combined = W2 @ W1
b_combined = W2 @ b1 + b2
combined_out = W_combined @ x + b_combined
print("Single combined layer output:    ", combined_out)
print("Are they equal?", np.allclose(layer2_out, combined_out))

Expected Output:

Two-layer (no activation) output: [  6.61229429 -18.6994426 ]
Single combined layer output:     [  6.61229429 -18.6994426 ]
Are they equal? True

This is the concrete proof: a genuinely two-layer network with no activation functions produces exactly the same output as a single combined layer (W_combined = W2 @ W1). Depth alone, without nonlinearity, adds zero representational power — it’s still just one linear transformation wearing a two-layer costume.

The functions

FunctionFormulaOutput rangeWhere used
Sigmoid1 / (1 + e^(-z))(0, 1)Binary classification output, gates in LSTMs (Module 14)
Tanhtanh(z)(−1, 1)Older hidden-layer default, still used in RNNs
ReLUmax(0, z)[0, ∞)The default for most modern hidden layers
Leaky ReLUz if z>0 else α×z(−∞, ∞)Fixes ReLU’s “dead neuron” problem
Softmaxe^zi / Σ(e^zj)(0, 1), sums to 1Multi-class classification output layer
# Build a tiny, inspectable example of Activation Functions.
# Follow the intermediate values; they reveal what the model is doing.
import numpy as np

def sigmoid(z): return 1 / (1 + np.exp(-z))
def tanh(z): return np.tanh(z)
def relu(z): return np.maximum(0, z)
def leaky_relu(z, alpha=0.01): return np.where(z > 0, z, alpha * z)
def softmax(z):
    exp_z = np.exp(z - np.max(z))   # subtracting max for numerical stability
    return exp_z / np.sum(exp_z)

z_values = np.array([-3, -1, 0, 1, 3])
print("Input z:       ", z_values)
print("Sigmoid:       ", np.round(sigmoid(z_values), 4))
print("Tanh:          ", np.round(tanh(z_values), 4))
print("ReLU:          ", relu(z_values))
print("Leaky ReLU:    ", np.round(leaky_relu(z_values), 4))

logits = np.array([2.0, 1.0, 0.1])
probs = softmax(logits)
print("\nLogits:", logits)
print("Softmax probabilities:", np.round(probs, 4))
print("Sum of probabilities:", np.sum(probs))

Expected Output:

Input z:        [-3 -1  0  1  3]
Sigmoid:        [0.0474 0.2689 0.5    0.7311 0.9526]
Tanh:           [-0.9951 -0.7616  0.      0.7616  0.9951]
ReLU:           [0 0 0 1 3]
Leaky ReLU:     [-0.03 -0.01  0.    1.    3.  ]

Logits: [2.  1.  0.1]
Softmax probabilities: [0.659  0.2424 0.0986]
Sum of probabilities: 1.0

5. How It Works — Step by Step

Sigmoid:      squashes ANY real number into (0, 1) — a smooth
              "how confident, 0 to 100%" curve, symmetric around
              z=0 giving 0.5

Tanh:         same shape as sigmoid, but centered on 0 instead of
              0.5, ranging (-1, 1) -- often trains better than
              sigmoid in hidden layers because its average output
              is closer to zero

ReLU:         literally just "if negative, output 0; if positive,
              output unchanged" -- extremely cheap to compute,
              and empirically trains deep networks very well

Leaky ReLU:   like ReLU, but negative inputs get a small non-zero
              slope instead of being fully zeroed out -- keeps a
              neuron from getting permanently "stuck" outputting
              zero forever (the "dying ReLU" problem)

Softmax:      converts a whole VECTOR of raw scores (logits) into
              a probability DISTRIBUTION -- every value between
              0 and 1, and the whole vector sums to exactly 1

Strengths and weaknesses, briefly

  • Sigmoid — strength: clean probability interpretation. Weakness: gradients become extremely small for very positive/negative inputs (“saturates”), slowing learning in deep networks (Module 10).
  • Tanh — strength: zero-centered, generally trains better than sigmoid. Weakness: still saturates at the extremes.
  • ReLU — strength: cheap, doesn’t saturate for positive inputs, the practical default for hidden layers. Weakness: a neuron whose weighted sum is always negative outputs zero forever and stops learning (“dying ReLU”).
  • Leaky ReLU — strength: fixes dying ReLU. Weakness: the extra hyperparameter (α) and marginal, not-always-necessary benefit over plain ReLU.
  • Softmax — strength: turns raw scores into a genuine probability distribution over multiple classes. Weakness: only meaningful as an output-layer choice for classification, not a general hidden-layer activation.

6. Mathematical Intuition

Read the mathematics as a story

An activation function decides how much of a node’s signal continues. Its nonlinearity lets many layers bend and combine decision boundaries instead of collapsing into one linear calculation.

weighted sum → activation function → useful nonlinear signal

Do not begin by memorizing the symbols. First identify what enters, what operation changes it, and what comes out. The symbols are a compact description of that journey. Sigmoid, worked by hand for z = 1:

sigmoid(1) = 1 / (1 + e^(-1))
           = 1 / (1 + 0.3679)
           = 1 / 1.3679
           ≈ 0.7311

This matches the code output above exactly (0.7311). Every variable: z is the neuron’s raw weighted sum (Module 2); e is Euler’s number (~2.71828); the whole expression squashes z into (0, 1), interpretable as “how strongly does this neuron favor the positive class.”

Softmax, worked by hand for logits = [2.0, 1.0, 0.1]:

e^2.0 ≈ 7.389,  e^1.0 ≈ 2.718,  e^0.1 ≈ 1.105
sum ≈ 11.212

softmax = [7.389/11.212, 2.718/11.212, 1.105/11.212]
        ≈ [0.659, 0.242, 0.099]

Matches the code output. Notice the highest logit (2.0) gets the largest probability (0.659), but the others aren’t zeroed out — softmax produces a genuine distribution, not a hard winner-takes-all decision.

7. Simple Example

Walk through the example

Read the example in three passes:

  1. Identify the input numbers and what each number represents.
  2. Follow one operation at a time instead of jumping directly to the answer.
  3. Interpret the final number in ordinary language and connect it back to the problem.

The purpose is not merely to calculate the result. It is to make the internal mechanism visible. A binary spam classifier’s output neuron uses sigmoid: its final weighted sum might be 2.5, and sigmoid(2.5) ≈ 0.924 — a 92.4% predicted probability of spam. A 5-way intent classifier’s output layer uses softmax across 5 logits, producing 5 probabilities that sum to 1 — e.g., [0.05, 0.72, 0.10, 0.08, 0.05], clearly favoring intent #2.

8. Python Example

Three Python symbols used below

  • NumPy (np) is a Python library for working efficiently with lists and grids of numbers.
  • np.array(...) creates a numeric vector or matrix.
  • @ performs matrix multiplication: many connected weighted sums calculated together.

You can understand the concept without memorizing the syntax. First follow what the numbers represent, and then connect each code operation to the worked example.

What the code will demonstrate

Before running the code, predict the flow: create a small input, apply the topic’s calculation, and inspect the intermediate or final values. The example uses small numbers so you can connect each printed result to the explanation above; a real model performs the same kind of operation with much larger tensors and learned parameters. Both verified snippets are shown above in Section 4, which is where they belong conceptually — the linear-collapse proof and the activation function comparison are this module’s two central pieces of code.

9. How It Works

Already explained inline above (Section 4/6) — restated briefly: the linear-collapse proof shows np.allclose(...) returns True, meaning two “layers” with no activation are mathematically indistinguishable from one layer. The activation comparison shows each function’s distinct shape on the same five input values, and confirms softmax’s outputs genuinely sum to 1.0.

10. Real-World Example

A modern Transformer’s feed-forward sub-layer (Module 16) doesn’t typically use plain ReLU — most current LLMs use GELU or SwiGLU, smoother variants that tend to train slightly better at scale.

The core principle is identical to this module’s: some nonlinearity is structurally required between linear transformations; GELU/SwiGLU are just more refined choices than the classic sigmoid/tanh/ReLU family, found empirically to work better for large Transformer-based models specifically.

11. How Is This Used in Modern AI?

Follow it from mechanism to product

Transformer-based LLMs commonly use nonlinear functions such as GELU or gated variants inside feed-forward sublayers. The activation helps each layer reshape information; it does not independently create reasoning or meaning.

How this connects to LLMs

prompt → tokens → deep-learning computations → next-token probabilities → generated response

The model computation is only the middle of the journey. Tokenization happens before it, while decoding and application controls happen afterward; the following example identifies this topic’s exact role.

🤖 Real-world connection

Every hidden layer in every modern neural network — including every layer of every LLM — has a nonlinear activation function between it and the next layer. Without this, Section 4’s proof applies at any scale: a 100-layer network with no activations would be mathematically equivalent to one layer, regardless of how many billions of parameters it has.

FunctionWhere in modern AI
SoftmaxEvery LLM’s final output layer — converting raw scores over the vocabulary into next-token probabilities
Softmax (again)Inside attention (Module 15) — converting attention scores into weights
SigmoidBinary classifiers, gates inside LSTMs (Module 14)
GELU / SwiGLUThe feed-forward sub-layers inside modern Transformer blocks (Module 16)

12. How Is This Used in Agentic AI?

Trace one agent step

goal + history + tool results → LLM proposal → runtime validation → tool or response

The deep-learning model produces a prediction or structured proposal. The agent runtime—ordinary software around the model—controls permissions, executes tools, stores state, handles retries, and decides whether another model call is needed.

Direct relevance to Agentic AI: Moderate. Softmax specifically shows up directly in classification-based routing components (Module 9 of the ML course’s logistic regression, generalized) that decide which tool or intent a request maps to — the same softmax mechanism computed by hand above is what produces the probability distribution a routing classifier uses to make that decision.

13. Common Beginner Mistakes / Misconceptions Corrected

⚠️ Mistake

Incorrect idea: any nonlinear function works equally well as an activation.

Why it is incorrect: Choice matters — ReLU’s cheap computation and non- saturating behavior for positive inputs is why it became the default; sigmoid/tanh’s saturation genuinely slows down training in deep networks (Module 10 covers exactly why).

⚠️ Mistake

Incorrect idea: softmax is just “normalize the numbers to sum to 1.”

Why it is incorrect: Simple normalization (z / sum(z)) doesn’t handle negative numbers sensibly and doesn’t have softmax’s “emphasize the largest value” behavior — the exponential is doing real, deliberate work.

⚠️ Mistake

Incorrect idea: adding more layers always helps, activation or not.

Why it is incorrect: Section 4 proves this is false without nonlinearity — and even with it, more layers introduce their own training challenges (Module 10).

14. Important Distinctions

SigmoidSoftmax
Takes ONE number, outputs ONE probability (0 to 1)Takes a VECTOR of numbers, outputs a full probability distribution
Used for binary classificationUsed for multi-class classification
Outputs don’t need to relate to each otherOutputs are constrained to sum to exactly 1
ReLULeaky ReLU
Negative inputs → exactly 0Negative inputs → small non-zero value
Can “die” (permanently output 0)Designed specifically to avoid dying

15. When to Use

Use ReLU as the default for hidden layers in most feedforward networks. Use sigmoid for a binary classification output. Use softmax for a multi-class classification output. Use tanh where zero-centered outputs specifically help (still common in RNN gates, Module 14). Use Leaky ReLU if you observe dead ReLU neurons in practice.

16. When Not to Use

Don’t use sigmoid or tanh as a default choice for deep hidden layers — their saturation tends to slow training noticeably compared to ReLU (Module 10 explains the vanishing-gradient mechanism precisely). Don’t use softmax anywhere except a genuine multi-class output layer — it’s not a general-purpose hidden-layer activation.

17. Interview Questions

Beginner

Q: Why are activation functions necessary in a neural network?

Ans: Without a nonlinear activation function between layers, stacking any number of linear layers is mathematically equivalent to a single linear layer — no additional representational power is gained. Activation functions introduce the non-linearity that lets a network represent complex, curved decision boundaries, which is precisely what’s needed to solve problems like XOR (Module 3).

Intermediate

Q: What’s the difference between sigmoid and softmax, and when would you use each?

Ans: Sigmoid takes a single number and squashes it into (0, 1), used for binary classification. Softmax takes a vector of numbers (logits) and converts them into a full probability distribution that sums to 1, used for multi-class classification where exactly one class should be selected (or its probability estimated) among several options.

Advanced

Q: Prove that a two-layer network without any activation function is equivalent to a single-layer network.

Ans: If layer 1 computes h = W1 @ x + b1 and layer 2 computes y = W2 @ h + b2, substituting gives y = W2 @ (W1 @ x + b1) + b2 = (W2 @ W1) @ x + (W2 @ b1 + b2). This has the exact same form as a single linear layer, y = W_combined @ x + b_combined, where W_combined = W2 @ W1 and b_combined = W2 @ b1 + b2 — confirmed numerically in Section 4, where both computations produce identical output.

Scenario

Q: You notice a large fraction of ReLU neurons in a deep network always output zero, regardless of input, and the network’s performance has plateaued. What’s happening, and what would you try?

Ans: This is the “dying ReLU” problem — these neurons’ weighted sums have ended up permanently negative, so ReLU always outputs zero for them, and since ReLU’s gradient is also zero for negative inputs, these neurons stop receiving any learning signal and never recover.

I’d try switching those layers to Leaky ReLU (which gives negative inputs a small non-zero gradient, allowing recovery), or investigate whether the learning rate was too high early in training, pushing many neurons into this dead state.

AI Engineering

Q: Where does softmax appear inside an LLM, beyond just the final output layer?

Ans: Softmax appears twice in a typical LLM’s forward pass: at the very end (Module 17), converting the final layer’s raw scores over the entire vocabulary into next-token probabilities, and inside every attention computation (Module 15), converting raw attention scores into weights that sum to 1, determining how much each token “attends to” every other token.

18. What You Should Remember

  • Without nonlinear activation functions, depth adds zero representational power — proven directly, not just claimed.
  • ReLU is the modern default for hidden layers; sigmoid for binary output; softmax for multi-class output.
  • Modern Transformers typically use GELU/SwiGLU, refinements on the same core nonlinearity principle, not a departure from it.

19. How This Helps Me Build AI Systems

Softmax specifically will reappear twice more in this course — inside attention (Module 15) and at an LLM’s final output layer (Module 17) — and you now understand exactly what it computes and why, from first principles, with real numbers you calculated by hand.

Next: Module 5 — Forward Propagation — tracing one input all the way through a tiny network, hand-calculated and then in code.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed