NLP with RNNs | TechByteByByte

Begin with the central question

What memory lets a model carry information from earlier words?

Essential words

An RNN processes a sequence step by step. Its hidden state is changing numerical memory. A time step is one processed sequence position.

What You Will Understand

Why RNNs were specifically introduced for language modeling — not RNN architecture mechanics (already covered in DL Module 14) but the motivation: a direct, verified demonstration that a sequential hidden state genuinely solves Module 9’s proven “one vector per word” problem.

token 1 -> hidden memory -> token 2 -> updated memory -> output

Why Language Models Need Memory

Module 9 proved static embeddings assign “bank” the exact same vector regardless of context. RNNs exist to fix exactly this: instead of a fixed lookup, a word’s representation at a given point in a sequence can now depend on everything that came before it, via the hidden state you already covered in your Deep Learning course (DL Module 14).

Reading a Sentence One Step at a Time

you already learned RNN mechanics — hidden state, sequential processing, DL Module 14. Here, the specific payoff for NLP: as an RNN reads a sentence word by word, its hidden state accumulates everything read so far. By the time it reaches “bank,” the hidden state already “knows” whether the sentence has been about “deposited money” or “sat by the river” — producing a genuinely different representation for the same final word, depending on what came before.

Analogy: The Cumulative Reading Journal / Memory Pager Imagine reading a book page-by-page and carrying a small pager (the hidden state) that tracks context:

Input Token Sequence ( $x_t$ ): The words arrive one at a time: “I” $\rightarrow$ “deposited” $\rightarrow$ “money” $\rightarrow$ “at” $\rightarrow$ “the” $\rightarrow$ “bank”.

Accumulating Context:

Word 1 (“I”): Pager writes down: “Subject: 1st person” ( $h_1$ ).

Word 2 (“deposited”): Pager combines Word 2 with old context, writing: “Action: Financial deposit” ( $h_2$ ).

Word 3 (“money”): Pager updates folder: “Topic: Cash transaction” ( $h_3$ ).

Resolving Ambiguity: By the time you reach the final word “bank” at Step 6, the pager’s current hidden state ( $h_5$ ) is already saturated with financial transaction data. When the network combines this memory folder with the incoming raw word “bank”, the resulting hidden state ( $h_6$ ) contains the vector for a financial vault, resolving the ambiguity of the surface word.

If the preceding sequence had instead been “sat by the muddy bank”, the pager memory would contain nature/outdoor details, mapping the final “bank” to a soil riverbank.

📊 Visual Flowchart: Sequential Hidden State Context Updates

Here is how sequence inputs modify the hidden state vector step-by-step to compute context-aware token representations:

graph LR
    h0["Initial State (h0)<br>[0, 0, 0]"] --> Update1["RNN Step 1"]
    x1["Word 1: 'deposited'"] --> Update1

Update1 --> h1["Hidden State (h1)<br>Financial Context"]

h1 --> Update2["RNN Step 2"]
    x2["Word 2: 'money'"] --> Update2

Update2 --> h2["Hidden State (h2)<br>Cash Deposit Context"]

h2 --> Update3["RNN Step 3"]
    x3["Word 3: 'bank'"] --> Update3

Update3 --> h3["Final Hidden State (h3)<br>Financial Vault Vector"]

4. Core Concept

I → love → machine → learning

hidden_state[0]  (after "I")
hidden_state[1]  (after "I love" -- carries forward info from "I")
hidden_state[2]  (after "I love machine" -- carries forward MORE)
hidden_state[3]  (after "I love machine learning" -- the FULL sentence)

Each hidden state isn’t just a function of the current word — it’s a function of the current word plus everything the hidden state has accumulated from every earlier word (DL Module 14’s recurrent update: h[t] = f(x[t], h[t-1])).

5. How It Works — Step by Step

1. Initialize the hidden state (typically zeros)
2. Process the sentence WORD BY WORD, in order
3. At each step, the hidden state is updated using BOTH the
   current word's embedding AND the PREVIOUS hidden state
   (DL Module 14 -- not re-derived here)
4. By the time the model reaches an AMBIGUOUS word (like "bank"),
   the hidden state already carries information from every
   PRECEDING word in the sentence
5. This means the model's representation AT that point in the
   sequence can differ depending on what preceded it -- even for
   the exact same final word

Language modeling and sequence classification, specifically:

Next-token prediction:   given the hidden state after processing
                         "I love machine", predict the next word
                         ("learning") -- the historical foundation
                         of what modern LLMs do at massive scale
                         (covered fully in the Transformers course)

Sequence classification:  use the FINAL hidden state (which has
                          "seen" the whole sequence) to classify
                          the entire sentence (e.g., sentiment)

6. Mathematical Intuition

No new math beyond DL Module 14’s recurrent update — the specific point worth verifying: does this mechanism actually produce different representations for the same word under different preceding context? Checked directly below.

7. Simple Example

Processing “I deposited money at the bank” word by word, the hidden state after “bank” has already incorporated “deposited” and “money” — strongly financial context. Processing “we sat by the river bank,” the hidden state after “bank” has instead incorporated “sat” and “river” — strongly non-financial context. Even though the word “bank” itself has the same static embedding in both cases (Module 9’s proven limitation), the RNN’s hidden state at that point in each sequence should differ.

8. Build It in Python

What the code will demonstrate

Both sentences deliberately use the identical static vector for “bank.” The RNN reads earlier words one at a time and repeatedly combines each new word vector with its previous hidden state.

If the two final hidden states differ, it shows that sequential context changed the sentence representation. Because the weights are random and untrained, it does not prove that the vectors encode the correct meanings.

import numpy as np

def tanh(x): return np.tanh(x)

# Fixed seed makes the teaching output repeatable; these weights are not trained.
np.random.seed(2)
d = 3

Wx = np.random.randn(d, 2) * 0.5
Wh = np.random.randn(d, d) * 0.5
b = np.zeros(d)

word_embeds = {
    "I": np.array([0.2, 0.1]), "deposited": np.array([0.8, 0.1]),
    "money": np.array([0.85, 0.05]), "at": np.array([0.1, 0.1]),
    "the": np.array([0.05, 0.05]), "bank": np.array([0.5, 0.5]),   # SAME embedding, always
}

sentence1 = ["I", "deposited", "money", "at", "the", "bank"]
# Start with empty memory, then update it once for every word.
h = np.zeros(d)
for word in sentence1:
    h = tanh(Wx @ word_embeds[word] + Wh @ h + b)
print("Hidden state after sentence 1 ('...deposited money at the bank'):", np.round(h, 4))

word_embeds2 = {
    "we": np.array([0.2, 0.15]), "sat": np.array([0.15, 0.6]),
    "by": np.array([0.1, 0.1]), "the": np.array([0.05, 0.05]),
    "river": np.array([0.1, 0.9]), "bank": np.array([0.5, 0.5]),    # SAME "bank" embedding
}
sentence2 = ["we", "sat", "by", "the", "river", "bank"]
# Reset memory before reading the second sentence through the same RNN weights.
h2 = np.zeros(d)
for word in sentence2:
    h2 = tanh(Wx @ word_embeds2[word] + Wh @ h2 + b)
print("Hidden state after sentence 2 ('...sat by the river bank'):      ", np.round(h2, 4))

print("\nDifferent hidden states, despite the identical final word 'bank'?", not np.allclose(h, h2))

Expected Output:

Hidden state after sentence 1 ('...deposited money at the bank'): [-0.0718  0.0885 -0.3718]
Hidden state after sentence 2 ('...sat by the river bank'):       [-0.2233 -0.3076 -0.732 ]

Different hidden states, despite the identical final word 'bank'? True

9. How It Works

This is the direct payoff for Module 9’s proven failure: even though the word “bank” uses the exact same static embedding ([0.5, 0.5]) in both sentences, the RNN’s hidden state at that point in the sequence is genuinely different (True) — because the hidden state carries forward everything processed before “bank,” and the two sentences’ preceding words (“deposited money” vs. “sat by the river”) are entirely different. This is precisely the property Module 9 asked for: a word’s representation now depends on the surrounding words, not just the word’s own static, fixed vector.

10. Limitations (Setting Up Modules 11-12)

1. SEQUENTIAL COMPUTATION -- processing word 5 requires having
   already processed words 1-4, in strict order (DL Module 14 --
   this prevents the kind of parallelization that later became
   critical for training at scale, covered in the Transformers
   course)
2. VANISHING GRADIENTS -- context from very early words can get
   diluted by the time the hidden state reaches much later words
   in a long sequence (DL Module 14)
3. LONG-RANGE DEPENDENCIES -- a word's meaning sometimes depends
   on something mentioned MANY words earlier -- RNNs struggle to
   preserve this over long distances

🧠 LSTM/GRU (DL Module 14) meaningfully improved the vanishing gradient issue via gating, extending how far back useful context could be preserved — but did not solve the sequential computation limitation. This remaining gap directly motivates Module 11’s sequence-to-sequence architectures and Module 12’s attention.

11. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

RNNs (and LSTMs/GRUs) were the dominant architecture for sequence modeling in NLP for years, and directly motivated the sequence-to- sequence and attention mechanisms (Modules 11-12) that eventually led to Transformers. Modern LLMs are not RNN-based — but understanding precisely what problem RNNs solved for NLP (context-dependent representations) is essential to understanding why attention, later, was such a significant leap.

Real systems you can recognize

RNNs, LSTMs, and GRUs remain available for sequence problems, especially smaller streaming or time-series systems, but GPT and Gemini belong to the Transformer era. The contrast matters: an RNN must finish the previous time step before computing the next, while Transformer training can process many sequence positions in parallel.

Hugging Face’s Transformers library focuses on Transformer architectures for modern language tasks; its task overview includes generation, translation, summarization, classification, and question answering.

12. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: Low, directly — modern agents run on Transformer-based LLMs, not RNNs. The value here is entirely historical and conceptual: understanding that context-dependent representation (what RNNs first enabled for NLP) is the property every subsequent architecture — sequence-to-sequence, attention, Transformers — continued to build on and improve, all the way to the LLMs powering modern agents.

13. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming this module reintroduces RNN mechanics from scratch. It doesn’t — DL Module 14 already covered hidden states, vanishing gradients, and LSTM/GRU gating in full. This module’s value is specifically the NLP motivation and the verified proof that RNNs solve Module 9’s context problem.

⚠️ Mistake: believing RNNs fully solved the context problem with no remaining limitations. As Section 10 states, sequential computation and long-range dependency issues remain — directly motivating Modules 11-12.

⚠️ Mistake: thinking modern LLMs use RNNs. They don’t — decoder- only Transformers (Transformers course) replaced RNNs specifically because of the limitations in Section 10.

14. Important Distinctions

Static Embeddings (Module 8-9)	RNN Hidden States (this module)
ONE vector per word, always	A DIFFERENT hidden state depending on preceding context — verified directly
No mechanism for context	Context accumulates sequentially, word by word

RNN	LSTM/GRU (DL Module 14)
Simple hidden state, prone to vanishing gradients over long sequences	Gating meaningfully reduces (not eliminates) this issue

15. When to Use

RNN-family models remain a reasonable choice for smaller-scale sequential tasks where context-dependent representation matters but Transformer-scale infrastructure isn’t warranted — genuinely useful for certain streaming or resource-constrained applications.

16. When Not to Use

Don’t use RNNs for large-scale, long-context, or highly parallelizable training needs — their sequential computation requirement (Section 10) is a genuine, significant limitation at scale, which is precisely why the field moved toward attention and Transformers.

17. Production Considerations

Sequential computation limits training throughput — a genuine, practical constraint that became increasingly significant as datasets and models grew, directly motivating the architectural shift covered later in this course and fully in the Transformers course.
Context window length matters differently for RNNs — since context is carried forward through a single hidden state rather than direct attention to every prior position, very long sequences stress RNNs differently than they stress attention-based models.

18. Interview Questions

Beginner

Q: Why were RNNs introduced specifically for language modeling?

Ans: Because static word representations (Module 8-9) assign a fixed vector to each word regardless of context — RNNs solve this by processing a sequence word by word, maintaining a hidden state that accumulates information from every preceding word, so a word’s effective representation at any point in the sequence can depend on everything that came before it.

Intermediate

Q: How does an RNN’s hidden state address the specific limitation proven in Module 9?

Ans: Module 9 proved static embeddings give the same word an identical vector regardless of surrounding context. An RNN’s hidden state, by contrast, is updated using both the current word’s embedding AND the previous hidden state — meaning that by the time an RNN processes an ambiguous word like “bank,” its hidden state already reflects everything read before it. This was verified directly: the same word “bank,” with the same static embedding, produced genuinely different hidden states depending on whether the preceding words were about “money” or “river.”

Advanced

Q: RNNs solved the static-embedding context problem, yet the field still moved toward attention and Transformers. Why weren’t RNNs sufficient?

Ans: While RNNs successfully make representations context-dependent (verified directly in this module), they carry two significant remaining limitations: sequential computation (each step depends on the previous one completing, preventing the kind of parallelization needed for efficient training at scale) and difficulty preserving long-range dependencies (information from much earlier in a sequence can get diluted by the time it reaches distant later positions, even with LSTM/GRU’s gating improvements).

Attention (Module 12) solves both of these simultaneously — connecting any two positions directly, regardless of distance, with no forced sequential dependency — which is why it represented a more complete solution than RNNs alone provided.

Scenario

Q: A team is choosing between an RNN-based and an attention-based approach for a task involving very long documents where information from the beginning of the document is often relevant to understanding the end. What would you recommend, and why?

Ans: I’d lean toward an attention-based approach — RNNs, even with LSTM/GRU gating, can struggle to preserve information over very long distances, since context has to be carried forward sequentially through a single hidden state that can dilute over many steps. Attention (Module 12) allows any position in the document to directly relate to any other position, regardless of distance, without relying on information having survived a long sequential chain — better suited to exactly this kind of long-range dependency requirement.

AI Engineering

Q: Why is understanding RNNs’ historical role in NLP still valuable for an AI engineer who will primarily work with modern Transformer-based LLMs?

Ans: Understanding what specific problem RNNs solved (context-dependent representation, verified directly in this module) and what problems they still had (sequential computation, long-range dependency issues) is what makes the subsequent architectural shift to attention and Transformers feel like a well-motivated engineering response to specific, identifiable limitations, rather than an arbitrary trend.

This deeper understanding directly supports reasoning about why modern LLMs behave the way they do — their strengths in long-range context handling and their parallelizable training are direct, traceable consequences of moving away from RNNs’ sequential architecture.

19. Next Step

Next: Module 11 — Sequence-to-Sequence NLP — encoder-decoder architectures, and the fixed-representation bottleneck problem that directly motivates attention.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026