TechByteByByte

Sequence-to-Sequence NLP

Understand encoder-decoder architectures for translation and summarization, and a direct, numerically verified demonstration of the fixed-representation bottleneck problem that naturally motivates attention.

#NLP#AI#Sequence-to-Sequence#Encoder-Decoder#Machine Translation

Begin with the central question

How can a model read English and produce French?

Essential words

An encoder reads input. A decoder generates output. In basic seq2seq, a context vector is the fixed-size summary passed between them.

What You Will Understand

Encoder-decoder architectures for tasks like translation and summarization, and a direct, numerically verified proof of their core weakness: compressing an entire sentence into one fixed-size vector causes early information to get diluted, worse as sentences get longer. This is the exact problem that motivates attention (Module 12).

input sequence -> encoder summary -> decoder -> output sequence

Why Some Tasks Transform One Sequence into Another

Module 10’s RNN processes one sequence and produces one output per position (or one final hidden state). But tasks like translation need something different: read an entire input sequence, then generate an entire, potentially different-length output sequence. Sequence-to- sequence (seq2seq) architectures exist to handle exactly this input-to- output-sequence structure.

Read, Summarize, Then Generate

imagine reading an entire paragraph, then having to summarize everything you read into a single sentence-length note before you’re allowed to start writing your response — and you can never look back at the original paragraph again. The longer the original paragraph, the more that single note has to compress, and the more early details inevitably get lost or blurred.

Analogy: The Game of Telephone & The Closed-Book Exam Note Imagine taking a closed-book translation exam under extremely strict rules:

  • The Encoder (Reading): You are given a 100-word English paragraph. You read it word-by-word.
  • The Bottleneck (The Index Card): Before you can translate, you must write down your understanding of the entire paragraph on a single, small post-it note (the fixed-size hidden state vector). You are then forced to hand the English paper back to the proctor.
  • The Decoder (Writing): You are handed a blank sheet of paper and must write the French translation using only your tiny post-it note as a reference.
  • The Dilution: If the English text was just three words (“The cat slept”), the post-it note is more than large enough. But if the English text was 50 words containing names, dates, and locations, your tiny post-it note cannot possibly hold all the details. Early details (the first sentences) are inevitably written over and forgotten as you squeeze the later sentences onto the same small card.

📊 Visual Flowchart: The Encoder-Decoder Vector Bottleneck

Here is how the fixed size of the final encoder hidden state compresses input sequences, reducing the resolution of early details:

graph TD
    classDef bottleneck fill:#f39c12,stroke:#333,stroke-width:2px;

subgraph Encoder ["Encoder RNN (Processes Input Step-by-Step)"]
        x1["Word 1: 'the'"] --> h1["Hidden State 1"]
        x2["Word 2: 'cat'"] --> h2["Hidden State 2"]
        x3["Word 3: 'sat'"] --> h3["Hidden State 3"]
        x4["Word 4: 'on'"] --> h4["Hidden State 4"]

h1 --> h2
        h2 --> h3
        h3 --> h4
    end

h4 -->|Compress| ContextVector["Encoder Final State (h4)<br>[Fixed-Size Vector: d-dimensions]"]:::bottleneck

subgraph Decoder ["Decoder RNN (Generates Output Seq)"]
        ContextVector -->|Initialize| dh0["Decoder State 0"]
        dh0 --> y1["Word 1: 'Le'"]
        y1 --> dh1["Decoder State 1"]
        dh1 --> y2["Word 2: 'chat'"]
    end

4. Core Concept

English sentence
      ↓
Encoder                (an RNN, DL Module 14/Module 10 of this course
                        -- reads the ENTIRE input sequence)
      ↓
Fixed-size representation   (the encoder's FINAL hidden state --
                            ONE vector, regardless of input length)
      ↓
Decoder                        (another RNN -- generates the output
                                sequence, using ONLY this one vector
                                as its starting point)
      ↓
French sentence
TermDefinition
EncoderProcesses the entire input sequence, producing a final, fixed-size representation
DecoderGenerates the output sequence, starting from the encoder’s fixed representation
BottleneckThe single fixed-size vector the encoder must compress the ENTIRE input into

5. How It Works — Step by Step

1. The ENCODER (an RNN) processes the source sentence WORD BY
   WORD, exactly like Module 10 -- but here, only its FINAL
   hidden state matters
2. This final hidden state is a FIXED-SIZE vector -- the SAME
   size regardless of whether the source sentence was 3 words
   or 300 words
3. The DECODER (another RNN) is initialized using ONLY this one
   fixed-size vector -- it has NO direct access to the original
   sequence, only this compressed summary
4. The decoder generates the output sequence one word at a time,
   using its own hidden state (updated as it generates) plus
   whatever information survived the initial compression

6. Mathematical Intuition

The bottleneck problem, precisely: however long the input sequence, the encoder’s output is always exactly d-dimensional (the hidden state size) — a fixed capacity that must somehow represent the entire input’s meaning. As sequence length grows, the amount of information being squeezed into that same fixed-size vector grows too — but the vector’s capacity doesn’t. Something has to give.

7. Simple Example

Encoding a 3-word sentence and a 12-word sentence both produce a final hidden state of the exact same size — but intuitively, the 12-word sentence has much more content to compress into that same fixed space. If you specifically test how much a single early word (like the very first word) still influences the final compressed representation, you’d expect that influence to be diluted more in the longer sentence, since it has to share that same fixed “budget” with many more words.

8. Build It in Python

What the code will demonstrate

The encoder below compresses both a 3-word sentence and a 12-word sentence into vectors of exactly the same size. We then remove the first word and measure how much the final vector changes.

This is a small illustration, not a universal proof that every longer sequence forgets more. Random weights can produce irregular results; the fixed-size bottleneck and long dependency path are the architectural ideas to notice.

import numpy as np

def tanh(x): return np.tanh(x)

# Fixed random weights keep the demonstration repeatable; this is not a trained translator.
np.random.seed(4)
d = 4

Wx_enc = np.random.randn(d, 3) * 0.4
Wh_enc = np.random.randn(d, d) * 0.4

source_embeds = {
    "the": np.array([0.1, 0.1, 0.05]), "cat": np.array([0.6, 0.3, 0.1]),
    "sat": np.array([0.2, 0.7, 0.2]), "on": np.array([0.1, 0.1, 0.1]),
    "mat": np.array([0.5, 0.2, 0.6]),
}

short_source = ["the", "cat", "sat"]
long_source = ["the", "cat", "sat", "on", "the", "mat", "the", "cat", "sat", "on", "the", "mat"]

# The encoder overwrites one fixed-size hidden state as it reads each word.
def encode(sentence, word_embeds):
    h = np.zeros(d)
    for word in sentence:
        h = tanh(Wx_enc @ word_embeds[word] + Wh_enc @ h)
    return h   # the FINAL hidden state -- entire sentence compressed into ONE vector

# Both inputs become four-number vectors even though their lengths differ.
short_encoding = encode(short_source, source_embeds)
long_encoding = encode(long_source, source_embeds)

print("Short sentence (3 words) final encoding:", np.round(short_encoding, 4))
print("Long sentence (12 words) final encoding:", np.round(long_encoding, 4))
print("Both have the SAME fixed size:", short_encoding.shape, "vs", long_encoding.shape)

# Remove the first word and measure its remaining influence on the final summary.
def encode_dropping_first(sentence, word_embeds):
    return encode(sentence[1:], word_embeds)

influence_short = np.linalg.norm(encode(short_source, source_embeds) - encode_dropping_first(short_source, source_embeds))
influence_long = np.linalg.norm(encode(long_source, source_embeds) - encode_dropping_first(long_source, source_embeds))

print(f"\nHow much does DROPPING the first word change the final encoding?")
print(f"  Short sentence (3 words):  change = {influence_short:.4f}")
print(f"  Long sentence (12 words):  change = {influence_long:.4f}")

Expected Output:

Short sentence (3 words) final encoding: [ 0.0749 -0.188   0.2182  0.1455]
Long sentence (12 words) final encoding: [-0.2267 -0.2802  0.1007 -0.1014]
Both have the SAME fixed size: (4,) vs (4,)

How much does DROPPING the first word change the final encoding?
  Short sentence (3 words):  change = 0.0098
  Long sentence (12 words):  change = 0.0003

9. How It Works

  • Both encodings are exactly (4,) — the same fixed size, regardless of whether the source had 3 words or 12 words — this is the bottleneck made concrete: the encoder’s output capacity never grows, no matter how much input it has to compress.
  • Dropping the very first word changes the short sentence’s final encoding by 0.0098, but changes the long sentence’s final encoding by only 0.0003 — roughly 32 times less influence. This is a direct, numerical demonstration of the bottleneck problem: as sequence length grows, any single early word’s contribution to the final, fixed-size compressed representation gets progressively diluted.

10. The Bottleneck Question

“How can one fixed representation capture a long sentence?”

This module’s verified result answers this directly: it increasingly can’t, at least not without losing early information as sequences grow longer. This is precisely the problem attention (Module 12) exists to solve — instead of forcing the decoder to work from one compressed summary, attention lets the decoder look back at every individual position in the source sequence directly, at every decoding step, completely sidestepping the bottleneck.

11. How Is This Used in Modern AI?

🤖 How Is This Used in Modern AI?

Sequence-to-sequence with attention (not the bottleneck-limited version shown here) remains a genuine architectural pattern for tasks with a clear source and target — covered fully in the Transformers course’s encoder-decoder module. The bottleneck problem demonstrated here is specifically what motivated adding attention to seq2seq models, historically, and directly foreshadows why Transformers’ attention mechanism became so significant.

Real systems you can recognize

Translation and summarization are real sequence-to-sequence tasks. Hugging Face’s Transformers project lists T5 for translation and BART for summarization among its supported examples; see the official Transformers repository.

Modern GPT- or Gemini-style generation is also sequence-to-sequence at the application level—prompt tokens enter and response tokens leave—but decoder-only LLMs do not use the original single-vector RNN encoder-decoder design taught in this module.

12. How Is This Used in Agentic AI?

Direct relevance to Agentic AI: Low, directly — modern agents don’t use bottleneck-limited seq2seq RNNs. The conceptual value is direct and important: this exact “compress everything into one fixed representation” limitation is precisely why RAG systems retrieve specific, relevant chunks rather than trying to compress an entire knowledge base into one summary vector — the same underlying problem, recognized and avoided architecturally.

13. Common Mistakes / Misunderstandings

⚠️ Mistake: assuming the encoder’s final hidden state contains “everything” from the input, just compressed. As demonstrated directly, information genuinely gets diluted, not just compressed losslessly — early words’ influence measurably shrinks in longer sequences.

⚠️ Mistake: thinking this bottleneck problem is unique to translation. Any task compressing a variable-length input into one fixed-size representation (summarization, some forms of retrieval) faces the same fundamental issue.

⚠️ Mistake: believing bigger hidden state dimensions fully solve this. A larger d helps somewhat, but doesn’t eliminate the structural problem — there’s always SOME sequence length long enough to overwhelm any fixed-size representation’s capacity.

14. Important Distinctions

Basic Seq2Seq (this module)Seq2Seq + Attention (Module 12)
Decoder only sees ONE fixed final encodingDecoder can look back at EVERY source position directly
Bottleneck — verified: early info dilutes with lengthNo single fixed-size bottleneck
EncoderDecoder
Reads the ENTIRE input sequenceGenerates the ENTIRE output sequence
Produces a fixed-size final representation (in basic seq2seq)Starts from that representation, generates step by step

15. When to Use

Basic (attention-free) seq2seq architectures are mostly of historical/ educational interest today — genuinely useful for understanding why attention was added, less so as a production choice given the proven bottleneck limitation.

16. When Not to Use

Don’t use bottleneck-only seq2seq (without attention) for tasks involving longer sequences — as demonstrated directly, information loss becomes increasingly severe as sequence length grows, exactly where this architecture is weakest.

17. Production Considerations

  • Any system compressing variable-length input into a fixed-size representation faces some version of this bottleneck — worth recognizing this pattern (and its risk) in system designs beyond just translation, including certain memory/summarization components in agent systems.
  • This is precisely why RAG retrieves specific chunks rather than summarizing an entire knowledge base into one vector — a direct, practical design choice motivated by exactly this bottleneck problem.

18. Interview Questions

Beginner

Q: What is the “bottleneck problem” in basic sequence-to-sequence models?

Ans: The encoder compresses the entire input sequence into one fixed-size vector, which the decoder then uses as its only source of information about the input. As input sequences get longer, this same fixed-size vector has to represent more and more content, causing information — especially from earlier in the sequence — to get diluted or lost.

Intermediate

Q: Why does the bottleneck problem get worse as input sequences get longer, rather than staying constant?

Ans: The encoder’s final hidden state has a FIXED size, regardless of input length — it doesn’t grow to accommodate longer inputs. As more words are processed, they all have to be compressed into this same fixed capacity, meaning each individual word’s (especially each early word’s) influence on the final representation gets proportionally diluted. This was demonstrated directly: dropping the first word changed a short sentence’s final encoding far more than it changed a much longer sentence’s final encoding — roughly 32 times more influence in the shorter case.

Advanced

Q: Why is a larger hidden state dimension not a complete solution to the bottleneck problem?

Ans: A larger hidden state dimension increases the encoder’s representational capacity, which can help to a degree — but it doesn’t eliminate the structural issue: for ANY fixed dimension, there exists some sequence length long enough that the amount of information being compressed exceeds what that fixed-size vector can meaningfully preserve.

The problem is architectural, not just a matter of insufficient capacity — which is why the actual solution (attention, Module 12) doesn’t try to make the bottleneck bigger, but eliminates the bottleneck’s forced single-vector compression entirely, letting the decoder access every source position directly instead.

Scenario

Q: A team’s basic (attention-free) sequence-to-sequence translation system performs well on short sentences but noticeably worse on longer paragraphs. Explain why, connecting directly to this module.

Ans: This is a direct, expected consequence of the bottleneck problem demonstrated in this module — the encoder must compress the entire input into one fixed-size vector, and as demonstrated, information (especially from earlier in a long input) gets progressively diluted as sequence length grows. For short sentences, the fixed-size representation has enough relative capacity to preserve most of the content; for longer paragraphs, meaningful information loss becomes increasingly likely.

Adding attention (Module 12) would directly address this by letting the decoder access every source position individually, rather than relying solely on one compressed summary.

AI Engineering

Q: How does the bottleneck problem demonstrated in this module relate to why RAG systems retrieve specific document chunks rather than summarizing an entire knowledge base into a single vector?

Ans: The same fundamental problem applies: compressing an entire, potentially large knowledge base into one fixed-size representation would create a severe information bottleneck.

RAG instead divides the knowledge base into smaller chunks, computes an embedding for each chunk, and retrieves only the chunks relevant to a query. Each embedding is still a compressed representation, so it can lose detail; the advantage is that it summarizes a much smaller piece of text. The system then gives the retrieved source text—not merely its vector—to the language model. This reduces the one-vector bottleneck without eliminating every retrieval or compression error.

19. Next Step

Next: Module 12 — Attention for NLP — the direct solution to this module’s proven bottleneck, letting a decoder access every source position individually rather than one compressed summary.

Author
TechByteByByte Editorial Team
Reviewed by
TechByteByByte Admin
Published
Last reviewed