Token

The journey this phase follows

This phase answers one practical question:

How can a computer find a document with the right meaning, even when it does not contain the exact words in the question?

flowchart LR
    A[Text] --> B[Tokens]
    B --> C[Sequence]
    C --> D[Embedding model]
    D --> E[Vector]
    E --> F[Vector database and index]
    F --> G[Similarity search]
    G --> H[Semantically relevant result]

The early articles explain how language becomes numbers. The middle articles explain how those numbers represent meaning. The final articles explain how systems search millions of those representations quickly.

Every mention of “billions of tokens” throughout this glossary — training data size, GPT-3’s context window, API pricing — has quietly assumed you know what a token actually is. It’s time to make that precise: a token.

The simple definition

A token is a small chunk of text — sometimes a whole word, sometimes just part of one, sometimes a single character or punctuation mark — that a language model treats as one individual unit. Recall from the Input article that models only compute on numbers, and recall from the Prediction article that a language model predicts “one token at a time.” A token is exactly that unit: the smallest piece of text the model actually reads, processes, and generates, one at a time.

Why text gets broken into these particular pieces

You might expect a model to simply process one word at a time, the way you might read. Text doesn’t split into words that cleanly for a computer, though — words vary wildly in length, new words appear constantly, and many languages don’t even separate words with spaces the way English does. Splitting text into individual characters instead would solve that consistency problem, but would make sequences extremely long and would force the model to learn spelling and grammar from scratch, letter by letter, an inefficient use of the network’s limited capacity. Tokens are a practical middle ground: common whole words become a single token (“the,” “cat,” “running”), while rarer or more complex words get broken into smaller, reusable pieces (“unbelievable” might become “un” + “believ” + “able”).

flowchart LR
    A[Raw text: 'unbelievable'] --> B[Tokenizer]
    B --> C[Token 1: 'un']
    B --> D[Token 2: 'believ']
    B --> E[Token 3: 'able']

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of assembling a message using a fixed set of rubber stamps, rather than writing freehand. Common whole words get their own dedicated stamp (“the,” “and,” “is”). Less common words have to be built by combining several smaller stamps together — prefixes, root pieces, suffixes — since it wouldn’t be practical to own a separate stamp for every possible word in the language.

Where this breaks down: A person choosing rubber stamps applies conscious judgment about which pieces to combine. Which specific pieces of text become tokens is decided automatically, ahead of time, by a statistical algorithm run once over a huge amount of text — a fixed, mechanical process covered fully in the next article, Tokenization, not a judgment call made fresh for every new sentence.

What a real token actually looks like

This is worth seeing concretely rather than described abstractly. Using OpenAI’s real, published tokenizer (called tiktoken), the two-word phrase “hello world” becomes exactly two tokens, represented internally as the numbers 15339 and 1917 — each number an index pointing to one specific entry in the model’s fixed vocabulary of possible tokens. A longer or less common word often becomes multiple tokens; a short, extremely common word or even punctuation mark is frequently its own single token. As a rough, commonly cited rule of thumb, one token corresponds to roughly ¾ of an English word on average — meaning 100 words of ordinary English text translates to roughly 130 tokens.

A concrete example, layered

For a simple beginner example: the sentence “Cats are great” might split into tokens roughly matching the words themselves — “Cats,” ” are,” ” great” — since these are all common, everyday words already in the model’s vocabulary. For a production example: OpenAI’s GPT-4 and GPT-3.5-turbo both use a tokenizer encoding called cl100k_base, with a fixed vocabulary of roughly 100,000 possible tokens; the newer o200k_base encoding, used by GPT-4o, expanded that vocabulary to roughly 200,000 tokens, reflecting an intentional design choice to represent more text (including non-English languages and code) more efficiently, using fewer tokens per unit of real content.

Why token count has real, practical consequences

This isn’t a purely academic detail — token count directly affects cost and capability in real AI products. Recall from the Prediction and Inference articles that API pricing is typically charged per token, and a model’s context window (how much text it can consider at once) is measured in tokens, not words or characters. This is why developers building on top of AI APIs actively monitor and manage token counts — a longer prompt costs more and consumes more of a limited context window, regardless of how many “words” it technically contains.

Follow one sentence into tokens

Consider:

How do I reset my password?

A hypothetical tokenizer might produce:

["How", " do", " I", " reset", " my", " password", "?"]

This illustration has seven tokens, but the exact split depends on the tokenizer. Another model may split password into smaller pieces or attach spaces differently.

Each vocabulary token maps to an integer ID:

text → token pieces → token IDs

The ID is an index, not the meaning itself. An embedding lookup later converts that ID into a learned vector.

What is a token ID?

A token ID is an integer that identifies one entry in a tokenizer’s fixed vocabulary.

Imagine this tiny teaching vocabulary:

Token piece	Token ID
`"How"`	41
`" reset"`	208
`" password"`	917
`"?"`	12

["How", " reset", " password", "?"]
                    ↓
[41, 208, 917, 12]

These example IDs do not come from a particular GPT or Gemini tokenizer.

How is a token ID generated?

The ID is normally assigned when the tokenizer’s vocabulary is built—not invented again whenever a user submits text.

flowchart TB
    A[Collect training text] --> B[Start with basic symbols or byte pieces]
    B --> C[Learn useful recurring token pieces]
    C --> D[Create fixed vocabulary]
    D --> E[Assign one integer ID to each entry]
    E --> F[Save tokenizer rules and vocabulary]

At runtime, the tokenizer applies its saved splitting rules and looks up the stored ID for each resulting piece.

The ID value itself has no semantic magnitude. Token ID 917 is not “more password-like” than token ID 208; it is simply a stable lookup address inside that tokenizer.

Does the same word always receive the same token ID?

The same exact token piece receives the same ID under the same tokenizer and version. The same visible word does not always become the same token piece.

The split can change because of:

Leading space: "apple" and " apple" may be separate entries.
Capitalization: "Apple" and "apple" may tokenize differently.
Punctuation: surrounding symbols can affect token boundaries.
Rare words: one word may become several subword or byte tokens.
Different tokenizers: GPT, Gemini, and other families can use different vocabularies and IDs.
Tokenizer versions: a changed vocabulary can produce different pieces or IDs.

Same tokenizer + same exact token piece → same token ID
Same-looking word in a different form   → possibly different tokenization
Different tokenizer                     → IDs cannot be compared directly

A token ID is meaningful only together with the tokenizer that created it.

How GPT and Gemini use tokens

flowchart LR
    A[User prompt] --> B[Model-specific tokenizer]
    B --> C[Token IDs]
    C --> D[Model processes token sequence]
    D --> E[Scores possible next tokens]
    E --> F[Selected token is appended]
    F --> D

GPT and Gemini-style text generation is autoregressive: the model repeatedly predicts the next token from the tokens already in its context. Tokens therefore affect context limits, latency, and usage measurement.

The token pieces and IDs are model-family specific. A GPT token ID cannot be sent to Gemini with the assumption that it refers to the same text piece.

Common misconception

A frequent beginner assumption: that a token is basically the same thing as a word. As the “unbelievable” and tokenizer examples above showed, this is often wrong — many real tokens represent word fragments, punctuation, or even parts of numbers, and the exact split for any given piece of text depends entirely on the specific tokenizer and vocabulary in use, not on any fixed, universal rule about what a “word” is.

Where this fits in what comes next

You now understand what a token actually is. The next article, Tokenization, covers the actual algorithm that decides where to split text into tokens in the first place — the process that produced the “un” + “believ” + “able” split shown above.

In one sentence

A token is the small, fixed unit of text a language model actually reads and generates one piece at a time, and understanding what it is — often a word fragment rather than a whole word — explains everything from a model’s context window to how AI API pricing actually works.

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026