The Throughput article covered the general concept of total processing capacity. This article covers the single, specific number most commonly used to actually report it in practice: Tokens Per Second, or TPS.
The simple definition
Tokens Per Second is the specific, standard metric measuring how many tokens a model generates in one second, used to report both an individual user’s experienced generation speed and a system’s overall throughput. Recall from the Token article’s core unit — the discrete pieces of text a model actually generates one at a time. TPS is simply the count of how many of those pieces appear per second, the concrete, headline number behind the more abstract latency and throughput concepts covered in the previous two articles.
Why this became the one number everyone actually quotes
Recall from the Latency and Throughput articles that “speed” genuinely breaks into several distinct measurements — TTFT, inter-token latency, total throughput. TPS became the field’s practical shorthand specifically because it’s easy to measure, easy to compare across different providers and models, and directly translates into something a person can intuitively picture — recall from the Latency article’s real example that 100 tokens per second means generating roughly a 4,000-word essay in just over a minute, a genuinely concrete, human-relatable way to understand an otherwise abstract number.
flowchart LR
A[Model generates tokens] --> B["Count tokens generated per second: TPS"]
B --> C["e.g., 40 TPS: ChatGPT-3.5 typical"]
B --> D["e.g., 500 TPS: Groq's real, published 2024 demo"]The real, viral moment that turned this obscure metric into headline news
This deserves telling with its full, specific numbers, directly continuing the story introduced in the Latency article. When Groq’s public demonstrations went viral in February 2024, the specific number everyone actually repeated across social media was TPS — approximately 500 tokens per second running the open-source Mixtral model, compared to the roughly 40 tokens per second typical of ChatGPT-3.5 at the time, and around 70 for Google’s Bard.
This wasn’t a vague, general claim of being “faster” — it was one precise, directly comparable number, which is exactly why it spread so quickly and became the industry’s default way of talking about inference speed afterward.
Groq’s Head of Brand later noted that at TPS rates around 100, a user could generate an entire 4,000-word essay in just over a minute — a genuinely concrete, easy-to-visualize translation of an abstract benchmark into real, everyday terms.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a car’s top speed in miles per hour — a single, simple, universally understood number that lets anyone instantly compare two cars, even though real driving experience actually depends on acceleration, handling, fuel efficiency, and many other factors a single top-speed number doesn’t capture at all.
Where this breaks down: A car’s top speed is measured under fairly standardized conditions. TPS, in practice, varies significantly based on the specific prompt length, the model being used, current server load, and even which part of the response is being measured — recall from the Latency article’s inter-token-versus-TTFT distinction — meaning a single published TPS figure, unlike a car’s top speed, can be measured and reported in several genuinely different, not always directly comparable ways.
A concrete example, layered
For a simple beginner example: a developer choosing between two API providers for a real-time voice application might directly compare their published TPS numbers — a provider offering 150 TPS will noticeably outpace one offering 50 TPS for the exact same generated response length, a genuinely tangible difference in how natural the resulting conversation feels. For a production example: ArtificialAnalysis.ai, a real, independent benchmarking platform referenced throughout this phase, publishes ongoing, updated TPS comparisons across OpenAI, Anthropic, Google, Groq, and other real providers, and these published numbers directly influence which provider companies choose for genuinely speed-sensitive applications.
Why a high TPS number alone doesn’t tell the whole story
It’s worth being honest about a real, important limitation here, tying together the previous two articles. A system reporting excellent TPS for one single user in isolation might still struggle to maintain that same speed once serving thousands of simultaneous users — recall from the Throughput article’s genuine trade-off between individual speed and total system capacity. A responsibly reported TPS benchmark should specify whether it’s measuring one isolated request or realistic, concurrent production load, since these can produce meaningfully different real numbers for the exact same underlying system.
Common misconception
Tokenizers make comparison less direct
Different model families can divide the same sentence into different numbers of tokens.
Sentence: "Unbelievable!"
Tokenizer A: ["Un", "believable", "!"] -> 3 tokens
Tokenizer B: ["Unbeliev", "able", "!"] -> 3 tokens
Tokenizer C: ["Un", "bel", "iev", "able", "!"] -> 5 tokensIf two systems generate the same visible text, the system producing more token pieces may report a larger TPS even though the user did not receive more words. For cross-model comparisons, also examine characters or words per second and the actual user experience.
Keep the famous comparison in its time period
The roughly 500 TPS Groq/Mixtral comparison with ChatGPT-3.5 and Bard above is a February 2024 historical example. It explains how TPS became widely discussed, but it must not be treated as a current GPT-versus-Gemini leaderboard.
Provider-managed GPT, Gemini, and Claude speeds change over time. A fair current test records the date, model version, region, prompt length, output length, number of simultaneous requests, and whether reasoning or tools were enabled.
Calculate TPS carefully
If the system generates 120 output tokens between the first and final token over 3 seconds:
generation TPS = 120 / 3 = 40 output tokens per secondSome reports divide by the complete request time, including the wait for the first token. Others measure only the generation period. Those two methods produce different numbers, so a TPS comparison needs a definition.
Three numbers that belong together
| Metric | Beginner question |
|---|---|
| TTFT | How long until something appears? |
| Per-request output TPS | How quickly does this answer continue? |
| Total token throughput | How much work does the whole server finish? |
A voice assistant usually needs low TTFT and steady per-request TPS. An overnight summarization job may care much more about total throughput and cost.
Why public model numbers change
GPT, Gemini, and Claude are served on infrastructure managed by their providers. Observed TPS can change with model version, service tier, region, system load, prompt length, output length, tool use, and provider-side optimizations.
For an open model, hardware and software are also part of the result. The same Llama model can generate at different speeds on an H100, H200, consumer GPU, or CPU, and vLLM may behave differently from another serving engine.
Never report “Model X runs at 100 TPS” without naming the test conditions.
Verified sources
A frequent beginner assumption: that TPS is a fixed, unchanging property of a given model — that “GPT-4 runs at X tokens per second,” full stop, the way a car has one fixed top speed.
As this article’s variability discussion explained, TPS actually depends heavily on the specific hardware, serving software, and current load a model happens to be running under at that moment, which is exactly why the same underlying model can post dramatically different TPS numbers on different infrastructure — Groq’s custom LPU hardware achieving roughly 10x the TPS of conventional GPU-based serving for comparable models being the clearest, most publicized real example.
Where this fits in what comes next
You now understand the single, concrete metric this phase’s latency and throughput concepts most often get reported through. The next article, Quantization, covers one of the most direct, practical techniques for actually improving TPS and reducing VRAM requirements at the same time — a real, widely adopted technique this entire phase has been building toward.
In one sentence
Tokens Per Second is the concrete, standard metric for reporting generation speed, and Groq’s real, viral February 2024 demonstration of roughly 500 TPS — about ten times faster than the ChatGPT-3.5 speeds users were accustomed to at the time — is the single moment that turned this once-obscure serving metric into a genuine, industry-wide talking point.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed