Interpretability

The Explainability article covered producing a plausible, useful account of a decision after the fact. This article covers the deeper, harder goal that account is often standing in for: interpretability.

The simple definition

Interpretability is the degree to which a human can genuinely understand a model’s internal mechanism — not an approximation of what it did, but an actual, accurate account of how it works. Recall from the Explainability article’s honest limitation: techniques like LIME and SHAP produce useful approximations by observing behavior from the outside. Interpretability asks a harder question entirely — can you actually open up the model and understand, in a real, verified sense, what its internal computations are doing.

Why this distinction is worth being genuinely precise about

This is one of the most frequently confused pairs in the entire field, and it’s worth resolving carefully. Recall from the Weights article’s discussion of distributed representation — a large neural network’s “knowledge” is smeared across millions or billions of individual weight values, with no single weight corresponding cleanly to any one human-understandable concept.

A model can be explainable (you can get a plausible, useful account of a specific decision, as covered in the previous article) while being genuinely uninterpretable (nobody can actually trace through its billions of parameters and verify that account is really, mechanically what happened). Explainability is about the output; interpretability is about the mechanism.

flowchart LR
    A[Interpretability: can you understand the actual internal mechanism?] --> B[Simple linear model: yes, genuinely interpretable]
    A --> C[Deep neural network: no, weights are distributed and opaque]
    D[Explainability: can you get a plausible account of a decision?] --> E[Achievable for both, through different means]

ANALOGY vs. TECHNICAL REALITY

Analogy: Think of the difference between a simple mechanical clock, where you can open the back and directly trace exactly how each gear turns the next, genuinely understanding the entire mechanism — versus a black-box vending machine that reliably gives you a plausible receipt explaining “item selected: B4, price: $1.50” without you ever being able to see or verify the actual internal circuitry that made that specific decision.

Where this breaks down: A mechanical clock’s transparency is a matter of literally being able to look inside. A model’s interpretability is a matter of whether its internal mathematical structure — millions of interacting weights, as covered throughout the Neural Networks phase — can be mapped onto concepts a human mind can actually hold and verify, a fundamentally harder, more research-intensive problem than simply removing a case and looking inside.

Why some models are genuinely interpretable, and most modern ones aren’t

Recall from the Model article’s earliest coverage of simple techniques. A basic linear regression model, or a small decision tree, genuinely is interpretable — you can look directly at its learned coefficients or its exact branching rules and understand precisely why it produced a given output, with no approximation involved at all.

Recall from the Parameters article’s discussion of scale — a model like GPT-3, with 175 billion parameters, is fundamentally different: there’s no realistic way for a human to trace through that many interacting weights and hold a genuine, verified understanding of the whole mechanism, which is exactly why real interpretability research on large models works through indirect, painstaking methods rather than simple direct inspection.

The real, active research field trying to close this gap

This is worth naming directly, since it’s a genuine, serious area of current research, not a hypothetical. Mechanistic interpretability, a real research discipline pursued directly by labs including Anthropic, attempts to reverse-engineer specific, identifiable circuits within a trained model — finding, for instance, a specific, traceable pathway of neurons responsible for a particular capability, like recognizing when a sentence is about to name a particular category of entity.

This work has produced genuine, published successes on smaller models and specific narrow behaviors, but scaling this kind of true, verified mechanistic understanding up to frontier-scale models remains a real, unsolved, actively pursued research problem.

The real, public demonstration that made this concrete

This is worth telling in full, because Anthropic didn’t just publish a paper — it let the public actually talk to the result. On May 23, 2024, alongside a major research paper on interpreting Claude 3 Sonnet, Anthropic identified a specific, isolatable feature inside the model corresponding to the concept of the Golden Gate Bridge — a combination of neurons that activated whenever the model encountered text or images related to the famous landmark.

To prove this wasn’t just a plausible-sounding interpretation but a genuine, causal understanding, Anthropic “clamped” that feature to 10 times its normal maximum activation and released the resulting model, nicknamed Golden Gate Claude, publicly on claude.ai for 24 hours. The model began identifying itself as the Golden Gate Bridge in ordinary conversation — describing itself in the first person as “a famous bridge” and working the landmark into unrelated answers.

Critically, Anthropic was explicit about what this wasn’t: not a system prompt telling the model to role-play, not fine-tuning on new data, but a precise, surgical intervention on the model’s actual internal activations — a real, causal demonstration that the identified feature was genuinely doing what the interpretation claimed, not just a plausible story about it.

A concrete example, layered

For a simple beginner example: a simple credit-scoring model using just three factors (income, debt, payment history) combined through a basic weighted formula is fully interpretable — anyone can look at the formula and know exactly how a given score was calculated. For a production example: Anthropic’s own published interpretability research has documented finding specific, identifiable internal features within Claude models corresponding to concrete concepts — a genuine, real demonstration that at least some meaningful interpretability is achievable even in large, modern language models, though covering only a small fraction of everything such a model actually does internally.

Why this gap matters for real, practical trust

It’s worth connecting this back to the AI Safety article’s core concerns. A model that’s explainable but not interpretable can still surprise you — its plausible, after-the-fact explanations might not actually reflect what it would do in a genuinely novel situation its explanation technique never got to observe. Genuine interpretability, if achieved at scale, would offer something explainability alone can’t: real, verified confidence about how a system will behave, not just a plausible story about how it behaved this one time.

Read a tiny model directly

Imagine a model with this learned rule:

risk score = (2 × missed payments) + (0.5 × debt percentage) - 4

For 3 missed payments and 40% debt, the calculation is:

(2 × 3) + (0.5 × 40) - 4 = 22

Here, a human can inspect every weight, the bias, and the calculation. The model is directly interpretable because its mechanism is small enough to follow. Now compare that with a large language model:

tokens -> embedding vectors -> many Transformer layers -> output probabilities
                         millions or billions of interactions

A concept such as “politeness” is usually not stored in one clearly named node. It can be represented by a pattern spread across many activations and weights. Researchers therefore study selected features and circuits instead of pretending they can read the whole model at once.

Interpretability and explainability side by side

Question	Interpretability	Explainability
Main goal	Understand the actual mechanism	Give a useful account of an output
Easy example	Read a small decision tree	Use SHAP to explain one prediction
Frontier-model reality	Only small pieces are understood	Application traces and approximations are practical
Main danger	Assuming a discovered feature explains everything	Mistaking a plausible explanation for proof

A real model connection

In Anthropic’s 2024 Claude 3 Sonnet research, researchers extracted millions of internal features and demonstrated that changing the activation of a Golden Gate Bridge feature could change the model’s behavior. This is stronger evidence than merely asking Claude to explain itself because the researchers intervened on an internal representation and observed the effect. Even so, understanding a feature is not the same as understanding the full model.

GPT, Gemini, and Claude process each token through many layers, and the meaning of an activation depends on its surrounding context and interactions. Mechanistic interpretability remains a research tool, not a complete instruction manual for a frontier model.

Common misconception

Hold the idea in your head

You can remember interpretability with four steps:

A model receives numbers.
Its learned weights transform those numbers.
Researchers inspect selected transformations and test what they affect.
They claim understanding only when evidence supports the connection.

Interpretability asks: “What mechanism inside the model produced this behavior?”

Finding one meaningful feature does not explain a whole layer. Explaining one layer does not explain a whole model. The safe conclusion should remain as small as the evidence.

A frequent beginner assumption: that “interpretability” and “explainability” are simply two words for the same idea, used interchangeably depending on which researcher you’re reading. As this article has explained, the field increasingly treats them as genuinely distinct — explainability is achievable today, at scale, through approximation techniques; interpretability, in its full, mechanistic sense, remains a real, unsolved research frontier for large models, and conflating the two risks overstating how well anyone actually understands what a large language model is doing internally.

Where this fits in what comes next

You now understand both the practical, achievable goal (explainability) and the harder, still largely unsolved one (interpretability). The next article, AI Bias, covers a real, well-documented problem that both of these concepts exist partly to help catch — patterns a model learns that produce unfair, discriminatory outcomes, sometimes only discoverable specifically because someone applied explainability or interpretability techniques to find them.

In one sentence

Interpretability is the genuine, verified understanding of a model’s actual internal mechanism, distinct from explainability’s plausible after-the-fact account — a much harder goal that remains largely unsolved for large, modern neural networks, even as real, active research fields like mechanistic interpretability make genuine, if still narrow, progress toward it.

Verified sources

Author: TechByteByByte Editorial Team
Reviewed by: TechByteByByte Admin
Published: August 31, 2026
Last reviewed: August 31, 2026