The Automated Evaluation article covered checking output against a fixed, predetermined answer. This article covers a more flexible, sophisticated automated technique — using a model to judge open-ended quality directly: LLM-as-a-judge.
The simple definition
LLM-as-a-judge means using a capable language model to evaluate another model’s output — scoring its quality, comparing it against an alternative, or checking it against specific criteria — rather than a human or a simple, fixed-answer comparison doing the judging. Recall from the RAG Evaluation article’s earlier mention of this exact technique, back in the RAG & Retrieval phase, where frameworks like RAGAS use it to assess faithfulness and relevance. This article gives that technique its full, dedicated treatment.
Why this genuinely works better than simple automated checks for open-ended tasks
Recall from the Automated Evaluation article’s core limitation — a fixed reference answer can’t capture the many different, equally valid ways a genuinely good response could be phrased. A capable language model, by contrast, can read an open-ended response and assess its actual quality — coherence, accuracy, helpfulness — much closer to how a human would, without needing an exact reference string to compare against, and without needing a human to personally review every single case.
flowchart LR
A[Model A generates a response] --> C[Judge LLM compares and scores]
B[Model B generates a response] --> C
C --> D["Verdict: which response is better, and why"]The real, landmark paper that established this technique, and its real, published limits
This deserves to be told precisely, because it’s a genuine, foundational piece of published research this entire technique is built on. In 2023, researchers led by Lianmin Zheng published MT-Bench and the LLM-as-a-Judge paradigm directly, demonstrating that GPT-4’s judgments correlated well with aggregate human preferences — a genuinely encouraging, validating result for the technique’s core premise.
But the same research, and substantial follow-up work, also identified real, specific, measurable biases in how LLM judges actually behave. Position bias means a judge can favor whichever response happens to appear first (or last) in the prompt, regardless of actual quality. Verbosity bias means judges tend to favor longer responses, even when a shorter one is equally or more correct.
Self-preference bias, documented in multiple follow-up studies, means a model used as a judge tends to rate its own outputs more favorably than an equally good response from a different model.
ANALOGY vs. TECHNICAL REALITY
Analogy: Think of a genuinely skilled, experienced editor asked to judge a writing contest anonymously — capable of recognizing real quality far faster and more consistently than an inexperienced reader, but still a human, still subject to real, documented cognitive quirks: preferring the entry they happened to read first, or unconsciously favoring longer, more elaborate submissions over concise, equally excellent ones.
Where this breaks down: A human editor’s biases stem from genuine psychological tendencies, built over a lifetime. An LLM judge’s biases are statistical artifacts of its own training — recall from the Next-Token Prediction article’s core mechanism — patterns learned from training data that happen to correlate position or length with quality in ways that don’t reliably hold for the specific case actually being judged.
How real evaluation frameworks correct for these documented biases
This is worth being concrete about, since real, practical mitigations exist and are widely used. To address position bias, real evaluation pipelines commonly run the same comparison twice, with the two responses swapped in order, and only trust a verdict that holds consistently both ways. To address self-preference bias, serious evaluations often use a different, independent model as the judge rather than the model being tested judging itself. These aren’t complete fixes — the biases remain real, documented tendencies — but they’re genuine, practical steps that meaningfully reduce their impact on a final evaluation.
A concrete example, layered
For a simple beginner example: comparing two draft email replies for helpfulness, a judge LLM prompted with clear criteria — did it answer the question, was the tone appropriate — can reliably pick the better one across thousands of test cases far faster than a human reviewer could, though a careful evaluator would still swap the order of the two drafts and re-check to guard against position bias.
For a production example: RAGAS, the real, published evaluation framework covered in the RAG Evaluation article, uses LLM-as-a-judge directly to calculate metrics like faithfulness and answer relevance, and Chatbot Arena’s own leaderboard, covered in the LLM Evaluation article, has increasingly incorporated LLM-judged comparisons alongside its real human votes to scale evaluation coverage further.
Why this technique remains a genuine trade-off, not a solved replacement for humans
It’s worth being direct about this, tying back to the Human Evaluation article’s core argument. LLM-as-a-judge is faster and cheaper than human evaluation by a wide margin, but its documented biases mean it isn’t a perfect substitute — a judge model’s own weaknesses can become invisible blind spots in evaluations that rely on it too heavily, which is exactly why the most careful, real evaluation setups use LLM-as-a-judge for fast, everyday iteration while still periodically validating results against genuine human evaluation.
Common misconception
Make a judge run reproducible
Record these together with every judge score:
judge model and dated version
judge system prompt
rubric and examples
candidate-answer order
generation settings
dataset version
raw verdict and explanationIf the provider changes the judge model, scores can move even though the candidate answers did not. Re-run a human-rated calibration set before mixing results from the new judge with the old judge.
Check consistency
Run difficult cases more than once or use more than one judge. If verdicts change frequently, send those cases to humans instead of hiding the disagreement inside an average.
Judge explanations help people inspect decisions, but a convincing explanation does not prove the verdict is correct. The most important evidence is agreement with qualified human ratings on the intended task.
Pointwise and pairwise judging
- Pointwise: score one answer against a rubric, such as 1–5 for correctness.
- Pairwise: compare answer A with answer B and choose A, B, or a tie.
Pointwise: question + answer + rubric -> score and reason
Pairwise: question + answer A + answer B + rubric -> A, B, or tiePairwise judgment is often easier because the judge makes a relative choice. Pointwise scores are convenient for tracking one system over time.
Reduce judge bias step by step
- Hide model names from the judge.
- Give a narrow rubric with examples.
- Swap answer A and B, then judge again.
- Allow a tie rather than forcing a winner.
- Compare judge decisions with human ratings.
- Send disagreements and important failures to humans.
A current Gemini example
Google Vertex AI’s current Python documentation shows a pairwise evaluation using gemini-2.5-pro as the baseline and gemini-2.5-flash as the candidate. The service returns candidate wins, baseline wins, ties, and explanations.
Google also documents calibrating the judge against human labels. A high agreement result on one rubric does not prove the judge is reliable for every language, topic, or safety decision.
Verified sources
A frequent beginner assumption: that using GPT-4 or a similarly capable model as a judge produces an objective, bias-free evaluation, simply because the judge itself is a highly capable model. As the MT-Bench paper’s own published findings directly demonstrated, capability and freedom from bias are genuinely separate properties — even a highly capable judge model exhibits real, measurable position, verbosity, and self-preference biases that a careful evaluation setup has to actively correct for.
Where this fits in what comes next
You now understand using a model to judge another model’s output, and the real, documented biases this introduces. The next article, Evaluation Dataset, covers the actual test material — human-evaluated, automated, or judge-based — all of these techniques ultimately need to operate on.
In one sentence
LLM-as-a-judge uses a capable language model to evaluate open-ended output quality, a technique the landmark 2023 MT-Bench paper validated as correlating well with human judgment in aggregate — while also documenting real, specific biases (position, verbosity, self-preference) that any serious evaluation pipeline using this technique has to actively, deliberately correct for.
Related Terms
- Author
- TechByteByByte Editorial Team
- Reviewed by
- TechByteByByte Admin
- Published
- Last reviewed