When LLM-as-Judge Fits (and When It Doesn't)

LLM-as-judge is the technique of using a model to grade the output of another model (or the same model). It has become the default way to evaluate open-ended generation in 2026 because nothing else scales. This lesson covers when to reach for it and when to reach for something else.

Where LLM-as-Judge Wins

Open-ended generation. Summarisation, drafting, creative writing, agent responses. There is no single correct answer, so exact match and string similarity are useless. A judge with a rubric handles this at scale.

Multi-dimensional grading. You want to score a response on helpfulness AND safety AND tone AND correctness. A judge can score all four in one pass. Building four separate metric pipelines would take much longer.

Nuanced correctness. The response is technically correct but subtly misleading. Or technically wrong in a way a user would forgive. String metrics miss the nuance; judges catch it.

Volume beyond human review. You have 10,000 examples per week. Human review would cost R50,000 per week. LLM-as-judge grades them for maybe R20-100.

If you find yourself thinking "I could tell if the response was good if I read it", you have a judge candidate.

Where LLM-as-Judge Fails

Objective, deterministic tasks. If the answer is 42, checking equals 42 is faster, cheaper and more accurate than asking a judge "is this response about equal to 42".

Tasks the judge model is bad at. Judges inherit their own model's blind spots. A judge that struggles with maths will grade a maths answer generously if it looks plausible.

Safety-critical judgments where false negatives cost lives. Medical, legal, financial advice at scale. Judges have false-negative rates you cannot ignore. Human review or specialised classifiers are needed here, not LLM-as-judge alone.

When the judge and the system under test are the same model. Judge-your-own-homework. Covered in Module 1 Lesson 3. Use a different (ideally stronger) model as the judge.

The Judge as Additional Failure Point

Adding a judge means adding another model call to your eval loop. That means:

• Cost per grade. Every eval example costs whatever the judge call costs. For frequent eval runs on large datasets, this adds up fast (Lesson 4 of this module).
• Latency per grade. Judge calls are typically 1-3 seconds each. Running 100 examples serially takes 2-5 minutes. Parallelising helps.
• Non-determinism in the grade itself. The judge might grade the same output 4 today and 3 tomorrow. Even the score is a distribution.

You mitigate these with sampling (only grade a portion of runs), caching (identical outputs get the same grade) and running multiple judge samples for critical evaluations.

A Judge is Never a Substitute for a Baseline

The most common misconception: teams reach for LLM-as-judge as a shortcut around defining what "good" means. They ask a judge "grade this response" without providing a rubric. The judge produces a number. The number is meaningless.

A judge is only as good as the rubric you give it. Lesson 2 is dedicated to writing rubrics.

When To Combine Judges With Other Metrics

Most production eval suites combine multiple metric families:

• Exact match on the parts that can be checked exactly (did the model return valid JSON, did the routing tag match the expected)
• LLM-as-judge on the open-ended parts (was the response helpful and safe)
• Occasional human review of a small sample to keep the judge calibrated

Never trust a single metric family for the whole eval. Layer them.

Key Takeaway

Reach for LLM-as-judge when you need to grade open-ended generation, multi-dimensional quality or large volumes beyond human review. Skip it for objective tasks with a clear correct answer. Every judge is another model call with cost, latency and non-determinism, so use it deliberately. And never confuse "we have a judge" with "we know what good means"; that is what the rubric is for.