LLM selection requires evaluating multiple dimensions beyond raw benchmark performance. A model that tops a public leaderboard can still perform poorly on your specific task, for reasons the leaderboard was never designed to catch: the benchmark itself may be contaminated, the task may not resemble anything in the benchmark suite, and the properties that matter in production — grounded, consistent, appropriately-hedged answers on your actual data — usually aren't what the benchmark measured in the first place.
Public benchmark scores are a weak signal for production model selection because of well-documented contamination — training data overlapping with test sets — that can inflate reported accuracy significantly. Practical LLM evaluation instead requires building a task-specific test set from your own data, evaluating groundedness and hallucination rate directly using explicit scored evaluators rather than impression, and being explicit about the known biases of "LLM-as-judge" evaluation approaches (position, verbosity, and self-preference bias) if you use them. Benchmarks are a reasonable first filter to narrow candidate models; they should never be the final decision criterion for a production use case.
Key Takeaways
- Public benchmarks can be significantly contaminated. Research on benchmark decontamination found that testing on a clean, uncontaminated mirror of the GSM8K math benchmark reduced measured accuracy by up to 13 percentage points for some model families, and separate studies found contamination levels reaching up to 45%, with some analyses reporting far higher rates for specific models.
- A high benchmark score does not transfer to your specific task. Benchmarks test general capability on curated problems; your production use case likely involves your own data formats, domain terminology, and failure modes the benchmark never covers.
- "LLM-as-judge" evaluation is useful but has documented, quantified biases — including position bias, verbosity bias (preferring longer answers), and self-preference bias (a model favoring its own outputs) — that should be actively mitigated, not assumed away.
- Groundedness matters more than fluency for most production use cases. A confident, well-written, factually wrong answer is more dangerous than an answer that correctly says "I don't know."
- Groundedness and relevance can be scored directly, not just impressionistically — evaluation frameworks like Azure AI Foundry's evaluation SDK return a numeric score, a pass/fail threshold, and a stated reason for each response, rather than a single aggregate quality rating.
- Evaluation should be continuous, not a one-time gate before launch. Model providers update models, your data distribution shifts, and users find new ways to prompt the system — a static pre-launch eval doesn't catch any of that.
Why Benchmark Scores Are a Weaker Signal Than They Look
Contamination inflates reported scores
Benchmark contamination happens when public test-set questions end up, directly or indirectly, in a model's pretraining data — through the original benchmark repository, derivative datasets, or model distillation from another contaminated model. Research explicitly measuring this found that evaluating models against a clean, uncontaminated mirror of the GSM8K math-reasoning benchmark reduced accuracy by up to 13 percentage points for model families like Mistral, and reported contamination levels reaching up to 45% on commonly used benchmarks, with some analyses of specific models finding contamination as high as roughly 92% on popular multilingual benchmarks (arXiv — "LLM Benchmark Datasets Should Be Contamination-Resistant"). In practical terms: a benchmark score may reflect how well a model memorized the test set during training, not how well it generalizes to problems it hasn't seen — which is exactly the property you actually need for a production use case involving your own data.
Benchmarks don't measure your task
Even an uncontaminated benchmark score describes performance on the benchmark's specific task distribution — general knowledge questions, curated math problems, coding challenges from a specific source. If your production use case is "answer customer questions grounded in our product documentation" or "extract structured data from our specific document format," no general benchmark directly measures that. The only evaluation that reliably predicts production performance is one built from examples that resemble your actual task.
Evaluating What Actually Matters in Production
Groundedness and hallucination rate
For most enterprise use cases — support, internal knowledge assistants, document processing — the central question is whether the model's claims are grounded in the source material it was given, not whether it sounds fluent. A practical groundedness evaluation checks each claim in a model's output against the retrieved source context and flags claims that aren't supported, distinct from checking whether the answer is well-written.
This no longer has to be a manual, impressionistic check. Azure AI Foundry's evaluation SDK, for example, provides dedicated Groundedness, Relevance, and Retrieval evaluators for RAG systems, each returning a 1–5 score with a pass/fail label (default threshold: 3) and a stated reason for the score, rather than a single opaque quality rating (Microsoft Learn — RAG evaluators in Microsoft Foundry). A separate Retrieval evaluator scores whether the chunks retrieved for a query were actually relevant to it, which is useful for isolating whether a groundedness failure came from the retrieval step or the generation step — a distinction that matters for knowing what to actually fix.
Consistency across near-identical inputs
Production users will phrase the same underlying request in many different ways. A model that gives materially different answers to functionally equivalent prompts is a reliability problem even if any single answer, in isolation, looks acceptable.
Calibrated refusal
A model that never says "I don't know" or "I'm not certain" will eventually state a wrong answer with full confidence on a question outside its actual knowledge. Evaluating whether a model appropriately hedges or escalates on questions it shouldn't answer confidently is as important as evaluating accuracy on questions it should be able to answer.
Task-specific accuracy on your own held-out examples
The most reliable evaluation set is one you build from real or realistic examples of your actual task, ideally including edge cases and known failure patterns from any existing pilot. This is more work than running a public benchmark, and it is also the only evaluation that actually predicts how the model will perform for you.
Using "LLM-as-Judge" Evaluation Carefully
Using a second, typically stronger LLM to score another model's outputs (an "LLM-as-judge" pattern) has become a common way to scale evaluation beyond what human review can cover — but research quantifying the biases in this approach found meaningful, structural problems: position bias (favoring answers based on their placement/order rather than content), verbosity bias (favoring longer responses independent of quality), and self-enhancement bias (a model favoring outputs it or a similar model generated), among twelve distinct bias categories the study measured, with "robustness rates" (consistency of judgment under bias-probing conditions) varying widely by model and bias type — position bias showed some of the lowest robustness scores in the study (Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge).
If you use LLM-as-judge scoring, randomize answer order to control for position bias, avoid using the same model family as both the system under test and the judge to reduce self-preference bias, and validate the judge's scores against a sample of human review — don't treat automated judge scores as ground truth without spot-checking them.
Comparison: Evaluation Approaches
| Approach | What it's good for | Limitation |
|---|---|---|
| Public benchmarks | Quickly narrowing a long list of candidate models | Vulnerable to contamination; doesn't measure your specific task |
| Task-specific held-out test set | The most reliable predictor of real production performance | Requires effort to build and maintain; needs real or realistic examples |
| Scored RAG evaluators (groundedness, relevance, retrieval) | Isolating whether a failure is in retrieval or generation | Still an automated proxy; benefits from periodic human validation |
| LLM-as-judge scoring | Scaling evaluation beyond manual human review capacity | Documented position, verbosity, and self-preference biases require active mitigation |
| Human review | Ground truth for validating both benchmarks and automated judges | Doesn't scale to high volume or continuous monitoring on its own |
Common Mistakes
Given documented contamination affecting some benchmarks significantly, a leaderboard position gap of a few points is often not a meaningful signal, particularly if you haven't checked whether your candidate models were evaluated against contamination-resistant test data.
Model providers update models behind the same API endpoint, usage patterns evolve, and data drifts — evaluation needs an ongoing cadence, not a single gate before go-live.
Not randomizing answer order, or using a judge from the same model family as the system under test, reintroduces exactly the biases the research on this method has already quantified.
A well-written, confident answer is not the same as a grounded one — evaluation criteria should explicitly separate the two rather than letting a human reviewer's impression of "this reads well" substitute for a factual accuracy check.
A handful of friendly example prompts won't surface the edge cases and adversarial phrasing production users will actually generate.
Cost, Complexity and Timeline
Running public benchmarks against candidate models is low effort and reasonable as an initial filter, but should not be the final decision step. Building a task-specific evaluation set is moderate-to-significant effort depending on how much real or representative data you have available, and is the highest-leverage investment for reliable model selection. Setting up scored evaluators (groundedness, relevance, retrieval) is low-to-moderate effort when using a managed evaluation SDK rather than building scoring logic from scratch, though it still requires representative test data to run against. Setting up LLM-as-judge scoring at scale is moderate effort technically, with an ongoing need to periodically validate judge output against human review to catch drift or bias creeping back in. Continuous evaluation in production (monitoring live outputs, not just pre-launch testing) is an ongoing operational commitment, not a one-time project, and should be resourced as such.
How Halden Solutions Approaches This
We treat public benchmark scores as a first-pass filter, never the final basis for a model decision. Model evaluation for a client's specific use case starts with building a small, representative task-specific test set from real or realistic examples, evaluating groundedness and calibrated refusal directly — using scored evaluators rather than relying solely on fluency impressions where a managed evaluation SDK is available — and, where LLM-as-judge scoring is used to scale evaluation, actively controlling for the position, verbosity, and self-preference biases that published research has quantified, rather than trusting automated scores without spot-checking against human review.
Frequently Asked Questions
No — they're a reasonable way to narrow a long list of candidate models quickly. The mistake is treating a benchmark score as the final basis for a production decision, given documented contamination issues and the gap between benchmark tasks and your actual use case.
It's when test questions from a public benchmark end up, directly or through derivative data, in a model's training data — so a high score partly reflects memorization of the test set rather than the ability to solve genuinely new problems.
Groundedness checks whether the model's final response is actually supported by the context it was given; retrieval checks whether the context that was fetched for the query was relevant in the first place. Scoring both separately helps identify whether a failure originated in the retrieval step or the generation step, rather than treating "the answer was wrong" as one undifferentiated problem.
It can meaningfully scale evaluation beyond what manual human review can cover, but research has quantified real biases (position, verbosity, self-preference) that require specific mitigation — randomized answer order, judge/system model diversity, and periodic validation against human review — rather than being used unchecked.
There's no universal fixed interval; it should be tied to signals that the input distribution, provider model version, or observed error patterns have changed, with a minimum periodic review cadence regardless, since silent model updates behind a provider's API can change behavior without your team changing anything.
Conclusion
Public benchmark scores are a starting filter, not a production decision — documented contamination and the mismatch between benchmark tasks and real use cases both limit how much a leaderboard rank should influence your choice. The evaluation that actually predicts production performance is built from your own task's examples, measures groundedness and calibrated refusal directly using scored evaluators rather than impression, and runs continuously rather than once before launch — with any LLM-as-judge scoring explicitly checked against its known biases rather than trusted by default.
If you're selecting or evaluating a model for a production AI use case, Halden Solutions' Data & AI team can help design an evaluation approach that actually predicts how it will perform for you. See also our guide to generative AI content moderation.
Sources
- arXiv — "LLM Benchmark Datasets Should Be Contamination-Resistant"
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (research site)
- Microsoft Learn — RAG evaluators in Microsoft Foundry (Groundedness, Relevance, Retrieval)
- Microsoft Learn — Observability in Generative AI
- NIST — "AI Risk Management Framework"