The Benchmark You Build

Every team that picks a model off a leaderboard is running an experiment somebody else designed, on a population that is not theirs, and treating the result as a decision.

That is rarely laziness. A public score is often the only number available, it arrives free, and it carries the authority of a paper and a chart. The trouble is that several forces pull that number away from your reality, and none of them announce themselves. The score does not become false. It becomes irrelevant, which is much harder to notice.

A leaderboard measures a model against a population of questions. You are buying a model to serve a population of users. Nobody has ever shown you that those two populations resemble each other.

Three ways a public number drifts

The first is contamination. Shuo Yang and colleagues showed that the standard defence, removing test items from training data by matching overlapping n-grams, does not survive paraphrase or translation. A rephrased item passes straight through the filter while remaining, to a model, the same item. They found 8 to 18 percent of the HumanEval benchmark present in widely used pretraining corpora, and demonstrated that a 13-billion-parameter model deliberately overfit to a rephrased test set can post numbers on par with GPT-4. The score is real. The competence behind it is not. Their proposed remedy, a decontaminator that compares embeddings rather than surface strings, works better and is still not a solution you can buy: it only runs on corpora you are allowed to see, which rules out every closed model on the board.

The second is ordinary reuse. Benjamin Recht and colleagues rebuilt the test sets for CIFAR-10 and ImageNet, following the original collection protocol as closely as they could, then re-scored a wide range of published models. Accuracy fell 11 to 14 percent on ImageNet and 3 to 15 percent on CIFAR-10. Nobody cheated. A decade of one research community tuning against one fixed sample was enough on its own to inflate every number in the literature. The useful detail is what survived: the ranking held up far better than the absolute scores. A leaderboard is a more honest ordering than it is a forecast, and teams get into trouble by reading it as the second.

The third is the incentive structure around the leaderboard itself. A 2025 audit of Chatbot Arena covering roughly two million battles, 42 providers and 243 models found undisclosed private testing, where a provider evaluates many variants and publishes only the winner, including 27 unreleased variants tested by one provider ahead of a single launch. The same work found that modest amounts of arena-specific data can produce relative gains up to 112 percent on the arena distribution, and that proprietary models are sampled in more battles than open-weight ones. The arena operators disputed parts of the analysis, which is worth knowing and does not change the structural point: a leaderboard that matters becomes a target, and a target measures fit to itself.

The leaderboards themselves keep telling you this

The strongest evidence here is not a critique paper. It is the behaviour of the people who run the leaderboards, and Hugging Face's Open LLM Leaderboard is the cleanest case.

Version 1 launched in mid-2023 and ran six benchmarks, ARC, HellaSwag, MMLU, TruthfulQA, WinoGrande and GSM8K, through a single standardized harness so numbers were comparable across submissions. It was archived in June 2024 because the models had saturated it: scores had bunched near the ceiling and the ruler no longer separated good from great. Version 2 replaced the set with harder tasks chosen to resist saturation, MMLU-Pro, GPQA, MuSR, IFEval, BBH and MATH Level 5. In March 2025, after evaluating more than 13,000 models, that was retired too. The stated reason reads better as an engineering warning than as an announcement: as capabilities moved toward reasoning and assistant behaviour, the benchmark suite was drifting toward measuring something the field had stopped caring about, and continuing to publish it would encourage teams to hill-climb an irrelevant direction.

Read that back. The best-resourced, most carefully maintained public evaluation in open-weights AI lasted about 21 months across two versions, and its maintainers shut it down rather than let people optimize against a stale target. Whatever leaderboard you are reading today sits somewhere on that same curve, and nothing on the page tells you where.

What Arabic makes impossible to ignore

Every team has this problem. Arabic teams find out sooner, because the gap between the benchmark and the user is visible in the language itself.

Arabic is diglossic. Modern Standard Arabic is the written and formal register, and it is what nearly every Arabic benchmark is composed in. Your users write dialect, and they switch into English mid-sentence for technical terms. A model can score well on formal Arabic comprehension and still fail a Gulf or Levantine support request, and no MSA benchmark will warn you.

The Arabic leaderboards have been unusually candid about this, and their revision history is worth studying. The Open Arabic LLM Leaderboard launched in May 2024 from 2A2I, TII and Hugging Face with fourteen benchmarks, several of them machine translations of English sets such as MMLU and EXAMS. Version 2, released in February 2025, dropped the translations outright. The reasoning was that a translated item carries the framing and cultural assumptions of the original and produces mismatches that have nothing to do with the model's Arabic. In their place came native sets: ArabicMMLU with roughly 15,000 items built from real school exams, AraTrust with 522 human-written questions on trustworthiness, MadinahQA, and ALRAGE, a retrieval-augmented generation set drawn from 40 Arabic books.

Buried in the same release is the detail that should stay with you. The maintainers found a silent bug in AlGhafa, one of the main tasks carried over from version 1, in how answer choices were handled. Correcting it moved scores by as much as 20 points, and it hit smaller models hardest. No model changed. The scoring code did. Anyone who had shortlisted candidates from that leaderboard the week before was partly ranking an artifact of the harness, and had no way to see it, because the harness is not the thing on the page.

AraGen, from Inception and MBZUAI, attacked the problem from the other end. Instead of multiple choice it scores free generation, using a metric called 3C3H: Correctness, Completeness, Conciseness, Helpfulness, Honesty and Harmlessness. Correctness is binary and gates the rest, so a wrong answer zeroes every other dimension and a fluent falsehood cannot out-score a blunt fact. The more interesting design choice is operational. AraGen Bench is only 279 human-verified questions, and it stays private for three months at a time before being published and replaced by a fresh set of comparable structure and difficulty. That is a public leaderboard voluntarily adopting the two disciplines this article argues for: keep the set small enough to curate properly, and keep it private long enough for the number to mean something.

Dialect is the remaining hole, and AraDiCE is the serious attempt at it. Built by QCRI and published at COLING 2025, it covers Gulf, Egyptian and Levantine Arabic across roughly 45,000 post-edited samples, plus a benchmark for cultural awareness in those regions. Its headline finding is one to keep: Arabic-specific models such as Jais and AceGPT beat larger multilingual models on dialect work, and dialect identification and generation remain hard for everything tested.

Some of that square I have had to build. In work on Jordanian dialect speech recognition with Ali Safieh and Rawan Ghnemat, there was no labelled corpus to evaluate against, so we assembled one: 97,000 records and 113 hours of speech, of which only 52 hours could be labelled at all. The comparison point for Modern Standard Arabic was the 1,000-hour Massive Arabic Speech Corpus. That ratio is the argument in a single line.

The resolution turned out to matter more than the label suggests. Jordanian Arabic carries phonemes the MSA set does not, and it splits into urban, rural and Bedouin varieties inside one country, which means even "Levantine" is too coarse a bucket to make a purchase on. Our best system reached 51 percent word error rate after five generations of self-supervised training. That number is not a failure. It is what an honest measurement looks like the first time it is taken on the right population, and no public leaderboard would have produced it.

All of these are good, and you should run them. They still stop one square short of where your product lives.

What each kind of evaluation is actually good for

None of this makes public benchmarks worthless, and a team that responds by ignoring them has traded one error for another. The useful move is to be explicit about the division of labour, including where your own set is the weaker instrument.

What a public leaderboard does well

• Screening. It cuts a field of forty candidates down to five for free, in an afternoon, before you have spent anything.
• Comparability at a scale you cannot reach. One harness, one prompt format, one parser, run across thousands of models. You will never reproduce that internally, and without it you cannot say whether a 4-point gap is the model or the plumbing.
• Ordering, more than absolute accuracy. This is the practical lesson from the ImageNet rebuild: rankings held up much better than the scores did. Trust a leaderboard to tell you which models are in the conversation, not what accuracy you will see.
• Regression detection. A new checkpoint that drops several points on IFEval has broken something general, and you want to know that before you evaluate it on anything of your own.
• Adversarial scrutiny. The AlGhafa scoring bug was found and disclosed because many independent people ran the same harness. Your internal set has one pair of eyes on it, and its bugs stay yours.
• Coverage you cannot afford to build. Expert-level science questions, multi-domain safety, long-context retrieval. You are not going to construct a GPQA.

What it structurally cannot do

• It carries no prior about your distribution, so it cannot tell you whether the gap between two models will appear at all on your traffic.
• It cannot be decontaminated for a closed model, because you cannot inspect the training corpus.
• It is perishable, on a horizon of one to two years, and the page never says how far along it is.
• It reports a point estimate with no interval, which invites teams to read a 1-point gap as a ranking.
• It measures a model, not your system. Your prompt, retrieval layer, tools, latency budget and cost ceiling are all excluded from the number and are usually where your quality actually comes from.
• Once it matters commercially, it becomes a target, and it starts measuring who most wants to be at the top of it.

What your own set costs you

Being honest about this is what keeps the practice alive after the first quarter.

• It costs real time. Two people for two weeks is the floor for the first version, and it recurs every quarter, forever.
• Small samples mean wide intervals. Five hundred items cannot resolve small differences, and pretending otherwise is the same mistake in a smaller font.
• It encodes your annotators' assumptions, including the ones they are not aware of. Your blind spots become your ceiling.
• It goes stale. Traffic drifts, the product changes, and a set built two years ago is a fossil of an older user.
• It says nothing about general capability. A model can be excellent on your 500 items and poor at everything adjacent, which matters the day someone expands the scope of the product.
• You can overfit to it exactly the way the field overfit ImageNet, and you will not notice, for the same reasons they did not.

That last risk is the one that dictates the build. It is the reason half the set stays sealed.

How to build one that survives contact

A useful evaluation set is smaller and duller than people expect. Three hundred to five hundred items will separate serious candidates, and that is arithmetic rather than a hedge. On 500 items with accuracy near 80 percent, the standard error of a single score is about 1.8 points, so the 95 percent interval spans roughly plus or minus 3.5 points. Two models three points apart on that set are tied, and reporting them as ranked repeats the error you are trying to escape.

You can recover most of that precision without buying more items by testing paired: run every candidate over the identical items and compare outcomes item by item rather than in aggregate. A paired test, McNemar's being the standard one, only looks at the items where two models disagree, so it detects far smaller true differences than two independent scores with overlapping intervals ever could. Ten thousand items, by contrast, mostly buys a slower feedback loop and a false sense of rigor.

Sample the items from real traffic rather than writing them from imagination. Imagination produces the questions you expect to receive, which your system already handles, because you built it while imagining the same things. Logs produce the ones you actually get, including the half-typed ones and the ones in the wrong language. Stratify the sample so dialects and task types appear in something close to their real proportions, then oversample the rare cases you cannot afford to fail.

Every item needs either a verifiable answer or an explicit rubric written before anyone scores a model. Label each item twice, independently, and keep a log of the disagreements, because that rate is the measurement's ceiling: if two of your own experts agree only 82 percent of the time, a model reporting 94 percent is telling you about your rubric, not about the model. Record it as Cohen's kappa rather than raw agreement. Raw agreement flatters you on skewed data, since two annotators who both answer "no violation" on a set that is 90 percent compliant will agree 90 percent of the time while having measured nothing at all. Kappa subtracts that chance floor. Below about 0.6, fix the rubric before you score a single model.

Then split the set. Keep a private half that never leaves machines you control, never goes into a prompt to a hosted API, and never gets pasted into a vendor console during a bake-off. Contamination stops being hypothetical the moment a set has been travelling to third parties for a year.

Scoring without fooling yourself

Use exact match wherever the task allows it, because it costs nothing and cannot flatter anyone. For open-ended work you will end up using a model as the judge, which is reasonable and needs care.

Lianmin Zheng and colleagues found that a strong judge model agrees with human preferences more than 80 percent of the time, which is about the rate at which humans agree with each other. The same paper documents the failure modes: position bias, where the answer shown first wins more often; verbosity bias, where longer answers score higher regardless of quality; and self-enhancement bias, where a judge prefers output from its own model family. Randomize the presentation order, control for length, and never let a vendor's model judge that vendor's model in a procurement decision.

The AraGen team measured all of this on their own leaderboard and published the numbers, which puts a useful floor under how much to trust a judge. Their judge was highly reliable at the narrow task of detecting hallucination, above 96 percent, and still landed at a Cohen's kappa near 0.44 against human raters on the full rubric, which is moderate agreement, not strong. They also quantified self-preference directly: GPT-4o scored itself 0.8025 against 0.7928 when scored by other judges. That gap is small, consistent, and almost exactly the size of margin that procurement decisions turn on. Notably, a jury of several judges did not change the rankings, so the answer is not more judges. It is publishing the agreement number next to the score.

So calibrate, and say so. A score without an agreement number attached is a rumour with a decimal point.

What leaders should do

1. Pull 500 real requests from last quarter's logs, stratified by task type and by dialect, and make that the seed of the evaluation set. Budget two people for two weeks, not a research program.
2. Split it in half. The public half is for iterating and for sharing with vendors. The private half never touches a hosted API, and it is the one that decides purchases.
3. Require every model claim, internal or from a vendor, to be restated as a score on your set, with a confidence interval and the judge's human agreement rate beside it. Two models whose intervals overlap are tied; break that tie on cost, latency, licence or data residency, not on the decimal.
4. Measure agreement between your own annotators as kappa and treat it as the ceiling. When a model reaches it, stop tuning and go work on something else, because past that point you are fitting noise in your own rubric.
5. Put the quarterly re-sample in the calendar and retire any item that turns up in vendor material or public documentation. This is the step that gets cut, and cutting it is how the set quietly becomes a fossil.

Model weights are a commodity now, and they change every few months. The evaluation set is the part of the stack that appreciates. It encodes what your organization means by correct, in your language and your domain, and it holds that meaning stable while everything underneath it is replaced. Teams that own one can switch providers in an afternoon. Teams that do not are choosing between vendors on the strength of report cards those vendors printed, from an exam that will be retired before the contract is up.

Arabic makes the argument concrete, because a benchmark written in formal Arabic plainly does not describe a user typing Gulf dialect with English technical terms mixed in. But nothing here is specific to Arabic. Any team whose traffic differs from the average of the internet is in exactly the same position, and most of them have not noticed, because the leaderboard number keeps going up.