Surveys the advanced benchmarks that define state-of-the-art LLM evaluation: contamination-free coding tests like LiveCodeBench, factuality benchmarks such as SimpleQA, multimodal reasoning challenges like MMMU-Pro, the FACTS suite, and agentic planning simulations like Vending-Bench 2. Also explains the reference-based versus reference-free metric distinction.
Once frontier models cluster near the ceiling of a benchmark, that benchmark stops telling you which model is better. The response has been a move toward contamination-free and task-specific tests that measure genuine capability rather than memorized training data. The scores below are representative reference points, not a live leaderboard — treat them as illustrations of what each benchmark probes.
The benchmark landscape
| Benchmark | Focus Area | Key Insight | Representative Score (SOTA) |
|---|---|---|---|
| LiveCodeBench | Coding (contamination-free) | Draws problems from LeetCode/AtCoder/CodeForces published after model training cutoffs to measure true generalization. | ~86.6% (GPT-5 Mini) |
| SimpleQA / SimpleQA Verified | Short-form factuality (parametric knowledge) | 1,000 curated prompts with single indisputable answers, testing internal knowledge with no tools. | ~55.6% (Gemini 2.5 Pro) |
| MMMU-Pro | Multimodal reasoning | Expert-level questions across 6 disciplines (30 subjects, 183 subfields) requiring vision-text integration. | ~81.0% (Gemini 3 Pro) |
| FACTS Benchmark Suite | Factuality across 4 dimensions | Covers Grounding, Multimodal, Parametric, and Search slices. | ~68.8% overall (Gemini 3 Pro) |
| Vending-Bench 2 | Agentic long-horizon planning | Simulates running a business for a full year, testing strategic planning and consistent tool use over time. | $5,478 mean net worth (Gemini 3 Pro) |
The through-line: each benchmark isolates a capability that a single generic knowledge test blurs together — fresh-code generalization, tool-free recall, multimodal reasoning, factual grounding, and sustained agentic planning.
Reference-based vs. reference-free metrics
Beyond named benchmarks, the metric you choose depends on whether a “correct answer” exists to compare against.
Reference-based metrics compare output to a gold-standard answer. They rely on lexical overlap (BLEU, ROUGE), semantic similarity (BERTScore), or exact match, and suit development and regression testing where there is one right answer — data extraction, math, structured parsing.
Reference-free metrics judge an output in isolation, with no target. They use proxy signals (fluency, coherence), safety classifiers (toxicity, bias), and custom LLM judges, and suit production monitoring of open-ended tasks — chatbots, creative writing, moderation — where ground truth is impractical.
For the judge-based side of reference-free scoring, see LLM-as-a-Judge Methodology.
Ask Lynx Assist
More Guides
Run disciplined SEO A/B tests in seven steps — one metric, two variations, randomized segments, run to significance, track, analyze the winner, and iterate.
Build a topic cluster in seven steps — select and score a pillar, validate it, map subtopics, align to intent, architect internal links, publish, and measure.
Prepare your site for AI search in five steps — content architecture, entity consistency, E-E-A-T, structured data, and machine-readable structure.
Get your content cited by AI in seven steps — answer capsules, link-free extraction, original data, digital PR, community presence, consistent messaging, and tracking.
A seven-step walkthrough for setting up Google Search Console on a new site — property type, DNS verification, sitemap, GA4 link, users, URL checks, and a monitoring routine.