A Field Guide to LLM and AI Agent Benchmarks

Model reports often list ARC-E, ARC-C, MMLU, GPQA, GSM8K, HumanEval, SWE-bench, GAIA, WebArena, and OSWorld side by side. Their scores are not interchangeable: each benchmark uses different tasks, tools, environments, inference budgets, and scoring rules.

This guide maps common evaluations from static question answering to agents completing real tasks, and explains what each benchmark can—and cannot—tell us.

Two common naming traps

1. SWE-bench is a benchmark; SWE-agent is not. SWE-bench provides repositories, issues, and tests. SWE-agent is an agent system that can attempt those tasks.
2. ARC-E / ARC-C are not ARC-AGI. The former are AI2 multiple-choice science questions; ARC-AGI uses colored grids for abstract rule induction.

1. Are we evaluating a model or a system?

An agent score is therefore a property of a complete setup:

1
2
3
4
5

agent result = model capability
             + agent scaffold
             + tools and environment
             + inference budget
             + evaluation protocol

2. Quick reference

3. Knowledge, commonsense, and textual reasoning

ARC-E and ARC-C

The AI2 Reasoning Challenge contains science exam questions from grades 3–9. ARC-Easy contains easier items; ARC-Challenge selects questions that retrieval and word-co-occurrence baselines struggle with.

It measures science knowledge, reading comprehension, option comparison, and short reasoning chains. Multiple-choice accuracy does not establish open-ended scientific ability, and easier splits can saturate.

HellaSwag and WinoGrande

HellaSwag asks the model to select the most plausible continuation of an everyday event. WinoGrande tests commonsense-sensitive coreference resolution.

Both are useful for lightweight commonsense testing, but candidates and templates can expose statistical shortcuts.

MMLU and MMLU-Pro

MMLU spans 57 subjects including mathematics, history, computer science, law, medicine, and philosophy. It measures breadth of knowledge and switching across academic formats—not browsing, tool use, or long-horizon execution.

MMLU-Pro expands questions from four to ten answer options and emphasizes harder reasoning. MMLU and MMLU-Pro scores are not directly interchangeable.

GPQA

GPQA contains 448 expert-written questions in biology, physics, and chemistry. The widely reported GPQA Diamond is a smaller, high-quality subset.

It targets difficult scientific reasoning, but its small size creates uncertainty. Reports should identify the split, tool access, and inference budget.

4. Mathematical reasoning

GSM8K

GSM8K contains roughly 8,500 grade-school word problems requiring two to eight elementary arithmetic steps. It tests extraction of quantitative relationships and multi-step calculation, usually via exact-match final answers.

It is not advanced mathematics, and its age makes contamination a concern.

MATH, MATH-500, and AIME

MATH covers competition-style algebra, geometry, number theory, and probability. MATH-500 is a common 500-problem subset.

AIME problems are harder competition questions with integer answers from 000 to 999. Always report the year range, number of samples, and reasoning budget: pass@1 and best-of-64 measure very different compute regimes.

5. Function-level code generation

HumanEval and MBPP

HumanEval contains 164 handwritten Python function tasks scored with hidden unit tests. MBPP contains mostly basic Python problems with reference code and tests.

Both commonly report pass@k: the probability that at least one of k generated candidates passes. They test small-program synthesis—not repository exploration, debugging, version control, or requirements discovery.

LiveCodeBench

LiveCodeBench continuously collects newer problems from competitive-programming platforms and uses temporal splits to reduce contamination. It also evaluates self-repair, code execution, and test-output prediction.

It better measures novel algorithmic coding than older static sets, but still differs from maintaining a real codebase.

6. Repository-level engineering: SWE-bench and SWE-agent

SWE-bench

SWE-bench gives a system a real GitHub repository and issue. The system must generate a patch that makes fail-to-pass tests succeed without breaking pass-to-pass tests. The main metric is % Resolved.

Important variants include Full, Lite, Verified, Multimodal, and Multilingual. SWE-bench Verified contains 500 tasks reviewed by software engineers for solvability and clarity.

Every score should name the dataset variant, agent scaffold, model version, step limit, test-time scaling, and harness version.

SWE-agent

SWE-agent is an Agent-Computer Interface that lets a model inspect files, search code, edit, run tests, and iterate. It can be evaluated on SWE-bench, but is not itself the benchmark.

1
2
3

SWE-bench = exam + repository environment + grader
SWE-agent = the problem-solving framework
LLM       = the model inside that framework

7. Function calling and tool use

BFCL

The Berkeley Function-Calling Leaderboard tests whether a model selects the right function and produces valid names, argument types, and values. It covers single, parallel, multiple, relevance-detection, multi-turn, and agentic scenarios.

AST evaluation checks call structure; executable evaluation runs calls. Strong single-turn function calling does not imply long-horizon planning or recovery.

ToolBench

ToolBench / ToolLLM evaluates API retrieval, argument generation, and multi-step use across a large API collection. It stresses finding the right tool among many candidates, though API availability and model-based judging can affect reproducibility.

τ-bench

τ-bench simulates realistic conversations where an agent must interact with a user, call domain APIs, and obey policies. It tests information gathering, state tracking, policy compliance, and actual environment changes.

Strict pass^k asks whether the system succeeds repeatedly, making reliability visible rather than rewarding one lucky trajectory.

8. General, browser, and computer-use agents

GAIA

GAIA contains 466 questions requiring combinations of reasoning, web research, multimodal file handling, and tools across three difficulty levels. It evaluates a general research assistant, not an isolated base-model skill.

AgentBench

AgentBench evaluates agents across operating systems, databases, knowledge graphs, games, web shopping, and browsing. Breadth is its strength; aggregate scores can hide environment-specific weaknesses.

WebArena

WebArena provides reproducible self-hosted commerce, forum, code-hosting, and content-management sites. Agents must turn goals into navigation, clicks, typing, and information retrieval. Evaluation usually checks final website state.

OSWorld

OSWorld evaluates multimodal agents in real operating systems and applications, including browsers, files, office tools, media apps, and cross-application workflows. Performance depends strongly on visual grounding, action interfaces, step limits, and environment stability.

PaperBench

PaperBench asks agents to understand AI papers, implement experiments, run code, and reproduce results. Fine-grained rubrics score long-horizon research execution. It is realistic but expensive and judge-dependent.

9. Reading the metrics

• Accuracy / exact match: simple and reproducible, but format-sensitive and blind to process quality.
• pass@k: at least one of k candidates succeeds; larger k means more compute.
• % Resolved / task success: execution-based and useful, but only as reliable as the environment and tests.
• LLM-as-a-judge / rubric score: handles open-ended outputs, but introduces judge bias and drift.
• Cost and latency: agent evaluations should also report tokens, calls, steps, latency, cost per attempt, and cost per success.
• Reliability: repeated-run averages, variance, and pass^k reveal stochastic failures hidden by a single run.

10. Selecting benchmarks for your use case

A practical evaluation suite should combine a relevant static capability set, an end-to-end task benchmark, a private continuously refreshed regression set, and measurements of cost, latency, and repeated-run stability.

11. Why high benchmark scores do not guarantee product reliability

• Contamination and saturation: public questions may appear in training data.
• Protocol mismatch: prompting, tools, samples, context, and step limits alter scores.
• Incomplete graders: passing tests does not prove style, performance, or hidden requirements.
• Averages hide profiles: equal overall scores can mask completely different strengths.
• Agents are stochastic: tools, web state, sampling, and long trajectories introduce variance.

Conclusion

Benchmarks form a progression toward real work:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

multiple choice
  ARC / MMLU / GPQA
      ↓
structured problem solving
  GSM8K / MATH / HumanEval
      ↓
tool invocation
  BFCL / ToolBench / τ-bench
      ↓
end-to-end environments
  SWE-bench / GAIA / WebArena / OSWorld
      ↓
long-horizon professional work
  PaperBench / private product evaluations

The closer an evaluation gets to real work, the more expensive and difficult it becomes to reproduce—and the harder it is to attribute results to the base model alone. The right question is not “Which benchmark is authoritative?” but “Does this task distribution, tool setup, and success criterion match the system I intend to deploy?”

Primary references

• AI2 ARC · MMLU · GPQA
• GSM8K · HumanEval · LiveCodeBench
• SWE-bench · SWE-agent
• BFCL · τ-bench
• GAIA · AgentBench
• WebArena · OSWorld · PaperBench