Evaluation Results · Hugging Face (original) (raw)

This is a work in progress feature.

The Hub provides a decentralized system for tracking model evaluation results. Benchmark datasets host leaderboards, and model repos store evaluation scores that automatically appear on both the model page and the benchmark’s leaderboard.

Benchmark Datasets

Dataset repos can be defined as Benchmarks (e.g., MMLU-Pro, HLE, GPQA). These display a “Benchmark” tag and automatically aggregate evaluation results from model repos across the Hub and display a leaderboard of top models.

Model Evaluation Results

Evaluation scores are stored in model repos as YAML files in the .eval_results/ folder. These results:

• Appear on the model page with links to the benchmark leaderboard
• Are aggregated into the benchmark dataset’s leaderboards
• Can be submitted via PRs and marked as “community-provided”

Adding Evaluation Results

To add evaluation results to a model, you can submit a PR to the model repo with a YAML file in the .eval_results/ folder.

Create a YAML file in .eval_results/*.yaml in your model repo:

• dataset: id: cais/hle
  task_id: default
  revision:
  value: 20.90
  verifyToken:
  date: "2025-01-15"
  source:
  url: https://huggingface.co/spaces/SaylorTwift/smollm3-mmlu-pro
  name: Eval traces
  user: SaylorTwift
  notes: "no-tools"

Or, with only the required attributes:

• dataset: id: Idavidrein/gpqa task_id: gpqa_diamond value: 0.412

Results display badges based on their metadata in the YAML file:

For more details on how to format this data, check out the Eval Results specifications.

Community Contributions

Anyone can submit evaluation results to any model via Pull Request:

1. Go to the model page and click on the “Community” tab and open a Pull Request.
2. Add a .eval_results/*.yaml file with your results.
3. The PR will show as “community-provided” on the model page while open.

For help evaluating a model, see the Evaluating models with Inspect guide.

Community scores are visible while the PR is open. If a score is disputed, the model author can close the PR to remove it. The goal is to surface existing evaluation data transparently while building toward a fully reproducible standard via verified scores.

Registering a Benchmark

To register your dataset as a benchmark:

1. Create a dataset repo containing your evaluation data
2. Add an eval.yaml file to the repo root with your benchmark configuration, conform to the specification defined below.
3. The file is validated at push time
4. (Beta) Get in touch so we can add it to the allow-list.

Examples can be found in these benchmarks: GPQA, MMLU-Pro, HLE, GSM8K.

Eval.yaml specification

The eval.yaml should contain the following fields:

• name — Human-readable display name for the benchmark (e.g. "Humanity's Last Exam").
• description — Short description of what the benchmark measures.
• evaluation_framework — Canonical evaluation framework identifier for this benchmark. This is an enumerable which the Hugging Face team maintains. Add your own to the list here. Exactly one framework is supported per benchmark.
• tasks[] — List of tasks (sub-leaderboards) defined by this benchmark (see below).

Required fields in each tasks[] item:

• id — Unique identifier for the task. (e.g. "gpqa_diamond"). A single benchmark can define several tasks, each producing its own leaderboard. Feel free to choose a leaderboard identifier for each task.

Optional fields in each tasks[] item:

• config — Configuration of the Hugging Face dataset to evaluate (e.g. "default"). Defaults to the dataset’s default config.
• split — Split of the Hugging Face dataset to evaluate (e.g. "test"). Defaults to "test".

When setting evaluation_framework: inspect-ai, one also requires to set the following fields:

• field_spec — Specification of the input and output fields. Consists of input, target, choices and optional input_image subfields. See the docs for more details.
• solvers — Solvers used to go from input to output using the AI model. This can range from a simple system prompt to self-critique loops. See the docs for more details.
• scores — Scorers used. Scorers determine whether solvers were successful in finding the right output for the target defined in the dataset, and in what measure. See the docs for more details.

Minimal example (required fields only):

name: MathArena AIME 2026 description: The American Invitational Mathematics Exam (AIME). evaluation_framework: math-arena

tasks:

• id: MathArena/aime_2026

Extended example:

name: MathArena AIME 2026 description: The American Invitational Mathematics Exam (AIME). evaluation_framework: "math-arena"

tasks:

• id: MathArena/aime_2026 config: default split: test

Extended example ("inspect-ai"-specific):

name: Humanity's Last Exam description: > Humanity's Last Exam (HLE) is a multi-modal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind with broad subject coverage. Humanity's Last Exam consists of 2,500 questions across dozens of subjects, including mathematics, humanities, and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. evaluation_framework: "inspect-ai"

tasks:

• id: hle config: default split: test

  field_spec: input: question input_image: image target: answer

  solvers:

  • name: system_message args: template: | Your response should be in the following format: Explanation: {your explanation for your answer choice} Answer: {your chosen answer} Confidence: {your confidence score between 0% and 100% for your answer}
  • name: generate

  scorers:

  • name: model_graded_fact args: model: openai/o3-mini

Update on GitHub