Research · vendor-reported scores · updated 2026-10-16
How the 2026 AI models score, by their makers’ own numbers
Every number below comes from Anthropic, OpenAI or Moonshot’s own launch posts, system cards and model cards, traced to the page that prints it. No column mixes two named versions of a test, though some vendors don’t name theirs. Select any score to see where it came from and the settings behind it.
- models
- 24
- benchmarks
- 15
- vendor scores
- 181
- verified in source
- 87%
Higher is better · 15 of 24 models reported
SWE stands for software engineering.
- What it tests
- Implementing features written from scratch for the benchmark, so no answer can be memorised: 113 tasks across 91 code repositories in five languages, graded by a separate verifier.
- What the score means
- Share of tasks whose code changes pass the verifier.
Before you compare
Only 113 tasks: near 70 percent, the sampling error is about 4 points, so close scores are effectively tied.
In other words: A building test with plans no candidate has seen before.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Partial: the number is confirmed, but a setting behind it is missing, or it is read only from a table cell.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Verified: the number is printed in the vendor’s own document.
Partial: the number is confirmed, but a setting behind it is missing, or it is read only from a table cell.
Partial: the number exists only as chart data on the vendor’s page.
Values that exist only as chart data are listed last, without a rank.
Not reported for this benchmark: Claude Haiku 5.5, Claude Sonnet 5, Claude Opus 4.7, Claude Sonnet 4.6, Claude Opus 4.6, GPT-5.5 Pro, Kimi K2.7 Code, Kimi K2.6, Kimi K2.5.
- Verified in the vendor’s text
- Partial: a setting is missing, or the value is chart data
- Other printed values (other settings or documents), or other chart points for a chart-only score
- Run by a third party
- Brightest bar: top printed score (settings can differ)
This table scrolls sideways to show all 15 benchmarks. Use the arrows, or pick a category above.
| Model | Coding | Reasoning | Agentic | Work | Safety | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Terminal-Bench 4.0, higher is better | DeepSWE v1.1, higher is better | FrontierCode 1.1 (Main), higher is better | SWE-bench Pro, higher is better | HLE, no tools: Humanity's Last Exam (no tools), higher is better | HLE, with tools: Humanity's Last Exam (with tools), higher is better | GPQA Diamond, higher is better | ARC-AGI-2, higher is better | OSWorld 2.1 (partial credit, 108 tasks), higher is better | BrowseComp (single agent), higher is better | AutomationBench (Zapier private set), higher is better | MCP Atlas, higher is better | Toolathlon-Verified (pass@1), higher is better | GDPval-AA v2 (Elo), higher is better | Prompt-injection attack success, k=15lower is better | |
| Anthropic11 models | |||||||||||||||
| Claude Haiku 5.52026-10-07 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||
| Claude Sonnet 5.52026-09-28 | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||||||
| Claude Opus 5.52026-09-22 | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||||||
| Claude Fable 5.12026-09-01 | Not reported | Not reported | Not reported | ||||||||||||
| Claude Opus 52026-07-24 | Not reported | ||||||||||||||
| Claude Sonnet 52026-06-30 | Not reported | Not reported | Not reported | Not reported | |||||||||||
| Claude Fable 52026-06-09 | Not reported | Not reported | Not reported | Not reported | |||||||||||
| Claude Opus 4.82026-05-28 | Not reported | Not reported | Not reported | ||||||||||||
| Claude Opus 4.72026-04-16 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||||
| Claude Sonnet 4.62026-02-17 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||||||||
| Claude Opus 4.62026-02-05 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||||||
| OpenAI9 models | |||||||||||||||
| GPT-6.1 Sol2026-09-29 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | ||
| GPT-6 Sol2026-09-22 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||
| GPT-6 Luna2026-09-22 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||
| GPT-6 Astra2026-09-03 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||||||||
| GPT-5.6 Sol2026-07-09 | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||||||
| GPT-5.6 Terra2026-07-09 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||
| GPT-5.6 Luna2026-07-09 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||||||
| GPT-5.52026-04-23 | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||||||
| GPT-5.5 Pro2026-04-23 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||
| Moonshot4 models | |||||||||||||||
| Kimi K32026-07-16 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||
| Kimi K2.7 Code2026-06-12 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |
| Kimi K2.62026-04-20 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | ||||||
| Kimi K2.52026-01-27 | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | Not reported | |||||
- Verified in the vendor’s text
- Partial: a setting is missing, or the value is chart data
- Other printed values (other settings or documents), or other chart points for a chart-only score
- Run by a third party
- Higher is better in every column except prompt injection
- Top printed score in the column (settings can differ)
- Not reported by the vendor
Scores by launch date
Each dot is one model, placed at the date it launched. Each line follows a maker’s best printed score so far. There are no “weak” or “strong” labels: every maker runs a test with its own tools and settings, so models a few points apart are effectively tied.
- Anthropic
- OpenAI
- Moonshot
- Hollow: partial, or chart data only
15 of 24 models reported DeepSWE v1.1. Values that exist only as chart data are hollow and never set a line point.
Show these scores as a table
| Model | Maker | Launched | Effort | Score |
|---|---|---|---|---|
| GPT-5.5 | OpenAI | 2026-04-23 | xhigh | 67 |
| Claude Opus 4.8 | Anthropic | 2026-05-28 | Not stated | 59.0 (partial) |
| Claude Fable 5 | Anthropic | 2026-06-09 | Not stated | 69.7 (partial) |
| GPT-5.6 Luna | OpenAI | 2026-07-09 | max | 67.2 |
| GPT-5.6 Sol | OpenAI | 2026-07-09 | max | 72.7 |
| GPT-5.6 Terra | OpenAI | 2026-07-09 | max | 69.6 |
| Kimi K3 | Moonshot | 2026-07-16 | max | 67.5 |
| Claude Opus 5 | Anthropic | 2026-07-24 | max | 68.8 |
| Claude Fable 5.1 | Anthropic | 2026-09-01 | Not stated | 67.4 |
| GPT-6 Astra | OpenAI | 2026-09-03 | xhigh | 74.1 |
| Claude Opus 5.5 | Anthropic | 2026-09-22 | Not stated | 74.2 |
| GPT-6 Luna | OpenAI | 2026-09-22 | max | 66.6 |
| GPT-6 Sol | OpenAI | 2026-09-22 | max | 68.8 |
| Claude Sonnet 5.5 | Anthropic | 2026-09-28 | Not stated | 71.0 |
| GPT-6.1 Sol | OpenAI | 2026-09-29 | max | 71.90 (chart data) |
The 15 benchmarks, explained
A benchmark is a fixed set of tasks with a way of checking the answers. Here is what each one in the table asks of a model, and what to keep in mind before comparing its numbers.
Coding
Terminal-Bench 4.0
- What it tests
- Getting real jobs done in a Linux terminal: building software, repairing a broken system, reproducing a result. Scripts check the machine’s final state.
- What the score means
- Share of tasks completed, usually averaged over several tries per task.
Results depend heavily on the agent program wrapped around the model, and with a small task count, gaps of a few points are noise. Never compare across versions (2.0, 2.1 and 4.0 are different tests).
In other words: A practical exam: here is a broken workshop and a job card; the examiner checks the finished workshop.
DeepSWE v1.1
SWE stands for software engineering.
- What it tests
- Implementing features written from scratch for the benchmark, so no answer can be memorised: 113 tasks across 91 code repositories in five languages, graded by a separate verifier.
- What the score means
- Share of tasks whose code changes pass the verifier.
Only 113 tasks: near 70 percent, the sampling error is about 4 points, so close scores are effectively tied.
In other words: A building test with plans no candidate has seen before.
FrontierCode 1.1 (Main)
- What it tests
- Whether a code change is one a project maintainer would actually merge: correct, in scope, in the project’s style and properly tested.
- What the score means
- A quality-weighted score from 0 to 100; edits nobody asked for are penalised.
No error bars, and more reasoning effort can lower the score, because over-eager models make out-of-scope edits.
In other words: A carpentry job judged not only on whether the shelf holds, but on whether the client would accept it.
SWE-bench Pro
SWE stands for software engineering.
- What it tests
- Harder code fixes that touch several files at once, in Python, Go, JavaScript and TypeScript projects.
- What the score means
- Share of tasks fixed, checked by hidden tests.
In July 2026 OpenAI published an audit finding a large share of tasks broken, and each vendor uses its own agent program, so cross-vendor numbers are not comparable.
In other words: A repair shop, but jobs that touch several parts of the machine at once.
Reasoning
Humanity's Last Exam (no tools)
Often shortened to HLE.
- What it tests
- 2,500 expert-written questions with short, exact answers, chosen because earlier models failed them. This column is the model alone, with no search or code.
- What the score means
- Percent of questions answered correctly, as judged by a grader model.
Each vendor runs its own setup, grader included, so cross-vendor gaps mean less than they look. Independent audits have questioned some answer keys.
In other words: A final exam written by hundreds of professors, each contributing their hardest question.
Humanity's Last Exam (with tools)
Often shortened to HLE.
- What it tests
- The same 2,500 expert questions, but the model may search the web, fetch pages and run code.
- What the score means
- Percent of questions answered correctly, as judged by a grader model.
Which tools are allowed, and which sites are blocked to stop answer leaks, differs by vendor, so this column is a weak cross-vendor comparison.
In other words: The same professors’ exam, taken open book.
GPQA Diamond
GPQA stands for Graduate-Level Google-Proof Q&A: questions that can’t be answered by searching.
- What it tests
- 198 graduate-level multiple-choice science questions that experts get right and skilled non-experts with web access get wrong.
- What the score means
- Percent correct; random guessing scores 25.
Saturated: 2026 flagships score in the 90s, and on 198 questions a 2-point gap is about 4 questions.
In other words: A PhD qualifying quiz that cannot be passed by searching the web.
ARC-AGI-2
ARC stands for Abstraction and Reasoning Corpus; AGI for artificial general intelligence.
- What it tests
- Learning a new rule from a few examples, in coloured-grid puzzles.
- What the score means
- Percent of puzzles solved.
Scores the ARC Prize Foundation verifies itself sit beside vendors’ own runs, and the public and semi-private puzzle sets are different tests.
In other words: An IQ-style puzzle book where every page follows a rule you have to work out.
Agentic
OSWorld 2.1 (partial credit, 108 tasks)
OS as in operating system: the model works a real desktop computer.
- What it tests
- Using a computer like a person: the model sees screenshots and works the mouse and keyboard through long office workflows (108 of them in this release).
- What the score means
- Partial credit: the average share of checkpoints reached in each workflow.
The same runs score far lower when every checkpoint must be met (Opus 5.5: 81.8 partial, 48.7 strict), and releases and subsets differ, so compare only within this column.
In other words: A temp worker given a login and a to-do list, judged by what the files look like at the end of the day.
BrowseComp (single agent)
- What it tests
- Finding hard-to-locate facts on the open web; answers are short and easy to check, but obscure. 1,266 questions.
- What the score means
- Percent of questions answered correctly, by a single agent.
Token budgets and context handling move the score, and multi-agent runs (shown elsewhere, not here) score higher. Agents have been caught finding leaked answers.
In other words: A scavenger hunt where each clue leads to a single fact buried deep in the library.
AutomationBench (Zapier private set)
- What it tests
- Carrying out a business request across 47 simulated apps (CRM, spreadsheets, email, chat) with nobody to ask for clarification, checked for both the result and any collateral damage.
- What the score means
- Strict share of tasks completed correctly; no partial credit.
Release numbers are rarely printed, so a column can mix runs, and some vendors count safeguard interventions differently. Kimi K3’s figure uses the public task set.
In other words: An office assistant handed one sticky note and access to every system, judged on whether everything ended up in the right place.
MCP Atlas
MCP stands for Model Context Protocol, a standard way to connect AI models to tools and data.
- What it tests
- Picking the right tools from real tool servers, among decoys, and chaining them correctly to answer a request.
- What the score means
- Answers scored claim by claim, as a percentage.
Scale AI changed the judge and re-scored models in April 2026, and Moonshot runs its own numbers on a subset with a different judge.
In other words: A mechanic choosing the right tools from a crowded wall and using them in the right order.
Toolathlon-Verified (pass@1)
Short for Tool Decathlon.
- What it tests
- Long jobs done through real applications such as Gmail, Google Calendar and Google Sheets, checked afterwards by inspecting what changed.
- What the score means
- Pass@1: share of tasks completed on a single try.
The original set and the Verified set are different tests, and Anthropic says its own harness runs a few points above the public one.
In other words: A week’s worth of office errands done through real apps, checked by an auditor afterwards.
Professional work
GDPval-AA v2 (Elo)
GDPval is named for GDP (gross domestic product); AA is Artificial Analysis, which runs this version.
- What it tests
- Producing real professional work (memos, spreadsheets, presentations) across 44 occupations, judged blind against other models’ work by an AI judge.
- What the score means
- An Elo rating, like chess: a higher number means the model’s work is preferred more often, not by how much in absolute terms.
Elo shifts with the pool of models and the date of the snapshot, and the scale changed twice in 2026; compare only within this column.
In other words: Asking applicants to do a sample piece of work for a real job and having a panel pick the better one.
Prompt injection
Prompt-injection attack success, k=15
Gray Swan is the security company that runs it; IPI stands for indirect prompt injection.
- What it tests
- Hidden instructions planted in what an AI agent reads (an email, a web page, a file), and how often a persistent attacker gets the agent to obey them across coding, tool use and computer use.
- What the score means
- Attack success within 15 attempts, across all surfaces. Lower is better.
It measures the model plus its maker’s safeguards, which differ by vendor. OpenAI tested a safeguards-enabled checkpoint.
In other words: A con artist slipping a forged instruction into the mail an assistant opens, with several tries to get the wording right.
Where the numbers come from
Only what each vendor published about its own models: system cards, launch posts, model cards and technical reports. No independent leaderboards, and no estimates read off charts. A score someone else ran and the vendor republished is marked 3P.
How it was checked
Four blind audit rounds re-found 1,383 values in the vendors’ documents with no copying errors. A final blind pass then re-read all 4,541 scores in the full dataset and found no wrong score; it corrected 14 effort or scoring-mode labels and made one printed bound exact.
Read it carefully
Each vendor picks its own tools and effort settings, so cross-vendor gaps of a few points rarely mean much. Where a vendor reports one model at several effort levels, the page shows the highest and lists the rest in the score’s details. An empty cell means the vendor did not report that test, not that the model failed it.