👑 Best overall
Haiku 5.5 · High
Cost 2.2× · Score 58.6 %
280
Grunt work
Haiku 5.5 · Low
Cost 1.0× · Score 51.0 %
122
Everyday tasks
Haiku 5.5 · High
Cost 2.2× · Score 58.6 %
280
Advanced reasoning
Opus 5.5 · xHigh
Cost 74× · Score 73.9 %
253
Cutting-Edge thinking
Opus 5.5 · xHigh
Cost 74× · Score 73.9 %
253
Quality vs. cost of every Claude model, at each effort level
One curve per model, one point per effort (low → max). Cost is a multiple of the cheapest couple shown; quality is placed on the model's latent scale and labelled with the score it predicts on the benchmark panel. Below the weakest tier target the quality axis is compressed, so the models short of today's level sit together at the bottom. Tier bands (off by default) shade the four usage tiers of the tier picker below : each band runs from that tier's target up to the next tier's — the qualities that clear its bar but not the next one. Uncertainty ovals (off by default) show each couple's 16–84 % interval, width = cost, height = quality; hover to bring the one under the cursor forward. Haiku 4.5 lives only in the matrix and Pareto.
Best Claude model for the price — the Pareto frontier
The non-dominated (model, effort) couples : nothing offers more quality at equal-or-lower cost. Solid = frontier, faded = dominated. The frontier couples are the candidates of every pick below. The faint grey line is the price trend — what a given quality costs today, fitted on every couple of the latest model of each family. Tier bands (off by default) shade the same four usage tiers as on the landscape above.
Haiku 5.5Low51.0 % · 1.0×→Haiku 5.5Medium55.5 % · 1.4×→Haiku 5.5High58.6 % · 2.2×→Haiku 5.5xHigh61.8 % · 4.3×→Haiku 5.5Max64.2 % · 12×→Sonnet 5.5High66.9 % · 18×→Sonnet 5.5xHigh70.9 % · 33×→Opus 5.5High72.2 % · 43×→Opus 5.5xHigh73.9 % · 74×→Opus 5.5Max74.7 % · 194×
Value — distance to the price trend
The price trend — ln(cost) = a + λ·θ, θ the latent quality — is what a given quality costs today across the 20 couples of the latest model of each family, not just the frontier: it is the going rate of the models on sale. Earlier generations, dearer per point than their successors, would flatten it and make quality look cheap; it does not move when older models are shown (R² ≈ 0.69, drawn faint grey above). Each couple weighs by its uncertainty across the line, both axes counted. On the latent scale the trend is a straight line: each step of quality costs the same ratio more, wherever it sits. A couple's distance to it says how many times cheaper or dearer it is than the trend charges for its quality — the same gap read on the other axis is how much more quality it gives than the trend at its cost. No couple serves as a reference. Only the frontier couples are listed.
| Model · effort | cost | expected score | value index trend = 100 |
|---|---|---|---|
| Haiku 5.5 · High | 2.2× | 58.6 % | 280 |
| Opus 5.5 · High | 43× | 72.2 % | 278 |
| Haiku 5.5 · xHigh | 4.3× | 61.8 % | 276 |
| Sonnet 5.5 · xHigh | 33× | 70.9 % | 264 |
| Opus 5.5 · xHigh | 74× | 73.9 % | 253 |
| Haiku 5.5 · Medium | 1.4× | 55.5 % | 231 |
| Sonnet 5.5 · High | 18× | 66.9 % | 191 |
| Haiku 5.5 · Max | 12× | 64.2 % | 166 |
| Haiku 5.5 · Low | 1.0× | 51.0 % | 122 |
| Opus 5.5 · Max | 194× | 74.7 % | 118 |
Value index = 100 × how many times cheaper than the trend the couple is for its quality (100 = the trend). Green = above 100, red = below; intensity = distance to the trend in decades, capped at one. A couple on the Pareto frontier can still be red : being non-dominated does not mean being good value for what it delivers.
Best value by task tier — what to actually run
Consolidating the frontier into a decision. Four target qualities are derived from the data, not fixed: they spread evenly between the frontier's weakest and strongest couples, each end drawn 5 % of the span inward, so that the bottom tier follows the weakest couple available and the top tier the best — as models improve, the targets move with them. The width follows the spacing on one rule : adjacent windows cross at half weight midway between their targets. For each tier we pick, among the frontier couples, the one that maximises window × eβ·λθ ⁄ cost, λ the slope of the price trend and β the tier's price of quality as a multiple of the market's : grunt work pays a quarter of it (β = 0.25), everyday tasks 0.63, advanced reasoning 1.59, cutting-edge thinking four times (β = 4) — evenly spread in ratio, symmetric about the market. Below its target a couple is penalised by a Gaussian window ; at or above it the window is 1, and eβ·λθ ⁄ cost rewards smoothly a couple that brings more quality than the tier is ready to pay for its extra cost. Cost counts in ratios, as people perceive prices : twice as dear weighs the same at every price. The big figure is the pick's value index : 100 = the price trend, 500 means five times cheaper than the trend charges for its quality, 50 twice as dear. The crown is the frontier couple that sits furthest below the trend. Sliders below override the targets, widths and prices of quality.
Target 52.2 % – Grunt work
Haiku 5.5 · Low
Cost 1.0× · Score 51.0 %
122
Target 59.7 % – Everyday tasks
Haiku 5.5 · High
Cost 2.2× · Score 58.6 %
280
Target 67.1 % – Advanced reasoning
Opus 5.5 · xHigh
Cost 74× · Score 73.9 %
253
Target 73.7 % – Cutting-Edge thinking
Opus 5.5 · xHigh
Cost 74× · Score 73.9 %
253
👑 Best overall
Haiku 5.5 · High
Cost 2.2× · Score 58.6 % · Value index 280
Picked as the frontier couple that sits furthest below the price trend : it costs 2.8× less than the trend charges for its quality — or, read on the other axis, it scores 4.9 points of expected score above what the trend gives for its cost. Its value index is that ratio times 100: 100 = the trend, fitted on every couple of the latest model of each family, so today's going rate of the models, not the frontier; no couple serves as a reference.
Tune the tiers (target, σ, price of quality β)
Normalized cost matrix — Claude effort levels compared
Cost per model × effort as a multiple of the cheapest couple, sorted by decreasing expected score. Every cell is computed from the measurements by the fusion model described in the method below: a cost on a task of typical size, with the couple's own 16–84 % interval. Intensity = cost.
| Model | low | medium | high | xhigh | max | Expected score top effort |
|---|---|---|---|---|---|---|
| Opus 5.5 | 2523–27 | 3533–38 | 4341–46 | 7468–81 | 194178–212 | 74.7 % |
| Sonnet 5.5 | 1211–13 | 1312–14 | 1817–20 | 3330–37 | 115104–126 | 72.8 % |
| Fable 5.1 | 7065–76 | 8478–90 | 10295–111 | 186171–202 | 293270–319 | 70.6 % |
| Opus 5 | 3935–43 | 5753–62 | 8477–90 | 111103–121 | 148137–159 | 69.6 % |
| Fable 5 | 6963–75 | 9588–102 | 115105–125 | 165151–180 | 270247–294 | 67.8 % |
| Haiku 5.5 | 1.00.89–1.1 | 1.41.3–1.5 | 2.22.0–2.4 | 4.33.8–4.8 | 1210–13 | 64.2 % |
| Sonnet 5 size-sensitive | 1614–18 | 2321–25 | 3330–36 | 5045–54 | 10999–120 | 60.1 % |
| Haiku 4.5 | 13merged · 11–15 | 40.0 % | ||||
"Expected score · top effort" = the score the model at its top effort is expected to reach on the benchmark panel, from the same fit as the landscape. Haiku 4.5 = a single merged cell (single operating point : no discrete effort parameter, see the measurement sources). n/a : effort not exposed, or measured by a single publisher (a couple is shown only once two publishers measured it).
Value-index matrix — every model and effort against the price trend
The value index of every couple shown, model × effort, in the same order as the cost matrix : 100 = the price trend, what a given quality costs today across the latest model of each family ; 500 means five times cheaper than the trend charges for that quality, 50 twice as dear. Under each value, its 16–84 % range.
| Model | low | medium | high | xhigh | max |
|---|---|---|---|---|---|
| Opus 5.5 | 11596–137 | 240204–283 | 278236–327 | 253210–304 | 11899–141 |
| Sonnet 5.5 | 5242–63 | 9276–111 | 191160–229 | 264216–322 | 12099–145 |
| Fable 5.1 | 3227–37 | 4438–52 | 4539–52 | 3631–43 | 2824–32 |
| Opus 5 | 3529–42 | 5143–59 | 5346–61 | 5245–60 | 4338–49 |
| Fable 5 | 1311–16 | 2017–23 | 2421–29 | 2118–25 | 1613–19 |
| Haiku 5.5 | 12297–154 | 231186–287 | 280227–345 | 276219–349 | 166134–206 |
| Sonnet 5 | 43–5 | 86–9 | 119–13 | 108–12 | 86–9 |
| Haiku 4.5 | 10–1 | ||||
Green = above 100, cheaper than the trend for its quality ; red = below ; the deepest green is the best index shown, the deepest red the lowest, on a log scale. A high index does not make a couple a pick : the tiers weigh it against their target quality.
What moves everything — four factors behind the numbers
The tier picks above are an order of magnitude. Four factors, each measured elsewhere, modulate them strongly — keep them in mind when reading the tiers and the matrix.
Task type
Effort and model choice don't weigh the same depending on the nature of the work — review, reasoning and pure coding reward effort very differently.
Complexity
The harder the task, the more effort costs — up to the point where extra thinking starts to hurt (the inverted-U).
Generation
Each release buys the same quality for less — Sonnet 5.5 beats Sonnet 5's max quality at its own high effort for about a fifth of the cost, Opus 5.5 beats Fable 5.1's max quality at its own high effort for about a sixth of the cost, Fable 5.1 beats Fable 5's max quality at its own xHigh effort for about three quarters of the cost, and Opus 5 reaches Fable-5-class quality for a little over half the cost per task; Sonnet 5 is the verbose exception, cheap per token but talkative.
Cache & harness
Two invisible factors — cache-read rate and the surrounding agent harness — that can dominate both cost and score.
Data Sources
Every source below measured at least two (model, effort) couples on the same task; 10378 cost and 16011 score measurements enter the fusion. Each row below is one such source, with the actual configuration we verified (harness · effort), its type, and the couples it links.
| Source | Actual config (harness · effort) | Type | Linked couples (same task) |
|---|---|---|---|
| AA CritPt artificialanalysis | AA model payload · CritPt physics, 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA GDP-PDF artificialanalysis | AA model payload · GDP-PDF all-pass, 4 models × ladder + 3 at max; noisy ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA GDPval v2 artificialanalysis | AA evaluation page · 220 tasks, Elo; cost derived from AA's published token counts ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 4.6·max · Opus 5·max · Sonnet 5·max · Opus 5·low · Sonnet 5·low · Opus 5·medium · Sonnet 5·medium · Opus 5·high · Sonnet 5·high · Opus 5·xhigh · Sonnet 5·xhigh |
| AA Harvey LAB artificialanalysis | AA model payload · Harvey's Legal Agent Bench (AA's own run, separate from Vals's HLAB), score-only, 8 models × ladder + 4 at max ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·max · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 4.6·max |
| AA HLE artificialanalysis | AA model payload · HLE no tools, 4 models × ladder + 3 at max ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA Index artificialanalysis | AA-Index · cross-model at max + Opus 5 sweep low→max ✓ | sweep | Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Fable 5·max · Opus 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh |
| AA Index v4 artificialanalysis | AA model pages · Fable 5.1 sweep low→max + Fable 5/Opus 5 at max, per-suite $ ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max |
| AA Index v4.3 artificialanalysis | AA · index v4.3 of 7 Sep (TB 4.0 + AutomationBench-AA), Fable 5.1 and Opus 5 × full ladder ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.8·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA MLCR artificialanalysis | AA model payload · MLCR-AA, score-only; Sonnet 5.5 max-only, Opus 5 full ladder, 5 others at max ✓ | sweep | Sonnet 5.5·max · Sonnet 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 4.6·max · Haiku 5.5·max |
| AA Omniscience artificialanalysis | AA model payload · Omniscience accuracy (not the penalised index) ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA SciCode artificialanalysis | AA model payload · SciCode, 4 models × ladder + 3 at max ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA TB 4.0 artificialanalysis | AA model payload · Terminal-Bench 4.0 (AA harness), 4 models × ladder + 3 at max; floor scores blank ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA Terminal-Bench-Science artificialanalysis | AA model payload · Terminal-Bench-Science 0.1 (AA harness, separate from Vals's board), score-only, 4 models × ladder + 2 at max ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA-Briefcase artificialanalysis | AA model payload 27 Sep · agentic knowledge work, Elo; 4 models × full ladder + 3 at max; costPerTask per eval ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 4.6·max · Opus 5·max · Sonnet 5·max · Opus 5·low · Sonnet 5·low · Opus 5·medium · Sonnet 5·medium · Opus 5·high · Sonnet 5·high · Opus 5·xhigh · Sonnet 5·xhigh |
| AA-LCR artificialanalysis | AA model payload · long-context reasoning, input-dominated cost ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| Agents on Rails · features rubyonrails | rubyonrails.org/ai + rails/ai-evals runs · 20 tickets ×3, miniswen; Opus 5 & Fable 5.1 high→max, Opus 5.5 default ✓ | sweep | Opus 5.5·medium · Fable 5.1·high · Fable 5.1·max · Opus 5·high · Opus 5·max |
| ai-arena london github-ai-arena | Claude Code · london-island, Fable 5 high vs max, tokens only ✓ | sweep | Fable 5·high · Fable 5·max |
| AI4AI-Bench arxiv-2608.20318 | arXiv 2608.20318 · 10 ML-research repos, measured exploration spend, 2 models × ladder ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| Angular shopping cart, Opus 5.5 effort ladder (OctAcademy video) youtube-octacademy | Claude Code v2.1.280; same CLAUDE.md and prompt; fresh branch per run; one run per level; 10-point checklist scored by a fresh Claude session; 2026-10-16; read 2026-10-18 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max |
| Anthropic docs: max_tokens cap vs cost (internal repo tasks) (16k cap) anthropic-docs | ~135 internal repository tasks ('cap measurement'); Opus 5.5 at its default (medium), Fable 5.1 at its default (high); cost = whole-dollar bar labels, digitised from cost-intel-max-tokens-saving-opus-5-5.png; read 2026-10-01 ✓; 16k max_tokens cap only; score only | sweep | Opus 5.5·medium · Fable 5.1·high |
| Anthropic docs: max_tokens cap vs cost (internal repo tasks) (64k cap) anthropic-docs | ~135 internal repository tasks ('cap measurement'); Opus 5.5 at its default (medium), Fable 5.1 at its default (high); cost = whole-dollar bar labels, digitised from cost-intel-max-tokens-saving-opus-5-5.png; read 2026-10-01 ✓; 64k max_tokens cap only; score only | sweep | Opus 5.5·medium · Fable 5.1·high |
| ARC-AGI-1 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; scores ≥97.5 % flagged gen5-saturated ✓ | sweep | Opus 5·high · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Sonnet 4.6·high · Sonnet 4.6·max · Haiku 4.5·nothink · Haiku 4.5·think-1k · Haiku 4.5·think-8k · Haiku 4.5·think-16k · Haiku 4.5·think-32k · Fable 5.1·max · Fable 5.1·xhigh · Fable 5.1·high · Fable 5.1·medium · Fable 5.1·low · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Haiku 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low |
| ARC-AGI-2 anthropic-syscard | Opus 4.7 card p213 · sweep low→max, $ cost ✓ | sweep | Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max |
| ARC-AGI-2 arcprize | ARC Prize evaluations.json · semi-private, costPerTask per model × effort; Opus 5.5 & Fable 5.1 full ladders ✓ | sweep | Opus 5·high · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Sonnet 4.6·high · Sonnet 4.6·max · Haiku 4.5·nothink · Haiku 4.5·think-1k · Haiku 4.5·think-8k · Haiku 4.5·think-16k · Haiku 4.5·think-32k · Fable 5.1·max · Fable 5.1·xhigh · Fable 5.1·high · Fable 5.1·medium · Fable 5.1·low · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low |
| arXiv 2609.13463 · TRAIL root-cause attribution arxiv-2609.13463 | Root-cause attribution of agent failures on TRAIL, Continual Search judge in the Claude Agent SDK, turn 4 · weighted F1 and cumulative mean cost per trajectory (USD), read from the vector coordinates of figures 4–5 (they agree with the numbers the text states); Opus 4.8 and Sonnet 5 at low/high/max (figure 5), Fable 5 at default settings (figure 4, whose Opus and Sonnet points are figure 5's high runs); read 30 Sep 2026 ✓ | sweep | Opus 4.8·low · Opus 4.8·high · Opus 4.8·max · Sonnet 5·low · Sonnet 5·high · Sonnet 5·max · Fable 5·default |
| ArXivMath no-tools anthropic-syscard | Opus 5.5 card p182 · ArXivMath Aug 2026, 57 problems, 3 models × low→max, $ cost, scores printed ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max |
| ArXivMath tools anthropic-syscard | Opus 5.5 card p183 · ArXivMath Aug 2026 with code sandbox, 3 models × low→max, scores printed ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max |
| AutomationB. 5 anthropic-syscard | Opus 5 card p180 · Zapier, linear-$ chart; Fable=Mythos 5, effort inferred ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| BBQ effort (interface-gap) arxiv-2609.08861 | direct API · budget_tokens low/med/high, Sonnet4.6 & Haiku4.5 ✓ | sweep | Sonnet 4.6·think-1k · Sonnet 4.6·think-4k · Sonnet 4.6·think-16k · Haiku 4.5·think-1k · Haiku 4.5·think-4k · Haiku 4.5·think-16k |
| BenchCAD V2C anthropic-syscard | Opus 5.5 card p205 · Vision2Code with tools, voxel IoU, 4 models × 5 efforts; effort inferred from cost order ✓ | sweep | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| BenchCAD V2C −tools anthropic-syscard | Sonnet 5.5 card p129 · tools disabled, 4 models × 5, separate regime kept out of the effort grid ✓ | sweep | Sonnet 5.5·nt-low · Sonnet 5.5·nt-medium · Sonnet 5.5·nt-high · Sonnet 5.5·nt-xhigh · Sonnet 5.5·nt-max · Opus 5.5·nt-low · Opus 5.5·nt-medium · Opus 5.5·nt-high · Opus 5.5·nt-xhigh · Opus 5.5·nt-max · Fable 5.1·nt-low · Fable 5.1·nt-medium · Fable 5.1·nt-high · Fable 5.1·nt-xhigh · Fable 5.1·nt-max · Sonnet 5·nt-low · Sonnet 5·nt-medium · Sonnet 5·nt-high · Sonnet 5·nt-xhigh · Sonnet 5·nt-max |
| BrowseComp 10M anthropic-syscard | Opus 5 card p160 · 10M budget, sweep low→max; Fable=Mythos 5, $ cost ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| BrowseComp 500-cut (Anthropic docs) anthropic-docs | Fable 5, 1-3 runs per setting ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·default |
| bug-hunt-bench github-bughuntbench | Claude Code 2.1.251 · 105 planted bugs, blind judge, 3 models × ladder ✓ | sweep | Fable 5.1·max · Fable 5.1·high · Fable 5.1·low · Fable 5·max · Fable 5·high · Opus 5·max · Opus 5·xhigh · Opus 5·high · Opus 5·medium · Opus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Opus 4.8·high · Opus 4.8·max · Sonnet 5·high · Sonnet 5·max · Fable 5.1·medium · Fable 5.1·xhigh · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo |
| BullshitBench v2 github-bullshitbench | petergpt/bullshit-benchmark c1d4ab6 · 100 nonsense prompts, share of clear push-backs (green), effort from the board's reasoning column; primary leaderboard ✓ | sweep | Opus 4.8·nothink · Opus 4.8·xhigh · Sonnet 4.6·high · Sonnet 4.6·nothink · Fable 5.1·low · Opus 4.7·nothink · Fable 5·xhigh · Opus 5·xhigh · Sonnet 5·low · Sonnet 5·max · Opus 5·low · Fable 5.1·max · Fable 5·low · Haiku 4.5·think-high · Opus 5.5·low · Opus 5.5·max · Haiku 4.5·solo · Opus 4.7·max · Sonnet 5.5·low · Sonnet 5.5·max · Haiku 5.5·max · Haiku 5.5·low |
| Buoy: Haiku 5.5 default vs high (256 food-ordering-app jobs) buoy.gg | 256 jobs in Buoy's own test app on iOS Simulators, 1 run each; score = jobs done right %; cost per job from tokens x list price; read 2026-10-15 ✓ | sweep | Haiku 5.5·default · Haiku 5.5·high |
| CFG k3s DevOps computingforgeeks | computingforgeeks Sonnet 5.5 review · 3 tasks × 3 runs, raw API single file; score = k3s deploy + HTTP 200 pass rate of the K8s task, cost summed over the 3 tasks ✓ | sweep | Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5·high · Opus 5.5·default |
| Chartography +tools anthropic-syscard | Fable 5.1 card p186 · 5 models × sweep low→max, $ cost; Opus 4.8 series joined from Opus 5 card p171 ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low |
| Chartography −tools anthropic-syscard | Fable 5.1 card p186 · tools disabled — separate regime, kept out of the effort grid ✓ | sweep | Fable 5.1·nt-low · Fable 5.1·nt-medium · Fable 5.1·nt-high · Fable 5.1·nt-xhigh · Fable 5.1·nt-max · Opus 5·nt-low · Opus 5·nt-medium · Opus 5·nt-high · Opus 5·nt-xhigh · Opus 5·nt-max · Fable 5·nt-low · Fable 5·nt-medium · Fable 5·nt-high · Fable 5·nt-xhigh · Fable 5·nt-max · Sonnet 5·nt-low · Sonnet 5·nt-medium · Sonnet 5·nt-high · Sonnet 5·nt-xhigh · Sonnet 5·nt-max · Opus 4.8·nt-low · Opus 4.8·nt-medium · Opus 4.8·nt-high · Opus 4.8·nt-xhigh · Opus 4.8·nt-max · Opus 5.5·nt-low · Opus 5.5·nt-medium · Opus 5.5·nt-high · Opus 5.5·nt-xhigh · Opus 5.5·nt-max · Sonnet 5.5·nt-low · Sonnet 5.5·nt-medium · Sonnet 5.5·nt-high · Sonnet 5.5·nt-xhigh · Sonnet 5.5·nt-max |
| Chartography −tools regraded anthropic-syscard | Sonnet 5.5 card p127 · re-graded no-tools scores, quality only, costs in chartogn ✓ | sweep | Sonnet 5.5·nt-low · Sonnet 5.5·nt-medium · Sonnet 5.5·nt-high · Sonnet 5.5·nt-xhigh · Sonnet 5.5·nt-max · Opus 5.5·nt-low · Opus 5.5·nt-medium · Opus 5.5·nt-high · Opus 5.5·nt-xhigh · Opus 5.5·nt-max · Fable 5.1·nt-low · Fable 5.1·nt-medium · Fable 5.1·nt-high · Fable 5.1·nt-xhigh · Fable 5.1·nt-max · Sonnet 5·nt-low · Sonnet 5·nt-medium · Sonnet 5·nt-high · Sonnet 5·nt-xhigh · Sonnet 5·nt-max |
| Chartography regraded anthropic-syscard | Opus 5.5 card p202 · same transcripts as chartogt, re-graded scores — quality only, costs stay in chartogt ✓ | sweep | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| Classmethod Bedrock prime fn classmethod | DevelopersIO 23 Sep · Opus 5.5 low/medium/high, 1 run, output tokens only ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high |
| claude-effort-bench github-effort-bench | Claude Code 2.1.206 · 3 multi-file tasks × 5 reps, Opus 5 & Sonnet 5 × medium→xhigh, CLI cost (Sonnet ×2/3), deterministic verifiers; ~30 % silent no-op runs scored 0 ✓ | sweep | Opus 5·medium · Opus 5·high · Opus 5·xhigh · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh |
| claudecode_lab · invoice check note-claudecode_lab | note.com 29 Sep · Claude Code 2.1.284 -p, Sonnet 5.5 and Opus 5.5 × low/medium/high/max, 7/7 everywhere (saturated), cost from Claude Code ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·max |
| CodeRabbit OSS-Aug coderabbit | CodeRabbit blog 22 Sep · 80 patterns, Opus 5.5 Standard vs Max, recall, score only ✓ | sweep | Opus 5.5·default · Opus 5.5·max |
| Cookbook cost optimization anthropic-cookbook | Anthropic cookbook cost_optimization.ipynb · 10-claim agent ×2 trials, Opus 5 & Sonnet 5 low→high, Haiku 4.5; list-price cost from usage ✓ | sweep | Opus 5·high · Opus 5·medium · Opus 5·low · Sonnet 5·high · Sonnet 5·medium · Sonnet 5·low · Haiku 4.5·solo |
| Corpus defect sweep (Anthropic docs) anthropic-docs | 21.6M-token corpus, F1, per episode ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Sonnet 5·default |
| CursorBench cursorbench | Sonnet 5 card p118 · 5 models × sweep low→max, $ cost, scores printed ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| CursorBench 3.2 cursorbench | Cursor 3.2 · matched effort, $ cost ✓ (vendor benchmark) | sweep | Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·xhigh · Fable 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Opus 5·low · Opus 5·medium · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| CursorBench 4.0 cursorbench | cursor.com payload · v4.0 long-horizon suite, Opus 5/5.5, Fable 5.1, Sonnet 5/5.5 × low→max, measured tokens × list incl. cache ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| CyberQ code-debug prompt cyberq | Opus 5.5 ladder + Opus 5 default/medium, single runs, list-price est. ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5.5·default · Opus 5·high · Opus 5·medium |
| CyberQ security short answer cyberq | Opus 5.5 ladder + Opus 5, no score ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·high · Opus 5·medium |
| CyberQ structured extraction cyberq | Opus 5.5 ladder + Opus 5, no score ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·high · Opus 5·medium |
| Databricks · Claude Code databricks | Opus 4.8 high/xhigh/max + Sonnet 5/4.6, Haiku 4.5 ✓ | sweep | Opus 4.8·high · Sonnet 5·default · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 4.6·default · Haiku 4.5·solo |
| Databricks · Pi harness databricks | Opus 4.8 high/xhigh/max ✓ | sweep | Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max |
| DeepResearch futuresearch | Deep Research Bench · low/high ✓ | sweep | Sonnet 4.6·low · Sonnet 4.6·high |
| DeepResearch Bench II (Anthropic docs) anthropic-docs | 50-task subset, 33-task basis, customer-billed ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5·low · Fable 5·medium · Fable 5·high · Sonnet 5·default · Opus 5·default |
| DeepSearchQA anthropic-syscard | Fable 5 card p270 · Fable=Mythos 5, sweep low→max, $ cost ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| DeepSearchQA anthropic-syscard | Opus 4.7 card p200 · Opus4.7 vs Sonnet4.6, sweep low→max ✓ | sweep | Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| DeepSearchQA (O5) anthropic-syscard | Opus 5 card p161 · 980k budget, 4 models × sweep low→max, $ cost ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| DeepSWE v1.1 datacurve | Datacurve live JSON · 113 tasks × 4 runs, mini-swe-agent, 4 models × full ladder + Sonnet 4.6 high, mean $/attempt ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·high |
| DeepWideSearch (Anthropic docs) anthropic-docs | Fable 5, 220 questions, 3 runs ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·default |
| dende SLA-deadline fn note-dende2023 | Claude Code 2.1.284 -p, no tools · Sonnet 5.5 low→max + Opus 5.5 medium/high, 15 tests, n=1–2, CLI list-price cost; saturated ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·max · Opus 5.5·medium · Opus 5.5·high |
| DRACO anthropic-syscard | Fable 5 card p271 · Fable=Mythos 5, sweep low→max, $ cost ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| DRACO (F5.1) anthropic-syscard | Fable 5.1 card p179 · 980k budget, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max |
| DRACO (O5.5) anthropic-syscard | Opus 5.5 card p187 · 980k budget, 3 models × low→max, $ cost, scores printed; a new run, not the F5.1-card one ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| DRACO (Opus 5) anthropic-syscard | Opus 5 card p162 · 980k budget, sweep low→max; Fable=Mythos 5, $ cost ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| effort-pick sweep github-effort-pick | API · 4 short writing tasks, Claude Opus (5, unverified) × low→xhigh, output tokens only ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh |
| effortmining pilot github-effortmining | Claude Code 2.1.201 · 12 short tasks × 3 reps, Opus 4.8 × low→max, median output tokens, pass rate (saturated ≥high) ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max |
| forge Phase 1 github-forge-benchmark | Claude Code · 3 tasks, Sonnet 4.6 medium/high/max, correct/partial only ✓ | sweep | Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| FrontierCode main cognition | Cognition leaderboard JSON · v1.1 main (100 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| FrontierCode v1 anthropic-syscard | Sonnet 5 card p117 · 4 models × sweep low→max, $ cost, scores printed ✓ | sweep | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| FrontierCode-D anthropic-syscard | Fable 5 card p257 · Fable=Mythos 5, sweep low→max, $ cost ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| FrontierCode-Ext cognition | Cognition leaderboard JSON · v1.1 Extended (150 tasks), Claude models × low→max incl. Sonnet 5.5, measured USD/rollout ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| FrontierFinance arxiv-2608.11683 | arXiv 2608.11683 Fig. 9 · Finance Agent v2 harness, Opus 4.8 low→xhigh, API cost/query (vector-read) ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh |
| GDPval (Anthropic docs) anthropic-docs | Fable 5, 1 run, cost incl. grading ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·default |
| George Liu · Opus 5 vs 5.5 ai.georgeliu | Claude Code 10-prompt suite · low→max · IFEval /9 ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max |
| george-liu ai.georgeliu | Claude Code · low/max ✓ | sweep | Opus 4.7·low · Opus 4.7·max |
| gitdiagram: Haiku 5.5 diagram generation (27 diagrams per setup), arrows backed by the code per blind judge GPT-6 Sol github-gitdiagram | ahmedkhaleel2004/gitdiagram experiments/diagram-haiku/README.md (2026-10-07, committed 22:30Z); Haiku 5.5 on Anthropic API, identical prompts; judge GPT-6 Sol (not Claude); score = 100 - %arrows not backed by code (judged runs: 19 / 19 / 9); cost = USD per diagram as billed in this eval (no caching for Haiku); read 2026-10-15 ✓ | sweep | Haiku 5.5·nothink · Haiku 5.5·low · Haiku 5.5·medium |
| Haiku 5.5 launch page: GDPval-AA v2.1, accuracy vs cost anthropic-chart | anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| Haiku 5.5 launch page: Humanity's Last Exam (no tools), accuracy vs cost anthropic-chart | anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| Haiku 5.5 launch page: OSWorld 2.1 (offline subset), accuracy vs cost anthropic-chart | anthropic.com/claude-haiku-5-5 launch page, embedded chart CSV (exact, not digitised); read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| Haiku 5.5 system card: AA-Briefcase v1.1, Haiku 5.5 at max vs default(medium) effort anthropic-syscard | text sentence (p.131), not a chart; only 2 points stated for Haiku 5.5 (max, default); read 2026-10-08 ✓ | sweep | Haiku 5.5·max · Haiku 5.5·default |
| Haiku 5.5 system card: HealthBench by effort, 5 levels anthropic-syscard | 1 run/level below max, max = mean of 5 runs; read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| Haiku 5.5 system card: HealthBench Professional (length-adjusted) by effort, 5 levels anthropic-syscard | 5 runs/level except 13 repeat runs at max on a different day (mean 64.2% vs reported 64.8%, see confound); read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| Haiku 5.5 system card: OSWorld 2.1 (offline subset) strict pass rate by effort, 5 levels (digitised) anthropic-syscard | strict pass rate (full-credit, binary) vs effort; cost x-axis identical to the partial-credit OSWorld chart and to the launch-page OSWorld chart (same underlying runs); only the max point (37.1%) is independently stated in text (p.128), the other four come only from this chart; read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| Haiku 5.5 system card: PhysicianBench by effort, 5 levels, pass rate anthropic-syscard | 5 attempts/task at each level; read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| haiku-thinking-bench: does thinking-off make Haiku 5.5 skip a needed handoff (24 cases x 10 runs, 80 handoff runs) github-haiku-thinking-bench | 17tayyy/haiku-thinking-bench README (Oct 2026); mechanical grading; score = handoffs made correctly of 80 (100 - missed%); cost = cost per conversation (tokens x list price, no caching); read 2026-10-15 ✓ | sweep | Haiku 4.5·solo · Haiku 5.5·nothink · Haiku 5.5·low · Haiku 5.5·medium |
| harnesseval github-harnesseval | Claude Code · 6 PRs common to every cell, CLI-metered cost ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Opus 5·low · Opus 5·medium · Opus 5·high |
| HealthBench (adj.) anthropic-syscard | Sonnet 5.5 card p138 · length-adjusted, Sonnet 5.5 low→max + 2 at max (digitized), score-only ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max |
| HealthBench Pro (adj.) anthropic-syscard | Sonnet 5.5 card p138 · length-adjusted, Sonnet 5.5 low→max + 3 at max, score-only ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5·max |
| HellaSwag effort (interface-gap) arxiv-2609.08861 | direct API · budget_tokens low/med/high, Sonnet4.6 & Haiku4.5 ✓ | sweep | Sonnet 4.6·think-1k · Sonnet 4.6·think-4k · Sonnet 4.6·think-16k · Haiku 4.5·think-1k · Haiku 4.5·think-4k · Haiku 4.5·think-16k |
| Hex DataBench (Opus 5.5) hex.tech | Hex DataBench v1.1, 100 analytics tasks in the Hex workspace, rubric-judged · score in %, average cost per task (USD), tokens per task (one total); effort from the row label; Opus 5.5 kept apart from the older rows: its scores fall on a grid of 1/200 against 1/300 for every other row, so it was run separately, possibly after the 17 Sep judge recalibration that 'especially impacted' Anthropic models; read 2026-10-01 ✓ | sweep | Opus 5.5·max · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low |
| HLE (Fable) anthropic-syscard | Fable 5 card p267 · Fable=Mythos 5, sweep low→max, $ cost ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| HLE (Opus 5) anthropic-syscard | Opus 5 card p157 · HLE tools, 4 models × sweep low→max, $ cost ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| HLE no-tools (F5.1) anthropic-syscard | Fable 5.1 card p178 · HLE without tools, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max |
| HLE no-tools (S55) anthropic-syscard | Sonnet 5.5 card p120 · 980k budget, 3 models × low→max, $ cost (perfect caching), scores printed ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| HLE sweep anthropic-syscard | Opus 4.8 card p203 · HLE tools, sweep low→max ✓ | sweep | Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max |
| HLE tools (F5.1) anthropic-syscard | Fable 5.1 card p177 · HLE with tools, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max |
| HLE tools (O5.5) anthropic-syscard | Opus 5.5 card p185 · HLE with tools, 3 models × sweep low→max, $ cost, scores printed ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| ibragim.dev · Claude ibragim-dev | Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Claude Code 2.1.281, effort per row, pass@1 and $/task; board of 28 Sep ✓ | sweep | Sonnet 5.5·high · Opus 5.5·high · Sonnet 5·high |
| ibragim.dev · Pi ibragim-dev | Personal coding evals core10 v1.3 · 10 tasks × 3 runs, Harbor 0.21.0, Pi 0.85.1, effort per row, pass@1 and $/task; board of 28 Sep ✓ | sweep | Sonnet 5.5·high · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5·high · Sonnet 5·high |
| Internal coding, 370 repo tasks (Anthropic docs) anthropic-docs | plain API agent, customer-billed per attempt ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high |
| jev-effort github-jev-effort | Claude Code 2.1.280 · Opus 5.5 baseline arm, high vs max fixed effort, 6 coding tasks x2 runs ✓ | sweep | Opus 5.5·high · Opus 5.5·max |
| kayben plan review github-kaybenleroll | claude -p no tools, CC 2.1.282 · 13 seeded defects, Opus 5.5 & Sonnet 5 × medium/high/xhigh, n=10, effort verified in transcripts ✓ | sweep | Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| kayben plan review f51 github-kaybenleroll | same doc, cap10 prompt · Fable 5.1 low→max, n=5 ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·max · Sonnet 5·max |
| Lisbon bike-rental website, Sonnet 5.5 effort ladder (Mahe Delich video) youtube-mahe-delich | Claude Code /cost; same prompt; empty folder; one try per level; 2026-09-30; read 2026-10-18 ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| llm-regression 2026-10-02 Opus 5.5 high vs xhigh (pass rate %) nikoamoretti-llm-regression | nikoamoretti/llm-regression data.json: 2026-10-02 first live runs, product suite, Claude Code, high (32 attempts) vs xhigh (28 attempts, 3 infra_fail); strict hidden-test pass rate over graded attempts; read 2026-10-20 ✓ | sweep | Opus 5.5·xhigh · Opus 5.5·high |
| Low or Bust github-low-or-bust | Claude Code -p · 9 one-shot coding/reasoning/writing tasks, Fable 5 & Opus 4.8 × low→xhigh, 1 trial, CLI cost incl. harness cache writes ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh |
| Low or Bust 2 monolith github-low-or-bust | Claude Code agentic · 2 ten-step packages × 3 trials, Fable 5 low/medium/high, hidden tests saturated → tokens only ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high |
| Low or Bust 2 workers github-low-or-bust | Claude Code · fresh low/high worker per step, Fable 5, saturated → tokens only ✓ | sweep | Fable 5·low · Fable 5·high |
| Low or Bust hard github-low-or-bust | Claude Code -p · 8 trap tasks, Fable 5 × low→xhigh, 1 trial ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh |
| Low-poly forest in Blender, Haiku 5.5 effort ladder (DopeShit video) youtube-dopeshit | Claude Code, Haiku 5.5 effort set per run; author grades 4 criteria (10 each) = /40; 2026-10-10; read 2026-10-18 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| m_izumo Opus 5.5 review sub-agents on a personal project (15 sections) qiita-m_izumo | Claude Code sub-agents (20 x high; 22 x xhigh); defects verified by main session; coverage = share of verified real defects found by one reviewer; read 2026-10-15 ✓ | sweep | Opus 5.5·high · Opus 5.5·xhigh |
| MARROWLIGHT one-page site, Opus 5.5 effort ladder (The Useful Stack video) youtube-usefulstack | Claude Code 2.1.285 --effort; auto-scored /100; redone runs only; one run per setting; 2026-09-30; read 2026-10-18 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh |
| Melvynx bench github-melvynx | Claude Code · 7 valid tasks, Opus 5 high vs max, manual rubric ✓ | sweep | Opus 5·high · Opus 5·max |
| MERC core-10 github-merc-bench | Claude Code -p, no tools · 10 graded tasks, 6 models × low→max, n=2; usage × list (5m cache-write); scores 94–98/98 → cost only ✓ | sweep | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·xhigh · Haiku 4.5·req-high · Haiku 4.5·req-low · Haiku 4.5·req-medium · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| nnakapa #35 Copilot zenn-nnakapa | GitHub Copilot CLI 1.0.79 · same 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, billed cost ✓ | sweep | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh |
| nnakapa #36 Copilot zenn-nnakapa | GitHub Copilot CLI 1.0.80 · same 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, billed cost ✓ | sweep | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh |
| nnakapa QCD #20 zenn-nnakapa | Claude Code 2.1.157 · 3 DB tasks × 10 runs, Opus 4.7 & Opus 4.8 × low→xhigh ✓ | sweep | Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh |
| nnakapa QCD #24 zenn-nnakapa | Claude Code 2.1.170 · 3 DB tasks × 10 runs, Fable 5 & Opus 4.8 × low→xhigh ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh |
| nnakapa QCD #29 zenn-nnakapa | Claude Code 2.1.220 · 3 DB tasks × 10 runs, Opus 5, Opus 4.8 & Fable 5 × low→xhigh, CLI-reconciled cost, hidden-test gate ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh |
| nnakapa QCD #34 zenn-nnakapa | Claude Code 2.1.226 · 3 DB tasks × 10 runs, Sonnet 5 & Opus 5 × low→xhigh, CLI-reconciled cost ($2/$10 Sonnet), hidden-test gate ✓ | sweep | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh |
| nnakapa QCD #35 CC zenn-nnakapa | Claude Code 2.1.229 (1M ctx) · 3 DB tasks × 10 runs, Sonnet 5 × low→xhigh, CLI-reconciled cost ($2/$10) ✓ | sweep | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh |
| nnakapa QCD #37 zenn-nnakapa | Claude Code 2.1.258 · 3 DB tasks × 10 runs, Opus 5 & Fable 5.1 × low→xhigh, CLI-reconciled cost, hidden-test gate ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh |
| nurb-benchmarks github-nurb | Claude Code · 6 CAD tasks, 2 models × full ladder; quality saturates → cost only ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Haiku 4.5·solo · Sonnet 5·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh |
| ObviousBench 0.2 github-obviousbench | inspect-ai · 144 items ×3 epochs, 3 models × full ladder, measured tokens ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·nothink · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Haiku 4.5·solo · Haiku 4.5·think-8k · Haiku 4.5·think-1k · Haiku 4.5·think-2k · Haiku 4.5·think-16k · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Opus 4.8·nothink · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 4.6·nothink · Sonnet 4.6·default · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 5.5·nothink · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| Occam bench, no plugin github-occam | Claude Code headless · 18 procedural tasks, hidden verifier · Sonnet 5.5 & Opus 5.5 at max/medium + Haiku 4.5 ✓ | sweep | Sonnet 5.5·max · Sonnet 5.5·medium · Opus 5.5·max · Opus 5.5·medium · Haiku 4.5·solo |
| Occam bench, Occam plugin github-occam | same 18 tasks, Occam rules loaded · Sonnet 5.5 & Opus 5.5 at max/medium + Haiku 4.5 ✓ | sweep | Sonnet 5.5·max · Sonnet 5.5·medium · Opus 5.5·max · Opus 5.5·medium · Haiku 4.5·solo |
| Occam bench, Ponytail 4.10 github-occam | same 18 tasks, Ponytail plugin · Sonnet 5.5 & Opus 5.5 at max/medium + Haiku 4.5 ✓ | sweep | Sonnet 5.5·max · Sonnet 5.5·medium · Opus 5.5·max · Opus 5.5·medium · Haiku 4.5·solo |
| Opus 5.5 AoE2 build github-barty-opus55 | Claude Code 2.1.281 · one long game-build prompt, Opus 5.5 × low→max, 1 run, CLI cost, no score → cost only ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·high · Sonnet 5.5·max |
| OSWorld anthropic-chart | Anthropic/AA · sweep low→max ✓ | sweep | Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max |
| OSWorld 2.0 anthropic-syscard | Opus 5 card p174 · price vs perf, 4 models; effort inferred from cost order ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| OSWorld 2.0 (F5.1) anthropic-syscard | Fable 5.1 card p190 · partial-credit price/perf, 3 models; effort inferred from cost order ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max |
| OSWorld 2.0 board (Opus 5) osworld-v2-board | osworld-v2.xlang.ai official JSON · v2.1 full 108 tasks, batched, 500 steps, Opus 5 low→max, output tokens/task, no cost ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| OSWorld 2.0 board v08.08 (Opus 5) osworld-v2-board | osworld-v2.xlang.ai official JSON · release v2026.08.08, 7-run avg, Opus 5 low→max, score only ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| OSWorld 2.1 (O5.5) anthropic-syscard | Opus 5.5 card p207 + Sonnet 5.5 card p131 · v2.1 files of 10 Sep, partial credit, 3 models × 5 efforts (effort from cost order) + Sonnets at max ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max · Sonnet 5·max |
| OSWorld 2.1 strict anthropic-syscard | Sonnet 5.5 card p132 · strict pass, Opus 5.5 × 5 (effort from cost order) + Sonnets at max; other Sonnet points held ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·max · Sonnet 5·max |
| OSWorld eff. (4.7) anthropic-syscard | Opus 4.7 card p209 · pass@1 vs output tokens, 3 models × low→max ✓ | sweep | Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| OSWorld sweep anthropic-syscard | Opus 4.8 card p222 · sweep low→max, output tokens ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| oubakiou godot-llm-gamebench: Opus 5.5 child, 5 efforts, parent fixed opus-5.5@medium zenn-oubakiou | Claude Code CLI child at claude-opus-5-5, parent claude-opus-5-5@medium fixed; Godot 4.4.1 'Ember Warden' mini-game vs private spec, scored by 57 hidden tests; n=3 reps per effort; zenn.dev/oubakiou/articles/fa3b3914e081c8; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| pelican-bicycle-sigma.vercel.app — Haiku 5.5 effort sweep (SVG pelican-on-bicycle) pelican-bicycle-sigma | independent site, one Claude model (Haiku 5.5) at five efforts = five couples per RULES.md; cost is the page's own $ figure computed from a usage panel (not a billed invoice) -> list-price-cost; page's own caveat: "Usage numbers are transcribed from each run's usage panel" (manual transcription, not raw API JSON); read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| PetriBench arxiv-2609.19883 | arXiv 2609.19883 · 4,800 Petri-net questions, raw API adaptive thinking + output_config.effort, Opus 5 M/H/XH + Sonnet 5 M/H, measured tokens × list ✓ | sweep | Opus 5·medium · Opus 5·high · Opus 5·xhigh · Sonnet 5·medium · Sonnet 5·high · Haiku 4.5·solo |
| PhysicianBench anthropic-syscard | Sonnet 5.5 card p138 · Sonnet 5.5 low→max labelled + 4 models at max, score-only (time axis) ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5·max |
| Playcode MacBook SVG playcode | Playcode · one-shot MacBook SVG, high/xhigh/max per model, provider-billed cost (quality ordinal, not scored) ✓ | sweep | Opus 5.5·high · Opus 5.5·xhigh · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 4.6·high · Sonnet 4.6·max · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·max |
| PostTrainB. arxiv-2603.08640 | papier · medium/high ✓ | sweep | Sonnet 4.6·default · Haiku 4.5·solo |
| Priced Guidance dev set (40 papers): Fable 5.1 high vs max arxiv-2610.04976 | dev set 40 papers, effort ablation; Figure 7 text of arXiv 2610.04976; read 2026-10-15 ✓ | sweep | Fable 5.1·high · Fable 5.1·max |
| Priced Guidance dev set (40 papers): Opus 5 high vs max arxiv-2610.04976 | dev set 40 papers, effort ablation; Figure 7 text of arXiv 2610.04976; read 2026-10-15 ✓ | sweep | Opus 5·high · Opus 5·max |
| qiita qiita | raw API · low/max ✓ | sweep | Opus 4.7·low · Opus 4.7·max |
| Qiita reasoning qiita-takuya | Qiita (Takuya) · 19 reasoning questions × 3 runs, Claude Code without tools, measured tokens × list price ✓ | sweep | Opus 5.5·high · Opus 5.5·medium · Opus 5·high · Fable 5.1·high |
| Qiita yunany90 Opus 5.5 effort sweep, brick-breaker single index.html qiita-yunany90 | Claude Code (Windows; Max 20x); same request 5 times, only effort changed; cost = ccusage API-equivalent; read 2026-10-15 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max |
| QuoteBench arxiv-2608.13547 | arXiv 2608.13547 · 56 frozen tasks, provider-reported output tokens, 3 models × ladder ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·xhigh · Haiku 4.5·req-low · Haiku 4.5·req-medium · Haiku 4.5·req-high · Haiku 4.5·req-xhigh |
| Ramen Bench github-ramenbench | not-stbenjam/ramen-bench · one-shot creative-coding HTML, Sonnet 5.5, Opus 5.5, Opus 5, Fable 5.1 × low→max, Claude Code run.json, cost only ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| Reddit 1x0iu5x (dergachoff): Haiku 5.5 research sub-agent at three efforts vs DeepSeek reddit-dergachoff | 4 real research briefs, Haiku 5.5 low/medium/high vs DeepSeek V4.1 Flash (low), Opus blind A/B judge both orders; score = briefs won vs DeepSeek of 4 (ties counted as not won); cost per run as self-reported (scope unstated); read 2026-10-15 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high |
| Reddit: pelican-on-a-bicycle SVG, Opus 5.5 effort sweep reddit-1wrdd57 | one-line prompt 'Write HTML code that displays a 2D animation of a pelican riding a bicycle using SVG graphics...', fresh session each, author's own agent setup (pi with custom skills); n=1 per level, times/costs from session logs, no numeric quality score (qualitative only); read 2026-10-05 ✓ | sweep | Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh |
| renchris review f51 github-renchris-infra | claude -p no tools, CC 2.1.260 (64K cap) · same 9-brief corpus, Fable 5.1 low→xhigh (max incomplete), 3 blind judges ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max |
| renchris review o55 github-renchris-infra | claude -p no tools, CC 2.1.280 · 9 code briefs / 36 anchored defects, Opus 5.5 low→max + Opus 5 @high, 3 blind judges; CLI list cost ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·high |
| renkon40 · 12 editable PowerPoint slides in a given style note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, 12 editable PowerPoint slides in a given style; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·xhigh |
| renkon40 · 3-minute talk slides note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, 3-minute talk slides; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| renkon40 · dashboard from an AI-secretary log note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, dashboard from an AI-secretary log; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Sonnet 5.5·high · Sonnet 5.5·max · Opus 5.5·high |
| renkon40 · Note article from an experiment log note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, Note article from an experiment log; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Sonnet 5.5·high · Sonnet 5.5·max · Opus 5.5·high |
| renkon40 · short video note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, short video (60 s from a 25-min YouTube talk); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Sonnet 5.5·max · Opus 5.5·max |
| renkon40 · YouTube-channel homepage note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, YouTube-channel homepage (no skill); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | sweep | Sonnet 5.5·high · Sonnet 5.5·max · Opus 5.5·high |
| retort exp-49 github-retort | Claude Code · one task, 2 models × low→max, n=3; repo states its own quality saturation → cost only ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Fable 5·default · Opus 4.7·default · Opus 4.8·default · Opus 5·default |
| retort exp-55 brazil go github-retort | Claude Code · brazil-soccer-mcp Go, Opus 5 low/high/xhigh/max (medium has no cost), n=1 ✓ | sweep | Opus 5·low · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low |
| retort exp-55 brazil py github-retort | Claude Code · brazil-soccer-mcp Python, Opus 5 low→max, n=1 ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low |
| retort exp-55 go github-retort | Claude Code · rest-api-crud Go, Opus 5 low→max, n=2 (max n=1); coverage 1.0 → cost only ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| retort exp-55 py github-retort | Claude Code · rest-api-crud Python, Opus 5 low→max, n=2 (max n=2); coverage 1.0 → cost only ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| retort exp-65 github-retort | Claude Code · rest-api-crud, Fable 5.1 low vs default, 4 languages ×3, prompt balanced ✓ | sweep | Fable 5.1·low · Fable 5.1·default |
| retort exp-74 go github-retort | Claude Code · rest-api-crud in Go, Opus 5.5 × low→max, n=3; requirement coverage 1.0 everywhere → cost only ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5.5·default |
| retort exp-74 py github-retort | Claude Code · rest-api-crud in Python, Opus 5.5 × low→max, n=3; requirement coverage 1.0 everywhere → cost only ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5.5·default |
| retort exp-77 go github-retort | Claude Code 2.1.284, host-config inherited · rest-api-crud in Go, Sonnet 5.5 × low→max (+default), n=3 (max n=2, one replicate crashed at the 90-min wall); requirement coverage 1.0 everywhere → cost only ✓; read 2026-10-01 ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5.5·default |
| retort exp-77 py github-retort | Claude Code 2.1.284, host-config inherited · rest-api-crud in Python, Sonnet 5.5 × low→max (+default), n=3 (go max n=2, one replicate crashed); requirement coverage 1.0 everywhere → cost only ✓; read 2026-10-01 ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5.5·default |
| retort exp-78 go (isolated) github-retort | Claude Code 2.1.284, agent isolated (no inherited host MCP/settings) · rest-api-crud in Go, Sonnet 5.5 × low/medium/high (+default), n=3; requirement coverage 1.0 everywhere → cost only; separate config from exp-74/77, not comparable without accounting for the ~43% host-inheritance token inflation ✓; read 2026-10-01 ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·default |
| retort exp-78 py (isolated) github-retort | Claude Code 2.1.284, agent isolated (no inherited host MCP/settings) · rest-api-crud in Python, Sonnet 5.5 × low/medium/high (+default), n=3; requirement coverage 1.0 everywhere → cost only; separate config from exp-74/77, not comparable without accounting for the ~43% host-inheritance token inflation ✓; read 2026-10-01 ✓ | sweep | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·default |
| retort exp-79 go isolated github-retort | Claude Code (isolated, MCP disabled) · rest-api-crud in go, Opus 5.5 × low/medium/high/default, n=3; requirement coverage 1.0 everywhere → cost only; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·default · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 5.5·default |
| retort exp-79 python isolated github-retort | Claude Code (isolated, MCP disabled) · rest-api-crud in python, Opus 5.5 × low/medium/high/default, n=3; requirement coverage 1.0 everywhere → cost only; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·default · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 5.5·default |
| retort exp-83 ATDD skill go (atdd-skill) github-retort | Claude Code + msec ATDD-review skill · brazil in go, prompt=atdd-skill, Opus 5.5 × low/high, n=2, score=atdd_review%; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·high |
| retort exp-83 ATDD skill go (neutral) github-retort | Claude Code + msec ATDD-review skill · brazil in go, prompt=neutral, Opus 5.5 × low/high, n=2, score=atdd_review%; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·high |
| retort exp-83 ATDD skill python (atdd-skill) github-retort | Claude Code + msec ATDD-review skill · brazil in python, prompt=atdd-skill, Opus 5.5 × low/high, n=2, score=atdd_review%; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·high |
| retort exp-83 ATDD skill python (neutral) github-retort | Claude Code + msec ATDD-review skill · brazil in python, prompt=neutral, Opus 5.5 × low/high, n=2, score=atdd_review%; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·high |
| retort exp-83 ATDD skill typescript (atdd-skill) github-retort | Claude Code + msec ATDD-review skill · brazil in typescript, prompt=atdd-skill, Opus 5.5 × low/high, n=2, score=atdd_review%; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·high |
| retort exp-83 ATDD skill typescript (neutral) github-retort | Claude Code + msec ATDD-review skill · brazil in typescript, prompt=neutral, Opus 5.5 × low/high, n=2, score=atdd_review%; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·high |
| Retro board +Playwright arxiv-2607.02436 | Claude Code 2.1.132 · retro-board build with Playwright MCP; Opus 4.7 high/xhigh, Opus 4.6, Sonnet 4.6; /cost medians ✓ | sweep | Opus 4.7·high · Opus 4.7·xhigh · Sonnet 4.6·high |
| Retro board abridged prompt arxiv-2607.02436 | Claude Code 2.1.132 · abridged design prompt; Opus 4.7 high/xhigh, Opus 4.6 ✓ | sweep | Opus 4.7·high · Opus 4.7·xhigh |
| Retro board design prompt arxiv-2607.02436 | Claude Code 2.1.132 · full Antigravity design prompt; Opus 4.7 high/xhigh, Opus 4.6, Sonnet 4.6 high/max ✓ | sweep | Opus 4.7·high · Opus 4.7·xhigh · Sonnet 4.6·high · Sonnet 4.6·max |
| RHI (Sonnet 4.6) arxiv-2607.15524 | arXiv 2607.15524 Fig. 5b · 30 ML tasks, cost normalized to sonnet-4.6-high ✓ | sweep | Sonnet 4.6·high · Sonnet 4.6·max |
| Self-conditioning test: 100 planted turns, 100% wrong totals (muninn.austegard) muninn-austegard | ICLR-2026 self-conditioning re-run, 50-word dictionary, claude -p --resume, 20 sessions per cell, Haiku 5.5; read 2026-10-18 ✓ | sweep | Haiku 5.5·high · Haiku 5.5·low |
| serotoninmax Haiku 5.5 effort sweep, tagging 146 Japanese book blurbs x 91 questions zenn-serotoninmax | API claude-haiku-5-5 max_tokens 32000, output_config.effort; cached 6.4K-token system prompt; per-item cost = measured tokens x list price; Jev reference not Claude; read 2026-10-15 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max |
| Simon Willison: pelican-riding-a-bicycle SVG test, own measured cost simonwillison | llm CLI (llm-anthropic plugin), own measured spend (not list-price computation), n=1 per point, quality is ordinal/qualitative (no numeric score) -> cost-only; all 5 efforts same session per h4's gist; read 2026-10-08 ✓ | sweep | Haiku 5.5·low · Haiku 5.5·max |
| Sonar Java sonarsource | Sonar leaderboard JSON · 4,444 Java tasks, single-shot, measured tokens × list price; Opus 5.5 medium/high ✓ | sweep | Opus 4.8·high · Opus 5·high · Opus 5.5·medium · Opus 5.5·high · Opus 4.7·high · Sonnet 5·default · Sonnet 4.6·high · Haiku 4.5·solo · Sonnet 5.5·high |
| SWE Refactor arxiv-2608.23564 | arXiv 2608.23564 · 20 whole-repo migrations, Claude Code, measured API spend, 2 models × ladder ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| SWE-bench Pro subset (Anthropic docs) anthropic-docs | Anthropic-internal 478-problem subset, customer-billed cost ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Fable 5.1·low · Fable 5.1·default · Sonnet 5·default |
| SWE-Effort 25 arxiv-2608.01347 | arXiv 2608.01347 App. N · 25 hard SWE-bench Verified, Claude Code 2.1.220, Opus 5 & Sonnet 5 × low/high/xhigh, billed cost ✓ | sweep | Opus 5·low · Opus 5·high · Opus 5·xhigh · Sonnet 5·low · Sonnet 5·high · Sonnet 5·xhigh |
| SWE-Pro (Fable) anthropic-syscard | Fable 5 card p255 · Fable=Mythos 5, sweep low→xhigh, $ cost ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh |
| SWE-Pro sweep anthropic-syscard | Opus 4.8 card p196 · sweep low→max, output tokens ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max |
| Synthorai (ja mirror): 13 single-shot answer tasks, Sonnet 5.5 / Sonnet 5 / Opus 5.5 by effort synthorai | native Messages API via gateway, 13 tasks x3 reps x6 efforts, cost computed from list price; qiita.com/Synthorai/items/98e776a66f483f3ceeaf (mirror of synthorai.io/ja/blog/claude-sonnet-5-5-cost/); read 2026-10-05 ✓ | sweep | Sonnet 5.5·default · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·default · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·default · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max |
| Synthorai multi-hop tool loop (4 Qs ×12) synthorai | Opus 5.5 default/low/max vs Opus 5 default ✓ | sweep | Opus 5.5·default · Opus 5.5·low · Opus 5.5·max · Opus 5·default |
| Synthorai single-shot arithmetic (13 tasks ×3) synthorai | Opus 5.5 and Opus 5 full ladders + default, API list-price cost; accuracy ≥97 % everywhere ✓ | sweep | Opus 5.5·default · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Opus 5·default · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| Tafla benchmark 2026-10-08: everyday calculations, default vs low effort tafla.is | 12 questions x 5 runs x (question + follow-up), live agent with GRID calculation engine; score = correct boards of 120 turns; cost per run (60 runs per lane); read 2026-10-15 ✓ | sweep | Haiku 5.5·default · Sonnet 5.5·default · Haiku 5.5·low · Sonnet 5.5·low |
| tank_ai wareki converter note-tank_ai | note.com 29 Sep · Claude Code subagents at fixed model × effort, n=1, 38 auto-graded tests, score only ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high |
| TB 3.0 (claude.dev chart) claude-dev-blog | claude.dev 'Spending your effort' SVG · 70 tasks, Opus 5.5 / Fable 5.1 / Opus 5 / Fable 5 × low→max, median tokens, no USD; Opus 5.5 run 3 wk later, 128k cap ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| TB 3.0 (claude.dev payload) claude-dev-blog | claude.dev 'Spending your effort' embedded data · 74 tasks×5, Opus 5 / Fable 5 / Fable 5.1 labelled rungs, median tokens, no USD ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·xhigh · Fable 5.1·max |
| TB 4.0 (Anthropic launch charts) anthropic-chart | Anthropic launch page chart CSVs (Opus 5.5 and Sonnet 5.5 pages) · 5 models × low→max; production safeguards on (fallbacks); Sonnet 5 anomalously low, flagged ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| Towards AI internal writing-Elo board, Sep 25, 2026 reddit-1wr03jc-img2 | 10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; Opus 5.5 six-effort scatter + Fable 5.1 reference point, footer 'Sep 25, 2026'; values are printed point labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·default · Opus 5.5·max · Fable 5.1·max |
| uhyo React proficiency zenn-uhyo | zenn.dev uhyo, round 17 (25 Sep) · 13 React specs × 3 samples, judge pinned to Sonnet 4.6, effort pinned; score only ✓ | sweep | Opus 5.5·max · Opus 5.5·high · Opus 5·max · Opus 5·high · Fable 5·max · Fable 5.1·high · Opus 4.8·max · Fable 5·high · Sonnet 5.5·high · Sonnet 5.5·max · Haiku 5.5·high · Haiku 5.5·max |
| Vals blog: Vibe Code Bench, Opus 5.5 single agent vals.ai | Claude Code CLI headless, Opus 5.5, single agent, 50 apps, 1 run per setup; score = share of UI tests passed; cost = list-price totals reported by Claude Code per app; read 2026-10-15 ✓ | sweep | Opus 5.5·medium · Opus 5.5·max |
| Vals blog: Vibe Code Bench, Opus 5.5 team (lead + up to 5 subagents) vals.ai | Claude Code CLI headless, Opus 5.5, team, 50 apps, 1 run per setup; score = share of UI tests passed; cost = list-price totals reported by Claude Code per app; read 2026-10-15 ✓ | sweep | Opus 5.5·medium · Opus 5.5·max |
| Vals VibeCode vals.ai | Vals AI · Opus 5 sweep low→max (scores), $ cost at leaderboard setting; board entries — snapshot 27 Sep, re-priced ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·max · Opus 5.5·max · Fable 5.1·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 4.6·max · Haiku 4.5·solo · Sonnet 5.5·max · Haiku 5.5·max |
| Vals Web Search (finance) vals.ai | vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓ | sweep | Fable 5·max |
| Vals Web Search (legal) vals.ai | vals.ai payload · Fable 5, Native vs Exa search backend, compute_effort=max ✓ | sweep | Fable 5·max |
| Vectorise doc-search effort test (Vidali gist) vidali-gist | 33 multilingual MCP questions, output tokens only, 24 Sep 2026 ✓ | sweep | Opus 5·default · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Sonnet 5·default · Sonnet 5·high · Sonnet 5·xhigh · Haiku 4.5·solo |
| vibe-openscad github-vibeopenscad | output_config.effort explicit · 3 models × ladder; quality saturates → cost only ✓ | sweep | Fable 5.1·low · Fable 5.1·default · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·low · Fable 5·default · Fable 5·xhigh · Fable 5·max · Opus 5·low · Opus 5·default · Opus 5·xhigh · Opus 5·max · Opus 4.7·low · Opus 4.8·low · Sonnet 4.6·low · Sonnet 5·low · Opus 4.7·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·nothink · Opus 4.8·nothink · Opus 5·nothink · Sonnet 4.6·nothink · Sonnet 5·nothink · Opus 4.7·xhigh · Opus 4.8·xhigh · Sonnet 5·xhigh · Haiku 4.5·solo · Opus 4.7·default · Opus 4.8·default · Sonnet 4.6·default · Sonnet 5·default |
| vlm-exam github-vlmexam | roboflow · identical 133-image set, 3 models × low/high, measured tokens ✓ | sweep | Fable 5·low · Fable 5·high · Fable 5.1·low · Fable 5.1·high · Opus 5·low · Opus 5·high · Opus 5.5·low · Opus 5.5·high · Sonnet 5·low · Sonnet 5·high · Opus 4.8·low · Opus 4.8·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high |
| VulcanBench CII v4 github-vulcanbench | Claude Code 2.1.280 · 23 legacy binary-parity tasks, Opus 5.5 × low→max, CLI-reported cost; refusal fallback on (0→48 % of replies by Opus 4.8) ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| VulcanBench CII-v4 code_quality (LLM judge) for the held Opus 5.5 ladder github-vulcanbench | github.com/morganlinton/VulcanBench, src/vulcan_gpt61sol-opus55-astra-quality-efforts.csv (commit 3f1bfa61, 2 Oct 2026), same combined-v3.15 Opus 5.5 ladder as group vulcanbench-ciiv4; read 2026-10-05 ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| VulcanBench F-v4 f51 github-vulcanbench | Claude Code 2.1.259–261 · same 23 tasks, Fable 5.1 × low→max, v3.4 judge, per-model list ledger; fallback 7–40 % ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max |
| VulcanBench F-v4 o5 github-vulcanbench | Claude Code 2.1.257 · same 23 tasks, Opus 5 × low/med/high/max, CLI list cost, functional pass only ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·max |
| VulcanBench v3 Opus 5 github-vulcanbench | VulcanBench harness · 23 tasks ×1, Opus 5 low/medium/high, report No.10 (26 Jul) ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high |
| WANDR anthropic-syscard | Opus 5.5 card p188 · Perplexity wide-search, offline index, 980k budget, 3 models × low→max, scores printed ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max |
| Weather-card CC github-weather-card | Claude Code · one-shot HTML weather card, 2×2 slots, Fable 5.1/5, Opus 5.5/5/4.8, Sonnet 5, Haiku × low→max, output tokens only ✓ | sweep | Fable 5·default · Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 4.5·req-low · Haiku 4.5·req-max · Haiku 4.5·req-xhigh · Haiku 4.5·req-high · Haiku 4.5·req-medium · Opus 4.8·high · Opus 4.8·low · Opus 4.8·max · Opus 4.8·medium · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh |
| Weather-card Cursor github-weather-card | Cursor CLI · same task, Claude models × effort (non-thinking ids), tokens only ✓ | sweep | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Opus 4.7·high · Opus 4.7·low · Opus 4.7·max · Opus 4.7·medium · Opus 4.7·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·max · Opus 4.8·medium · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·medium · Sonnet 4.6·medium · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh |
| Weather-card Cursor thinking github-weather-card | Cursor CLI · -thinking model ids × effort, tokens only ✓ | sweep | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Opus 4.7·high · Opus 4.7·low · Opus 4.7·max · Opus 4.7·medium · Opus 4.7·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·max · Opus 4.8·medium · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 4.6·medium · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh |
| Weather-card Cursor-xl github-weather-card | Cursor CLI xl · same task, tokens only ✓ | sweep | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Opus 4.7·high · Opus 4.7·max · Opus 4.7·medium · Opus 4.8·high · Opus 4.8·low · Opus 4.8·max · Opus 4.8·medium · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·medium · Sonnet 4.6·medium · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh |
| Weather-card Cursor-xl thinking github-weather-card | Cursor CLI xl · -thinking ids × effort, tokens only ✓ | sweep | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Opus 4.7·high · Opus 4.7·low · Opus 4.7·xhigh · Opus 4.8·high · Opus 4.8·low · Opus 4.8·max · Opus 4.8·medium · Opus 5·high · Opus 5·low · Opus 5·medium · Opus 5·xhigh · Sonnet 4.6·medium · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh |
| WeirdML epoch-archive | Epoch archive · 3 models × high/max, cost per run ✓ | sweep | Fable 5.1·high · Fable 5.1·max · Fable 5·high · Fable 5·max · Opus 5·high · Opus 5·max |
| WideSearch (Anthropic docs) anthropic-docs | Fable 5 low/medium/default, 3 runs ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·default |
| Willison SVG simonwillison | llm CLI · sweep low→max, trivial task (SVG) ✓ | sweep | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| Willison SVG 5.1 simonwillison | llm CLI · one fixed SVG prompt, Fable 5.1 low→max, measured tokens; trivial task ✓ | sweep | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max |
| Willison SVG 5.5 simonwillison | llm CLI · one fixed SVG prompt, Opus 5.5 and Sonnet 5.5 low→xhigh, measured tokens; max hit the 128k cap on both (omitted); trivial task ✓ | sweep | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh |
| wook3024 CC bench github-wook3024 | Claude Code · 3 tasks × 2 runs, Sonnet 4.6 & Opus 4.6 × medium/high (+ fixed-thinking Opus), CLI cost ✓ | sweep | Sonnet 4.6·medium · Sonnet 4.6·high |
| Zapier AutomationB. zapier | AutomationBench 1.0.6 · 657 held-out tasks, strict pass/fail, measured tokens × list ✓ | sweep | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·xhigh · Fable 5.1·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Haiku 4.5·solo · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| Zenn nttdata_tech compaction nttdata_tech | bedrock-runtime · Opus 5.5 auto-compaction summary tokens by effort, n=1, low/default/max(8192) ✓; max@2048 dropped, censored ✓ | sweep | Opus 5.5·low · Opus 5.5·default · Opus 5.5·max |
| zenn QCD #27 zenn-nnakapa | Claude Code 2.1.197 · 3 DB tasks × 10 runs, Opus 4.8 & Sonnet 5 × low→xhigh, CLI-reconciled cost ✓ | sweep | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh |
| deepseek-v41-techreport: deepswe deepseek-v41-techreport | DeepSWE v1.1 (Resolved), mini-SWE harness; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness mini-SWE; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | Opus 5·max |
| deepseek-v41-techreport: gpqa deepseek-v41-techreport | GPQA Diamond (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | Opus 5·max |
| deepseek-v41-techreport: tb21 deepseek-v41-techreport | Terminal-Bench 2.1 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; + effort sweep 25..100 of deepseek-v4.1-flash from Fig 9/11 (same runs; effort-100 point = Table 3 cell, kept once), tokens_out per response/trajectory; read 2026-10-19 ✓ | sweep+xmodel | Opus 5·max |
| BioXArena (76 biomedical ML tasks), Claude Code, Opus 5.5 / 5 / 4.6 arxiv-2605.15766 | arXiv 2605.15766 v2 (15 Oct 2026); Claude Code harness; 76 tasks; 2-h budget; penalized all-task normalized score x100; single run; read 2026-10-18 ✓ | xgen | Opus 5.5·default · Opus 5·default |
| Box · consumer products box.com | Box AI blog · Opus 5 vs 5.5, one representative task, score-only ✓ | xgen | Opus 5·default · Opus 5.5·default |
| Box · financial services box.com | Box AI blog · Opus 5 vs 5.5, one representative task, score-only ✓ | xgen | Opus 5·default · Opus 5.5·default |
| Box · technology box.com | Box AI blog · Opus 5 vs 5.5, one representative task, score-only ✓ | xgen | Opus 5·default · Opus 5.5·default |
| CliffCompaction TB2.0 arxiv-2609.26779 | Terminus-2 · k=1 proprietary baselines, Opus 4.6 & 4.7 ✓ | xgen | Opus 4.7·default |
| Deictic ambiguity ablation experiment balanced accuracy arxiv-2609.12162 | Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); ablation experiment; read 2026-10-19 ✓ | xgen | Haiku 4.5·solo · Haiku 4.5·nothink |
| Deictic ambiguity primary experiment balanced accuracy arxiv-2609.12162 | Draft-Verify-Revise deictic-ambiguity task: meta-evaluator must accept/reject grader claims in 3 conditions, 200 paired trials per configuration (20 trials per cell), collection Feb 2026; reasoning-effort setting translated to each provider's native parameter (OpenAI reasoning_effort, Anthropic effort field for Opus 4.6 / extended-thinking budget for Haiku 4.5, Google thinking_level); primary experiment; read 2026-10-19 ✓ | xgen | Haiku 4.5·solo · Haiku 4.5·nothink |
| drona23 github-drona23 | Claude Code CLI · thinking NS, identical ✓ | xgen | Haiku 4.5·solo · Sonnet 4.6·default |
| Epoch Capabilities Index (composite), every Claude model on the board epoch.ai | epoch.ai/eci (epoch.ai/data/benchmark_data.zip -> epoch_capabilities_index/eci_scores.csv); aggregate meta-index over many sub-benchmarks (mixed/unstated per-model efforts feeding it); one row per model, 95% CI as published; COMPOSITE — see composites.json note below; read 2026-10-05 ✓ | xgen | Opus 5.5·default · Fable 5.1·default · Sonnet 5.5·default · Opus 5·default · Fable 5·default · Haiku 5.5·default · Opus 4.8·default · Sonnet 5·default · Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| FeatherBench ed-is-ai/featherbench | OpenRouter(Anthropic) · 28-task multi-domain suite (coding/data/realworld/security/tool-use), effort=high (Haiku=solo), mean cost/tokens per task-run, consolidated results/summary.json; only Opus 5.5 rung saturated ✓ | xgen | Fable 5·high · Fable 5.1·high · Sonnet 4.6·high · Sonnet 5·high · Haiku 4.5·solo · Opus 5·high · Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high |
| FuzzingBrain-Bench arxiv-2608.25158 | arXiv 2608.25158 · 77 fuzzing challenges, raw API no thinking param, 3 models, tokens × list (cache-aware) ✓ | xgen | Opus 4.8·default · Sonnet 4.6·default · Haiku 4.5·solo |
| George Liu · Opus 4.8 vs Fable 5 ai.georgeliu | Claude Code 10-prompt suite · medium/high baselines ✓ | xgen | Opus 4.8·medium · Opus 4.8·high · Fable 5·medium · Fable 5·high |
| George Liu · Sonnet 5 vs 4.6 ai.georgeliu | Claude Code 10-prompt suite · medium/high baselines ✓ | xgen | Sonnet 5·medium · Sonnet 5·high · Sonnet 4.6·medium · Sonnet 4.6·high |
| GOAD direct prompting, no harness (Claude only): chain completion arxiv-2610.04243 | Opus 4.8/4.7/4.6 prompted directly with Kali shell access and no structured agents (Table V direct columns); median of 3 runs; only Claude models (paper omits others); cost only for Opus 4.8 (Table IV); read 2026-10-19 ✓ | xgen | Opus 4.8·default · Opus 4.7·default |
| Google model cards: BioMysteryBench human difficult google-gemini-3.7-flash-card | BioMysteryBench human difficult; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 5·default · Opus 5·default |
| Google model cards: BioMysteryBench human solvable google-gemini-3.7-flash-card | BioMysteryBench human solvable; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Opus 5·default |
| Google model cards: CharXiv Reasoning no tools google-gemini-3.5-flash-card | CharXiv Reasoning no tools; Google model-card table(s) 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·default · Opus 4.7·default · Sonnet 5·default |
| Google model cards: CharXiv Reasoning with tools google-gemini-3.6-flash-card | CharXiv Reasoning with tools (Gemini: search + code execution); Google model-card table(s) 21 Jul 2026 (google-gemini-3.6-flash-card); 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 5·default |
| Google model cards: FACTS Benchmark Suite google-gemini-3-flash-card | FACTS Benchmark Suite; Google model-card table(s) Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Haiku 4.5·solo |
| Google model cards: GPQA Diamond (no tools) google-gemini-3-pro-card | GPQA Diamond no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·max |
| Google model cards: HLE-Verified google-gemini-3.7-flash-card | HLE-Verified, 1811 items; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 5·default · Opus 5·default |
| Google model cards: Humanity's Last Exam (full set, no tools) google-gemini-3-pro-card | HLE full set text+MM, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·max · Haiku 4.5·solo |
| Google model cards: Humanity's Last Exam (search + code) google-gemini-3-pro-card | HLE full set text+MM, search (blocklist) + code execution; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·max |
| Google model cards: LABBench2 google-gemini-3.7-flash-card | LABBench2, Linux terminal with bioinfo tools, Python, R and internet; macro-average of 11 sub-tasks; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 5·default · Opus 5·default |
| Google model cards: LVBench google-gemini-3.7-flash-card | LVBench long video understanding, no tools; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 5·default · Opus 5·default |
| Google model cards: MMMLU google-gemini-3-pro-card | MMMLU multilingual MMLU; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·max · Haiku 4.5·solo |
| Google model cards: MMMU-Pro (avg Standard + Vision, no tools) google-gemini-3-pro-card | MMMU-Pro averaged over Standard (10 options) and Vision, no tools; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·max · Haiku 4.5·solo · Opus 4.7·default |
| Google model cards: MRCR v2 8-needle 128k (average) google-gemini-3-pro-card | MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Haiku 4.5·solo |
| Google model cards: MRCR v2 8-needle 128k (average) google-gemini-3.1-pro-card | MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·max · Opus 4.7·default · Sonnet 5·default |
| Google model cards: OSWorld-Verified google-gemini-3.5-flash-card | OSWorld-Verified, 100 max steps, pyautogui; Google model-card table(s) 19 May 2026 (google-gemini-3.5-flash-card); 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·default · Sonnet 5·default |
| Google model cards: SimpleQA Verified google-gemini-3-pro-card | SimpleQA Verified; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Haiku 4.5·solo |
| Google model cards: SWE-bench Verified (single attempt) google-gemini-3-pro-card | SWE-bench Verified, single attempt; Google model-card table(s) Nov 2025 (google-gemini-3-pro-card); Dec 2025 (google-gemini-3-flash-card); 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 4.6·max |
| Google model cards: Terminal-Bench 2.1 (Terminus-2) google-gemini-3.6-flash-card | Terminal-Bench 2.1, default agent harness Terminus 2 only; Google model-card table(s) 21 Jul 2026 (google-gemini-3.6-flash-card); 13 Aug 2026 (google-gemini-3.7-flash-card); 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xgen | Sonnet 5·default · Opus 5·default |
| Graphite AI Tells: count of distinct AI tells (Opus 5.5 update) graphite | writing task, 9974 topic-aligned pairs, words/phrases/frames used >=2x human rate; ref graphite.io/five-percent/research/ai-tells-opus-5-5-update; read 2026-10-05 ✓ | xgen | Opus 5·default · Opus 5.5·default |
| Graphite AI Tells: mannered-prose score (Opus 5.5 update) graphite | writing task, 1000-topic subsample, score 0-100 judged by Claude Opus 5; ref graphite.io/five-percent/research/ai-tells-opus-5-5-update; read 2026-10-05 ✓ | xgen | Opus 5·default · Opus 5.5·default |
| Graphite AI Tells: word-distribution divergence from human writing (Opus 5.5 update) graphite | writing task, 9974 topic-aligned LLM-vs-human article pairs, Jensen-Shannon divergence over unigrams; ref graphite.io/five-percent/research/ai-tells-opus-5-5-update; read 2026-10-05 ✓ | xgen | Opus 5·default · Opus 5.5·default |
| ianlpaterson ianlpaterson | OpenRouter · reasoning OFF ✓ | xgen | Haiku 4.5·nothink · Sonnet 4.6·nothink |
| kai_kou Haiku 5.5 vs 4.5 — support-message classification (Claude Code CLI) qiita-kaikou | claude -p --model claude-haiku-5-5 / claude-haiku-4-5, --tools "", 1-sentence system prompt, same classification task; cost left out of both rows (billing basis differs: Haiku 5.5 input billed as 1h-cache-write $0.20/MTok, Haiku 4.5 as normal input $1/MTok -- not comparable, see notes); score = correct-of-1 (both models classified the message as billing, correctly); read 2026-10-08 ✓ | xgen | Haiku 5.5·default · Haiku 4.5·solo |
| Marginlab SWE-Bench-Pro marginlab.ai | Claude Code CLI · daily rolling tracker, 6 Opus generations by release window, score-only ✓ | xgen | Opus 4.7·default · Opus 4.8·default · Opus 5·default · Opus 5.5·high |
| MobileCybench token/cost accounting arxiv-2609.23980 | Claude Code 100-run aggregate · Opus4.8/Opus5 default, tokens+cost ✓ | xgen | Opus 4.8·default · Opus 5·default |
| MrMerkus 3-task tests github-merkus | 3 tasks, 70 hidden checks · harness differs (Claude Code vs Antigravity), saturated, score only ✓ | xgen | Sonnet 5.5·default · Sonnet 4.6·default |
| Occam bench seed 1 github-occam | 9 tasks (seed 1), no plugin; Sonnet 5 max from the calibration run ✓ | xgen | Sonnet 5.5·max · Sonnet 5.5·medium · Opus 5.5·max · Opus 5.5·medium · Haiku 4.5·solo · Sonnet 5·max |
| OfficeQA arxiv-2603.08655 | Claude Agent SDK · reasoning HIGH ✓ | xgen | Sonnet 4.6·default · Haiku 4.5·solo |
| Official tau2-bench leaderboard, banking_knowledge (AllTools), Claude submissions by Sierra github-sierra-tau2-bench | banking_knowledge domain, 4 trials, user simulator gpt-5.2 (low), reasoning_effort max for all; pass^1 %; read 2026-10-18 ✓ | xgen | Opus 5·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max |
| Plotly Data Analytics Bench — Haiku 5.5 vs Haiku 4.5 (43-question exam) plotly-blog | plotly.com in-house 43-question data-analytics exam; effort not stated for either Haiku model (no config/slug found in the page's own JSON payload) -> default for both; read 2026-10-08 ✓ | xgen | Haiku 5.5·default · Haiku 4.5·solo · Opus 5.5·default |
| ponytail github-ponytail | Claude Code headless · thinking NS ✓ | xgen | Haiku 4.5·solo · Sonnet 4.6·nothink · Opus 4.8·nothink |
| Qiita suwa_nobu 12 models qiita-suwanobu | Claude Code 2.1.283 · trivial read/count/write task, cold cache, median of 3, CLI cost, CLI default effort ✓ | xgen | Haiku 4.5·solo · Sonnet 4.6·default · Opus 4.7·default · Opus 4.8·default · Fable 5·default · Sonnet 5·default · Opus 5·default · Fable 5.1·default · Opus 5.5·default |
| retort exp-46/75 bookshop Clojure github-retort | Claude Code · bookshop/rest-api-crud in Clojure, Opus 5 default (n=1) vs Opus 5.5 low (n=3) ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 bookshop Elixir github-retort | Claude Code · bookshop/rest-api-crud in Elixir, Opus 5 default (cost untraced) vs Opus 5.5 low, both saturated ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 bookshop Rust github-retort | Claude Code · bookshop/rest-api-crud in Rust, Opus 5 default (n=1) vs Opus 5.5 low (n=3) ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil C github-retort | Claude Code · brazil-soccer-mcp in C, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil C# github-retort | Claude Code · brazil-soccer-mcp in C#, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil C++ github-retort | Claude Code · brazil-soccer-mcp in C++, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil Clojure github-retort | Claude Code · brazil-soccer-mcp in Clojure, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil Elixir github-retort | Claude Code · brazil-soccer-mcp in Elixir, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil Erlang github-retort | Claude Code · brazil-soccer-mcp in Erlang, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil Java github-retort | Claude Code · brazil-soccer-mcp in Java, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil Obj-C github-retort | Claude Code · brazil-soccer-mcp in Objective-C, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil Rust github-retort | Claude Code · brazil-soccer-mcp in Rust, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil Swift github-retort | Claude Code · brazil-soccer-mcp in Swift, Opus 5 default vs Opus 5.5 low, n=1 ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-46/75 brazil TypeScript github-retort | Claude Code · brazil-soccer-mcp in TypeScript, Opus 5 default (cost untraced) vs Opus 5.5 low, both saturated ✓ | xgen | Opus 5·default · Opus 5.5·low |
| retort exp-79 vs exp-80 go (Opus 5.5 vs Fable 5.1, isolated) github-retort | Claude Code (isolated) · rest-api-crud in go, prompt=neutral, effort=default, Opus 5.5 vs Fable 5.1, n=3 each side; requirement coverage 1.0 → cost only; read 2026-10-05 ✓ | xgen | Opus 5.5·default · Fable 5.1·default |
| retort exp-79 vs exp-80 python (Opus 5.5 vs Fable 5.1, isolated) github-retort | Claude Code (isolated) · rest-api-crud in python, prompt=neutral, effort=low, Opus 5.5 vs Fable 5.1, n=3 each side; requirement coverage 1.0 → cost only; read 2026-10-05 ✓ | xgen | Opus 5.5·low · Fable 5.1·low |
| retort exp-79 vs exp-80 rust (Opus 5.5 vs Fable 5.1, isolated) github-retort | Claude Code (isolated) · rest-api-crud in rust, prompt=neutral, effort=low, Opus 5.5 vs Fable 5.1, n=3 each side; requirement coverage 1.0 → cost only; read 2026-10-05 ✓ | xgen | Opus 5.5·low · Fable 5.1·low |
| riku bug-fix (5 planted) note-riku_techlab | Claude Code 2.1.284, no tools · Sonnet 5 vs 5.5, n=3 medians, relative cost only ✓ | xgen | Sonnet 5·default · Sonnet 5.5·default |
| riku slides task note-riku_techlab | Claude Code 2.1.284, no tools · Sonnet 5 vs 5.5, n=3 medians, relative cost only; saturated ✓ | xgen | Sonnet 5·default · Sonnet 5.5·default |
| riku table task note-riku_techlab | Claude Code 2.1.284, no tools · Sonnet 5 vs 5.5, n=3 medians, relative cost only; saturated ✓ | xgen | Sonnet 5·default · Sonnet 5.5·default |
| SlopCode arxiv-2603.24755 | Claude Code · Reasoning HIGH ✓ | xgen | Sonnet 4.6·high · Opus 4.7·high |
| SlopCodeBench Oct 2026 (humanlayer writeup): strict checkpoints passed of 196 humanlayer-scb-2026-10 | 36 problems / 196 checkpoints, one campaign per model+effort; cost = recorded estimate per campaign (USD); read 2026-10-18 ✓ | xgen | Opus 5.5·medium · Opus 5.5·xhigh · Fable 5.1·xhigh |
| SWE-bench Verified swebench.com | swebench.com official board · mini-SWE-agent (v0.0.0->v2.0.0) submissions, instance_cost=mean $/instance, resolved%; 3 rows unverified-pending on site ✓ | xgen | Haiku 4.5·high |
| swe-rebench swe-rebench | ReAct minimal · Opus4.8 xhigh/4.7 high, Sonnet default ✓ | xgen | Opus 4.8·xhigh · Opus 4.7·high · Sonnet 4.6·default |
| TB2.1 HF · 0error harborframework | harborframework HF dataset · 0error-Ledger scaffold, opus-4.7 default, n≈230-241/446 ✓ | xgen | Opus 4.7·default |
| TB2.1 HF · Polaris harborframework | harborframework HF dataset · Polaris scaffold, opus-4.7 default (multi-model agent, sampled trials 100% opus-4.7), real $ cost, n≈268-444/446 ✓ | xgen | Opus 4.7·default |
| TB2.1 HF · Simplai harborframework | harborframework HF dataset · Simplai-Agent scaffold, sonnet-4.6 default, n≈360-428/446 ✓ | xgen | Sonnet 4.6·default |
| TB2.1 HF · vix harborframework | harborframework HF dataset · vix scaffold, opus-4.7 default, n≈388-440/446, tokens_out only ✓ | xgen | Opus 4.7·default |
| TB2.1 HF · WOZCODE harborframework | harborframework HF dataset · WOZCODE scaffold, opus-4.7 default, n≈443-445/446 ✓ | xgen | Opus 4.7·default |
| TOBench arxiv-2605.16909 | ReAct · thinking NS ✓ | xgen | Haiku 4.5·nothink · Sonnet 4.6·nothink |
| WildClaw arxiv-2605.10912 | OpenRouter (4 harness) · thinking NS ✓ | xgen | Opus 4.7·nothink · Opus 4.8·default · Fable 5·default |
| Writing-Elo effort/cost chart, Reddit 1wrkwp0 (r/LLMDevs) reddit-1wrkwp0 | same 'Writing Elo' metric family, 10 real YouTube scripts per OP, scored blind by three judges and a rubric; cost per script, log-x scatter with effort-labelled points; no board date printed on the chart; a different Reddit post/source snapshot than 1wr03jc, kept as its own board version; read 2026-10-05 ✓ | xgen | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·default · Opus 5.5·max · Fable 5.1·default · Fable 5.1·max |
| 1GC-7RC T1-val_bpb mean of 5 runs (Table 4) arxiv-2605.17046 | 1GC-7RC: seven ML-research tasks from scratch on one GPU, wall-clock budgets 40-120 min, no internet; CLI agent per provider (Claude Code / Codex CLI / OpenCode via OpenRouter); 5 runs per agent; runs April 2026; effort not stated; T1-val_bpb (lower-is-better); mean of 5 runs (Table 4); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| 1GC-7RC T2-top1 mean of 5 runs (Table 4) arxiv-2605.17046 | 1GC-7RC: seven ML-research tasks from scratch on one GPU, wall-clock budgets 40-120 min, no internet; CLI agent per provider (Claude Code / Codex CLI / OpenCode via OpenRouter); 5 runs per agent; runs April 2026; effort not stated; T2-top1 (higher-is-better); mean of 5 runs (Table 4); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| 1GC-7RC T3-mIoU mean of 5 runs (Table 4) arxiv-2605.17046 | 1GC-7RC: seven ML-research tasks from scratch on one GPU, wall-clock budgets 40-120 min, no internet; CLI agent per provider (Claude Code / Codex CLI / OpenCode via OpenRouter); 5 runs per agent; runs April 2026; effort not stated; T3-mIoU (higher-is-better); mean of 5 runs (Table 4); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| 1GC-7RC T4-AUROC mean of 5 runs (Table 4) arxiv-2605.17046 | 1GC-7RC: seven ML-research tasks from scratch on one GPU, wall-clock budgets 40-120 min, no internet; CLI agent per provider (Claude Code / Codex CLI / OpenCode via OpenRouter); 5 runs per agent; runs April 2026; effort not stated; T4-AUROC (higher-is-better); mean of 5 runs (Table 4); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| 1GC-7RC T5-acc mean of 5 runs (Table 4) arxiv-2605.17046 | 1GC-7RC: seven ML-research tasks from scratch on one GPU, wall-clock budgets 40-120 min, no internet; CLI agent per provider (Claude Code / Codex CLI / OpenCode via OpenRouter); 5 runs per agent; runs April 2026; effort not stated; T5-acc (higher-is-better); mean of 5 runs (Table 4); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| 1GC-7RC T6-MSE mean of 5 runs (Table 4) arxiv-2605.17046 | 1GC-7RC: seven ML-research tasks from scratch on one GPU, wall-clock budgets 40-120 min, no internet; CLI agent per provider (Claude Code / Codex CLI / OpenCode via OpenRouter); 5 runs per agent; runs April 2026; effort not stated; T6-MSE (lower-is-better); mean of 5 runs (Table 4); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| 1GC-7RC T7-acc mean of 5 runs (Table 4) arxiv-2605.17046 | 1GC-7RC: seven ML-research tasks from scratch on one GPU, wall-clock budgets 40-120 min, no internet; CLI agent per provider (Claude Code / Codex CLI / OpenCode via OpenRouter); 5 runs per agent; runs April 2026; effort not stated; T7-acc (higher-is-better); mean of 5 runs (Table 4); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| 1M-context retrieval: multi-hop chain retrieval (15 cells: 5 chains x 256K/512K/1M) arxiv-2605.02173 | Retrieval and multi-hop reasoning in 1M-token contexts (arXiv 2605.02173, 4 May 2026) Tables 5/6: multi-hop chain retrieval (15 cells: 5 chains x 256K/512K/1M); five models via official APIs; temperature 0 where allowed (Claude Opus 4.7 and GPT-5.5 enforce 1); score = correct cells / total as %; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| 1M-context retrieval: single-needle retrieval at 1M tokens (18 cells) arxiv-2605.02173 | Retrieval and multi-hop reasoning in 1M-token contexts (arXiv 2605.02173, 4 May 2026) Tables 5/6: single-needle retrieval at 1M tokens (18 cells); five models via official APIs; temperature 0 where allowed (Claude Opus 4.7 and GPT-5.5 enforce 1); score = correct cells / total as %; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| 34bench, single Thai bus-route scan transcription, 5 runs/model (dtinth/34bench) dtinth/34bench | results.svg (table-in-SVG, parsed as text) for median accuracy (1-weighted-CER over header/forward/return/footer); meta/ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Fable 5.1·default · Sonnet 5·default · Haiku 5.5·default |
| 3D car configurator, one run each, 40 min cap (u/Sufficient-Storage87) reddit-sufficient-storage87 | same prompt, fresh empty folder, 40-minute cap, self-screenshot; one run each; cost at API list price; read 2026-10-18 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Haiku 5.5·default |
| 4DCodeBench render-completion rate (of 200 planned cases) hf-4dcodebench | completion-rate proxy only (did the system produce any render at all), NOT a quality score of the renders; actual quality scoring (if any) is unpublished/not found; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Opus 5·high · Fable 5.1·high |
| 4DCodeBench: inverse-graphics-of-dynamic-scenes leaderboard (200 scenes), Claude Code agent 4dcodebench | arxiv.org/html/2610.03715v1 Figure 4 (src/main_leaderboard.png); Overall = mean of 5 family scores [0,1]; values printed on heatmap cells, read directly (not pixel-digitized); n=1 run/model, no repeats stated; read-from-image; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high |
| AA /task artificialanalysis | AA-Index · max ✓ | xmodel | Sonnet 5·max · Opus 4.8·max · Sonnet 4.6·max |
| AA /task v3 artificialanalysis | AA-Index v3 · max ✓ | xmodel | Opus 5·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max |
| AA /task v4 artificialanalysis | AA launch article · per-task $, Fable 5.1 xhigh/max vs Fable 5/Opus 5 max ✓ | xmodel | Fable 5.1·low · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·max · Fable 5·max |
| AA AutomationB. artificialanalysis | AA evaluation page · 657 tasks partial-credit, 3 models at max ✓ | xmodel | Fable 5·max · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Opus 4.8·max · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 4.6·max · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| AA Briefcase v1.1 (xAI Grok 4.7 post) xai-grok-4-7-launch | AA Briefcase v1.1 Elo, multi-hour office work; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Fable 5.1·max |
| AA Coding Agent Index artificialanalysis | AA · Claude Code harness, DeepSWE+TB4.0+SWE-Atlas, 3 Claude models at max, measured $ incl. cache; fallback 3–9 % ✓ | xmodel | Fable 5.1·max · Opus 5.5·max · Opus 5·max · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh |
| AA FrontierCode 1.1 (AA-run, weighted score) artificialanalysis | Artificial Analysis AA FrontierCode 1.1 (AA-run, weighted score); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·max |
| AA MMMU-Pro artificialanalysis | Artificial Analysis AA MMMU-Pro; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max |
| AA-AnalystAgent artificialanalysis | Artificial Analysis AA-AnalystAgent; score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Haiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max |
| AA-Briefcase (xAI Grok 4.6 post) xai-grok-4-6-launch | AA-Briefcase Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| AA-Briefcase Elo, Artificial Analysis article of 21 July 2026 artificialanalysis-article | AA-Briefcase Elo as printed in the AA article "Kimi K3: second only to Fable 5 on AA-Briefcase" (2026-07-21); earlier snapshot than the live board; read 2026-10-20 ✓ | xmodel | Sonnet 5·max · Opus 4.8·max · Fable 5·default |
| adebench github-ade-bench | config ✓ | xmodel | Fable 5·low · Opus 4.8·xhigh |
| Agent Island social-deduction arena (Plackett-Luce skill), OpenRouter models arxiv-2605.04312 | Agent Island (arXiv 2605.04312, 5 May 2026): 999 multi-agent social-deduction games with 49 models via OpenRouter, Bayesian Plackett-Luce posterior skill (mean of lambda; prior mean 1; Table 1); reasoning effort not stated; no cost; read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| Agent Security League endorlabs | Endor Labs post 24 Sep · 200 tasks, Claude Code, total run cost/200, effort unstated ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Opus 5·default |
| AgentAbstain paired accuracy, Claude SDK harness agentabstain-2607.10059 | AgentAbstain 263 paired tasks (should-act / should-abstain), 42 sandboxes, each model run once in its vendor harness (Claude SDK), judge GPT-5.4 + deterministic commit check, macro over 8 categories; metric Paired Accuracy % (Act/Abstain accuracy = same runs); arXiv 2607.10059 Table 7 (Jul 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| AgentHop agentic multi-hop scientific QA accuracy and cost agenthop-2609.34428 | AgentHop 1,011 four-option items, 7 tools, 20 turns, 200K-token cap, one attempt, same harness for all; metric accuracy 0-1; cost = USD for the full 1,011-item pass at official per-token pricing recorded in the released harness (price date not given); arXiv 2609.34428 Tables 2/8 (Sep 2026); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| Agentic RSR scene refinement via DeepSeek Harness (scenes passing visual check, of 18) arxiv-2610.10479 | 18 real scenes per configuration, Oct 6 2026 evaluation summary, DSH harness; Table 3 of arXiv 2610.10479; read 2026-10-15 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default · Sonnet 5.5·default |
| Agents on Guren, Stage 2 re-run, bare condition (gurenjs/agents-on-guren) gurenjs/agents-on-guren | RESULTS-STAGE2.md at commit 2b43507 ("Stage 2 re-run with auto memory off and Sonnet 5.5", 2026-09-30), bare condition; Claude Code 2.1.284, 3 trials per cell, 9 tasks; pass is hidden-behaviour-test pass/fail per cell (27 cells); cost is median API-equivalent USD per cell from Claude Code total_cost_usd; effort: run.sh passes only --model "$MODEL", no effort flag, so default for every model; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Haiku 4.5·solo · Fable 5.1·default |
| Agents on Guren, Stage 2 re-run, shipped condition (gurenjs/agents-on-guren) gurenjs/agents-on-guren | RESULTS-STAGE2.md at commit 2b43507 ("Stage 2 re-run with auto memory off and Sonnet 5.5", 2026-09-30), shipped condition; Claude Code 2.1.284, 3 trials per cell, 9 tasks; pass is hidden-behaviour-test pass/fail per cell (27 cells); cost is median API-equivalent USD per cell from Claude Code total_cost_usd; effort: run.sh passes only --model "$MODEL", no effort flag, so default for every model; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Haiku 4.5·solo · Fable 5.1·default |
| Agents on Rails · atomic rubyonrails | rubyonrails.org/ai + rails/ai-evals runs · 21 Writebook tasks ×3, miniswen, list-price cost/run ✓ | xmodel | Opus 5·high · Fable 5.1·high · Fable 5·high · Opus 4.8·high · Sonnet 5·high |
| Agents' Last Exam (GLM-5.3-Flash chart) zai-hf-glm-5.3-flash | Agents' Last Exam; Flash README footnote empty; GLM-5.2 = 20.4 here vs 23.8 in the GLM-5.3 card (other snapshot, own group); chart 2026-10-15; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| Agents' Last Exam (OpenAI GPT-5.6 post chart) openai-gpt56-launch | Agents' Last Exam, professional workflows, code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max |
| Agents' Last Exam (OpenAI GPT-6 Astra post chart) openai-gpt6-astra-launch | Agents' Last Exam; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| Agents' Last Exam (OpenAI GPT-6 Sol and Luna post) openai-gpt6-sol-luna-launch | Agents' Last Exam; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Fable 5·default · Fable 5·xhigh · Opus 5·high |
| Agents' Last Exam ALE-CLI (GLM-5.3 card) zai-hf-glm-5.3 | ALE official protocol, Claude Code harness, 105 tasks, effort=max 1M context 64K output, 4h timeout; GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max · Fable 5·max |
| Agents' Last Exam computing-and-math split (ALE-CM, 18 tasks), mean graded score S@1 arxiv-2610.18868 | 18 tasks x 5 rollouts, Claude Code harness; Table 2 of arXiv 2610.18868; read 2026-10-15 ✓ | xmodel | Fable 5·default · Opus 5·default · Sonnet 5·default · Opus 4.8·default |
| Agents' Last Exam Near-term (67 tasks) partial score, RSIAgent paper table arxiv-2609.15364 | Agents' Last Exam Near-term 67 tasks, partial %; table baselines mostly copied from official reports/blogs; paper 2026-09-15; differs from held ALE groups; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Fable 5·default · Opus 5·default |
| AgentTime (arXiv 2610.09944) Agents’ Last Exam under duration requests arxiv-2610.09944 | AgentTime Table 5 column Agents’ Last Exam: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) AppWorld under duration requests arxiv-2610.09944 | AgentTime Table 5 column AppWorld: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) AssistantBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column AssistantBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) CORE-Bench v1.1 under duration requests arxiv-2610.09944 | AgentTime Table 5 column CORE-Bench v1.1: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) DeepSWE v1.1 under duration requests arxiv-2610.09944 | AgentTime Table 5 column DeepSWE v1.1: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) GPQA Diamond under duration requests arxiv-2610.09944 | AgentTime Table 5 column GPQA Diamond: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) Humanity’s Last Exam under duration requests arxiv-2610.09944 | AgentTime Table 5 column Humanity’s Last Exam: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) METR public tasks under duration requests arxiv-2610.09944 | AgentTime Table 5 column METR public tasks: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) OSWorld 2.0 under duration requests arxiv-2610.09944 | AgentTime Table 5 column OSWorld 2.0: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) PaperBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column PaperBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) PostTrainBench v1.1 under duration requests arxiv-2610.09944 | AgentTime Table 5 column PostTrainBench v1.1: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) PPTArena under duration requests arxiv-2610.09944 | AgentTime Table 5 column PPTArena: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) ProgramBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column ProgramBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) Sakana ALE-Bench under duration requests arxiv-2610.09944 | AgentTime Table 5 column Sakana ALE-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) Terminal-Bench 4.0 under duration requests arxiv-2610.09944 | AgentTime Table 5 column Terminal-Bench 4.0: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) TUA-Bench under duration requests arxiv-2610.09944 | AgentTime Table 5 column TUA-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) WildClawBench under duration requests arxiv-2610.09944 | AgentTime Table 5 column WildClawBench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| AgentTime (arXiv 2610.09944) YC-Bench under duration requests arxiv-2610.09944 | AgentTime Table 5 column YC-Bench: mean native score of each agent (model in its own harness, one run per task-duration, pooled over the three requested durations; duration-following instruction appended to each prompt); max reasoning effort; submitted 7 Oct 2026; read 2026-10-20 ✓ | xmodel | Fable 5.1·max |
| agp-gsm8k github-agentic-pipeline | config ✓ | xmodel | Sonnet 5·low · Haiku 4.5·req-low |
| agp-mbpp github-agentic-pipeline | config ✓ | xmodel | Sonnet 5·low · Haiku 4.5·req-low |
| agp-mmlupro github-agentic-pipeline | config ✓ | xmodel | Sonnet 5·low · Haiku 4.5·req-low |
| AI Coding Daily LLM Coding Leaderboard (4 projects, max 70) aicodingdaily | Laravel 20 + React-TS 20 + bug finding 10 + edge-case 20 = 70; Claude Code rows; author-scored; page 'Last update: October 17th, 2026'; read 2026-10-18 ✓ | xmodel | Opus 5.5·high · Opus 5.5·medium · Sonnet 5.5·high · Fable 5.1·medium · Opus 5.5·low · Sonnet 5.5·medium · Haiku 5.5·high · Haiku 5.5·medium |
| AI for Mortals · Backrooms game aiformortals | Claude Code /goal, high effort, Blender + web game, one build per model, cost only, 29 Sep ✓ | xmodel | Sonnet 5.5·high · Sonnet 5·high · Opus 5.5·high · Fable 5.1·high |
| AI for Mortals · demoscene aiformortals | Claude Code /goal, high effort, code-only demo, one build per model, cost only, earlier round ✓ | xmodel | Sonnet 5.5·high · Sonnet 5·high · Opus 5.5·high · Fable 5.1·high |
| AI for Mortals · launch video aiformortals | Claude Code /goal, high effort, one build per model, no numeric score (cost, tokens, time only), 29 Sep ✓ | xmodel | Sonnet 5.5·high · Sonnet 5·high · Opus 5.5·high · Fable 5.1·high |
| AI for Mortals · RPG aiformortals | Claude Code /goal, high effort, Diablo-style RPG, one build per model (no Sonnet 5), cost only, earlier round ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high · Fable 5.1·high |
| AI for Mortals · website clone aiformortals | Claude Code /goal, high effort, prompt as file, one build per model, cost only, 29 Sep ✓ | xmodel | Sonnet 5.5·high · Sonnet 5·high · Opus 5.5·high · Fable 5.1·high |
| AI Pricing Guru Labs aipricing.guru | OpenRouter cost-per-task-v2 · 49-task suite, 6 Claude models, effort unstated ✓ | xmodel | Opus 5.5·nothink · Sonnet 5·nothink · Opus 4.8·nothink · Opus 5·nothink · Fable 5·nothink · Haiku 4.5·solo |
| ai-arena savannah github-ai-arena | Claude Code · savannah-ecosystem, Fable 5 vs Sonnet 5 max, tokens only ✓ | xmodel | Fable 5·max · Sonnet 5·max |
| ai-arena tokyo github-ai-arena | Claude Code · tokyo-island, Fable 5 vs Sonnet 5 max, tokens only; Sonnet metrics identical to savannah ✓ | xmodel | Fable 5·max · Sonnet 5·max |
| aiarena-balls github-ai-arena | config ✓ | xmodel | Opus 4.8·low · Opus 4.8·high |
| aiarena-cube github-ai-arena | config ✓ | xmodel | Opus 4.8·low · Opus 4.8·high |
| aibenchy.com private benchmark suite — Haiku 5.5 (xhigh) vs Opus 5.5 (high) aibenchy | aibenchy.com undisclosed-task private suite, score 1-10 scale (site's own verdict text quotes '9.1 vs 9.4'); page states 'Last updated at: 2026-10-07'; Haiku-5.5-xhigh row is the same run repeated across two /compare/ pages (identical score/cost/time/rank), not two independent measurements; read 2026-10-08 ✓ | xmodel | Haiku 5.5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5·high · Opus 5·medium · Opus 5.5·medium · Sonnet 5.5·max · Opus 5·low · Fable 5·medium · Sonnet 5.5·high · Haiku 5.5·low · Sonnet 5·medium · Haiku 5.5·max · Haiku 5.5·high · Opus 4.8·medium · Opus 4.7·medium · Fable 5.1·high · Haiku 5.5·medium · Fable 5.1·medium · Opus 4.8·low · Opus 5·nothink · Sonnet 5.5·low · Opus 4.8·nothink · Sonnet 5.5·medium · Fable 5.1·low · Sonnet 5·nothink |
| AIFM: Adizero shoe in Blender (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| AIFM: Awwwards clone (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| AIFM: Launch film (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| AIFM: Mana Yerba Mate site (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| AIFM: Moby Dick illustration in code (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| AIFM: RollerCoaster Tycoon clone (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| AIFM: THPS-style 3D game (Opus 5.5 vs Fable 5.1) aiformortals | AI for Mortals one-shot build, Claude Code /goal mode, high effort both arms; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| AIME 2026 (GLM-5.2 card) zai-hf-glm-5.2 | AIME 2026, temperature=1.0 top_p=0.95 163,840 tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| aimecontract arxiv-2608.16956 | config ✓ | xmodel | Sonnet 5·default · Sonnet 5·high · Fable 5·high |
| AIMultiple text-to-SQL (Claude Code) aimultiple.com | AIMultiple AIM-Text-to-SQL, Opus 5.5 and Fable 5.1 in Claude Code at medium effort · SQL accuracy in %; each model answered its own subset of questions (506 and 491); no cost; read 2026-10-01 ✓ | xmodel | Opus 5.5·medium · Fable 5.1·medium |
| AIMultiple text-to-SQL (OpenRouter) aimultiple.com | AIMultiple AIM-Text-to-SQL, Fable 5 and Opus 5 via OpenRouter at temperature 0, effort unstated (default) · SQL accuracy in %; question subsets differ by model; no cost; read 2026-10-01 ✓ | xmodel | Fable 5·default · Opus 5·default · Sonnet 5.5·default · Haiku 5.5·default · Sonnet 5·default · Haiku 4.5·solo · Opus 4.8·default |
| AIReiter aireiter | Claude Code · high ✓ | xmodel | Sonnet 5·high · Opus 4.8·high |
| AI生活 (YouTube): 25 questions x 99 runs on claude.ai chat, Haiku 5.5 vs Sonnet 5.5 vs Opus 5.5 youtube-aiseikatsu | claude.ai chat (not API); new chat per question; Q1-12 x5 + Q13-25 x3 = 99 runs; same grading as 29 Sep; runs on different days (Opus 26+29 Sep; Sonnet 29 Sep; Haiku 8 Oct); read 2026-10-15 ✓ | xmodel | Haiku 5.5·high · Haiku 5.5·max · Sonnet 5.5·high · Opus 5.5·high |
| Akita LLM Benchmark v4 akitaonrails | akitaonrails.com 23 Sep · 7-sprint Rails app with 14 planted security sabotages, Claude Code, effort not stated (tool default: out of the fit), n=1 ✓ | xmodel | Opus 5·default · Fable 5·default · Opus 5.5·default · Fable 5.1·default · Sonnet 5·default · Opus 4.8·default · Sonnet 4.6·default |
| ALE-Bench epoch-archive | Epoch archive · 3 models at matched high, measured token splits ✓ | xmodel | Opus 5·high · Fable 5.1·high · Fable 5·high · Opus 5.5·high · Sonnet 5.5·high |
| AlgoWorlds (240 algorithmic worlds), exact optimality arxiv-2608.29397 | AlgoWorlds (arXiv 2608.29397, 29 Aug 2026): 240 algorithmic worlds, 3 trials, agent acquires information with paid tools then decides; score = exact optimality % (first final-decision metric of Table 2; feasibility, reference utility, sufficiency, coverage are other columns of the same runs); acquisition spend in tool-budget units, not USD (no cost row); read 2026-10-20 ✓ | xmodel | Sonnet 5·max · Opus 4.8·max |
| AminBlg/SimpleEnglish: ASD-STE100 skill vs baseline, 9 Claude models x 8 tasks, violations per 100 words (regex linter), --effort low github-simpleenglish | evals/results/RESULTS.md (pushed 5 Oct); score = reduction of STE violations per 100 words with the skill vs baseline (higher is better); output tokens = skill condition; read 2026-10-18 ✓ | xmodel | Fable 5.1·low · Fable 5·low · Opus 5·low · Opus 4.8·low · Opus 4.7·low · Sonnet 5·low · Sonnet 4.6·low |
| Analog Design Bench: pass@1 arxiv-2609.33356 | Analog Design Bench (arXiv 2609.33356 v1 27 Sep 2026); 50 transistor-level analog design tasks x 3 rollouts = 150 two-hour attempts per configuration; main metric pass@1 (full-spec pass); GPT/Claude in native Codex/Claude Code, others via Claude Code on compatible APIs; cost = per-attempt mean; read 2026-10-20 ✓ | xmodel | Opus 5·max · Fable 5·max |
| Andon Labs Blueprint-Bench 2: floorplan-from-photos, Opus 5.5 vs seven older Claude models andonlabs-bb2 | 32-model board, floorplan drawn from apartment-interior photos, score 0-1 vs human baseline 0.59; https://andonlabs.com/evals/blueprint-bench-2, src/andonlabs_bb2.html (SVG bar chart, y/height attributes, axis calibrated y=355->0.0 / y=0->1.0); no Sonnet 5.5 bar on this board; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Fable 5·default · Opus 5·default · Sonnet 5·default · Opus 4.7·default · Opus 4.8·default · Sonnet 4.6·default · Haiku 4.5·solo · Sonnet 5.5·default · Haiku 5.5·default |
| APEX-Agents (xAI Grok 4.6 post) xai-grok-4-6-launch | APEX-Agents %, harness not stated (Mercor's benchmark by name; page does not attribute); xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| APEX-Agents public leaderboard rows quoted by StagedWorkspace (Pass@1) stagedws-2608.18050 | APEX-Agents public leaderboard rows quoted as published by arXiv 2608.18050 Table 2b (Pass@1 with 8 independent attempts per task), snapshot unknown (older than the held Mercor board: cells differ); same-benchmark-as:mercor-apexagents; metric Pass@1 %; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| APEX-Harm mean task score under attack (all 6 categories) apexharm-2610.13623 | APEX-Harm 180 attack task instances per model (15 underlying APEX-Agents tasks x 12 attack variants), APEX-Agents harness with MCP tools, 150 steps, no timer, judge/rules per paper; metric mean task score 0-1 (utility under attack); ASR/ER safety rates not extracted; arXiv 2610.13623 Table 4 (Oct 2026); cost = Table 1 total for Gemini 3.8 Flash only; read 2026-10-20 ✓ | xmodel | Opus 5·max · Sonnet 5·max |
| ARC-AGI-3 (OpenAI GPT-6 Astra post chart; Sol and Opus 5 only) openai-gpt6-astra-launch | ARC-AGI-3; Astra post chart; official-harness style rows only (Sol 7.78, Opus 5 30.16); the Astra row (Responses API harness, code mode) is a different setup and is left out; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Opus 5·default |
| ARC-AGI-3 (semi-private) arcprize | ARC Prize ARC-AGI-3 semi-private interactive-game set, score in % (as published x100), cost = total run USD; Haiku 5.5 on the Standard harness (the Provider-Adapter harness entry, 2.82 % / $437.14, is a different configuration and is not mixed in); read 2026-10-15 ✓ | xmodel | Opus 4.7·high · Opus 4.8·high · Opus 5·high · Haiku 5.5·max |
| Argo-Bench: enterprise data-agent tasks (210 tasks), Inspect AI 0.3.263 harness argo-bench | arxiv.org/html/2610.02122v1 Table 2 (best-effort snapshot, extra-high where offered) + Table 10 (effort ladder, mean score /100, tasks completed at every level); n=1 rollout/task; score = mean task score out of 100 with a model's forecast tasks floored at 0 as a block; cost = API spend per task, excluding BigQuery (Table 2 only, not itemized by effort in Table 10); read 2026-10-05 ✓ | xmodel | Opus 5.5·xhigh · Sonnet 5.5·xhigh · Sonnet 5·xhigh · Haiku 4.5·solo · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high |
| ArtifactArena model leaderboard, Design Lab Harness (DLH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Fable 5·high · Sonnet 5·high · Opus 4.8·high |
| ArtifactArena model leaderboard, Sampling Harness (SH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Fable 5·high · Sonnet 5·high · Opus 4.8·high |
| ArtifactArena model leaderboard, Verifier-Grounded Refinement Harness (VGH) Elo arxiv-2610.06511 | Best artifact per (model, harness) in a cross-model Bradley-Terry round robin, MuJoCo robot arena; Figure 2 of arXiv 2610.06511 (read from image); read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Fable 5·high · Sonnet 5·high · Opus 4.8·high |
| Artificial Analysis AA-3D v1.0 (blind pairwise Elo, interactive 3D scenes) artificialanalysis | agent in offline sandbox (Stirrup harness), 3 h / 250 turns, brief -> three.js scene, panel blind pairwise votes -> Elo; cost = mean per scene (AA list-price); read 2026-10-15 ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Haiku 5.5·max |
| Artificial Analysis AA-Robotics v0.1 (8 tasks, 6-axis arm) artificialanalysis | 8 zero-shot tabletop tasks, 1 attempt each, human-judged end state, score 0-100 averaged; Claude in Claude Code at high; cost per task; read 2026-10-15 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| Artificial Analysis Coding Agent Index v1.1 (OpenAI GPT-5.6 post chart) openai-gpt56-launch | Artificial Analysis Coding Agent Index v1.1; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·medium · Opus 4.8·max |
| Artificial Analysis Coding Agent Index v1.4 (OpenAI GPT-6 Astra post chart, non-code) openai-gpt6-astra-launch | Artificial Analysis Coding Agent Index v1.4; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 5·nothink · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| Artificial Analysis Intelligence Index (OpenAI GPT-5.5 post chart) openai-gpt55-launch | Artificial Analysis Intelligence Index; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | Opus 4.7·max · Opus 4.7·nothink |
| Artificial Analysis: AA-Briefcase (live board, 20 Oct 2026) artificialanalysis | AA-Briefcase live board read 2026-10-20 (cells absent from the 15 Oct read) | xmodel | Opus 4.7·max · Opus 4.7·high · Haiku 4.5·solo |
| ASP theory distillation CLEVR-B0 arxiv-2607.28086 | Distilling ASP theories: agent (read/edit/write/glob/grep tools + clingo) writes a complete theory.lp from an empty file within a 1-hour limit; acc_strict on val/200 (%), mean ± sd of 3 samples; B = number of reference theories; all models other than Anthropic and OpenAI via OpenRouter; configuration clevr-b0; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| ASP theory distillation CLEVRER-B0 arxiv-2607.28086 | Distilling ASP theories: agent (read/edit/write/glob/grep tools + clingo) writes a complete theory.lp from an empty file within a 1-hour limit; acc_strict on val/200 (%), mean ± sd of 3 samples; B = number of reference theories; all models other than Anthropic and OpenAI via OpenRouter; configuration clevrer-b0; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| ASP theory distillation CLEVRER-B1C arxiv-2607.28086 | Distilling ASP theories: agent (read/edit/write/glob/grep tools + clingo) writes a complete theory.lp from an empty file within a 1-hour limit; acc_strict on val/200 (%), mean ± sd of 3 samples; B = number of reference theories; all models other than Anthropic and OpenAI via OpenRouter; configuration clevrer-b1c; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| ASP theory distillation CLEVRER-B1G arxiv-2607.28086 | Distilling ASP theories: agent (read/edit/write/glob/grep tools + clingo) writes a complete theory.lp from an empty file within a 1-hour limit; acc_strict on val/200 (%), mean ± sd of 3 samples; B = number of reference theories; all models other than Anthropic and OpenAI via OpenRouter; configuration clevrer-b1g; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| ASP theory distillation CLEVRER-B2 arxiv-2607.28086 | Distilling ASP theories: agent (read/edit/write/glob/grep tools + clingo) writes a complete theory.lp from an empty file within a 1-hour limit; acc_strict on val/200 (%), mean ± sd of 3 samples; B = number of reference theories; all models other than Anthropic and OpenAI via OpenRouter; configuration clevrer-b2; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default |
| ASP theory distillation GQA-B0 arxiv-2607.28086 | Distilling ASP theories: agent (read/edit/write/glob/grep tools + clingo) writes a complete theory.lp from an empty file within a 1-hour limit; acc_strict on val/200 (%), mean ± sd of 3 samples; B = number of reference theories; all models other than Anthropic and OpenAI via OpenRouter; configuration gqa-b0; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| ASP theory distillation GQA-B1 arxiv-2607.28086 | Distilling ASP theories: agent (read/edit/write/glob/grep tools + clingo) writes a complete theory.lp from an empty file within a 1-hour limit; acc_strict on val/200 (%), mean ± sd of 3 samples; B = number of reference theories; all models other than Anthropic and OpenAI via OpenRouter; configuration gqa-b1; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| Atomic Agent expense-lib atomicagent | Atomic Agent 0.6.5 via OpenRouter · 5 bugs + 2 features, 32 hidden tests, n=3, provider default effort, billed cost; saturated ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Sonnet 5·default |
| AttractorBench v2.0.0 NLSpec (T1/T2/T3+judge), claude-code harness, best run per model strongdm-attractorbench | strongdm/attractorbench bench_version 2.0.0, main task only; score=5% build+5% self-test+25% T1+25% T2+25% T3+15% judge; Harbor, agent claude-code; leaderboard statistic = best run per agent/model (snapshot 2026-02-26); cost = Harbor result.json x litellm pricing; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| AutoBCI (2609.35456) — agentic EEG architecture search, Claude Code, 6-round search arxiv-2609.35456 | Table 3, average test bAcc%/wF1/Kappa over MI/emotion/sleep held-out sets; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Sonnet 5·default · Opus 5.5·default |
| AutomationB. arxiv-2604.18934 | harness+effort NOT stated ✓ | xmodel | Opus 4.7·max · Sonnet 4.6·max · Sonnet 5·max · Haiku 4.5·solo |
| AutomationBench (OpenAI GPT-5.6 post chart) openai-gpt56-launch | AutomationBench pass rate, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| AutomationBench (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | AutomationBench; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh |
| AutomationBench (OpenAI GPT-6 Sol and Luna post) openai-gpt6-sol-luna-launch | AutomationBench; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| AutomationBench v1.0.6 (GLM-5.3 card) zai-hf-glm-5.3 | AutomationBench v1.0.6 (null-type fix PR #13); GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| Autonomous Science Bench (6 lab campaigns x 3 trials) arxiv-2610.18449 | Claude Code agent, 18 trials per configuration, resolved = discovery and learning gates both pass; cost = agent-reported API cost summed over the 18 trials; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·high · Haiku 4.5·solo |
| aws-smoke github-aws-samples | config ✓ | xmodel | Opus 5·default · Opus 4.8·default |
| BAIS single-cell analysis (18 tasks x 3 replicates), official accuracy arxiv-2610.04915 | Claude Code headless, official benchmark score on the original evidence (E); read 2026-10-15 ✓ | xmodel | Opus 5.5·default · Sonnet 5·default · Haiku 4.5·solo |
| Bakery website, Sonnet 5.5 vs Opus 5.5 at high (AI Field Notes video) youtube-aifieldnotes | Claude Code; same brief; empty folder; high effort; one try; 2026-10-10; read 2026-10-18 ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| bangla-llm-bench, belebele track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Sonnet 5·default · Haiku 4.5·solo |
| bangla-llm-bench, bnexam track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=300 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Sonnet 5·default · Haiku 4.5·solo |
| bangla-llm-bench, mgsm track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=250 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Sonnet 5·default |
| bangla-llm-bench, mmlu track (paired EN/BN; deluair/bangla-llm-bench) deluair/bangla-llm-bench | scores.csv at commit 0cc4f72 (2026-09-28); n=285 items; score is Bengali (bn) accuracy (the EN score is confound-only); cost is the CLI-reported api_cost_usd for both languages combined (one cost figure covers the en+bn pair); effort: models.toml sets only cli+model, README says "Each model runs at its provider's default settings", so default for every Claude model; read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Sonnet 5·default · Haiku 4.5·solo |
| Bazaar multi-attribute auction, cumulative profit bazaar-2608.00102 | Bazaar sealed-bid multi-attribute auction, ~79 rounds, 10 seeds per row (GPT-5.5 xhigh 3 seeds), closed-form evaluation, no tools beyond bidding API; metric cumulative profit USD (game money, not spend; efficiency eta of Table 6 = same runs, not kept); arXiv 2608.00102 Table 3 (Aug 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·xhigh · Sonnet 4.6·medium · Sonnet 4.6·high · Opus 4.7·medium · Opus 4.7·high |
| BenchCAD (python tool) (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | BenchCAD (python tool); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max |
| BenchCAD (python tool) (table) openai-gpt56-launch | BenchCAD (python tool); OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| BenchCAD (table) openai-gpt56-launch | BenchCAD; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| BenchCAD (table) openai-gpt6-astra-launch | BenchCAD; OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Fable 5·default · Opus 5·default |
| Big Finance Bench (table) openai-gpt56-launch | Big Finance Bench; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| Binder design (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| Bio-imaging (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| BioMystery hard-dist (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| BioMystery hard-sub (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| BioPhys-Bridge no-scaffold evaluation: evidence-ID F1 arxiv-2609.19180 | BioPhys-Bridge (independent researcher, arXiv 2609.19180 v1 15 Sep 2026) Table 7 'preliminary de-leaked no-scaffold evaluation': 154-task held-out test set, temperature 0, JSON-mode decoding; score = Evidence-ID F1 (answer token F1 = same runs); read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default |
| Bito agentic coding battery bito.ai | Bito's internal agentic coding battery in Claude Code (6 task types, multi-turn sessions), each turn scored out of 50 by Opus 5.5 · mean score /50, cost per session (USD); effort from the post; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·medium · Sonnet 5.5·high · Opus 5.5·medium · Haiku 5.5·medium · Haiku 5.5·high · Haiku 4.5·solo |
| BixBench3 mean artifact score (20 comp-bio tasks) bixbench3-2608.25286 | BixBench3 20 tasks / 138 artifacts, Inspect AI ReAct agent, 5 tools, 24 h / 5000 message cap, programmatic artifact grading, 20 runs per model (260 runs), each model at its maximum thinking effort; metric mean task score 0-1; cost = mean USD per task over 20 tasks from per-call tokens x provider list prices; arXiv 2608.25286 (Aug 2026); read 2026-10-20 ✓ | xmodel | Opus 5·max · Opus 4.8·max · Sonnet 4.6·max · Haiku 4.5·req-64k |
| BizBench (arXiv 2610.20424) strict success arxiv-2610.20424 | BizBench: 50 long-horizon business-workflow tasks of a payments company (789 documents, 601 tools), user simulator DeepSeek-V4.1-Flash, 3 runs per task (1 run for Claude Opus 5), strict success = goal AND process AND protection (%), Table 1; no cost; arXiv v1 2026-10-14; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| Blast Radius (SkinGraph repo-in-context symbol usage, Opus 4.8 vs GLM-5.2) github-dvallslanaquera-glm5.2-benchmark | github.com/dvallslanaquera/glm5.2-benchmark results/latest.json generated 2026-06-24: whole repo (62 files, ~131.7k tokens) in context, 10 symbols, list files/lines using each, oracle = grep; score = macro file-F1 over the 10 symbols (line-F1 from the same calls in confound), one shot, temperature 0 for GLM; cost = sum of the 10 calls at list-price x tokens; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: $OneMillion-Bench (expert score) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark '$OneMillion-Bench (expert score)'; no cost or tokens in the source (score only); footnotes: "Evaluated using gemini-3.1-pro-preview."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: Agents' Last Exam (Pass / Score) [pass] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Agents' Last Exam (Pass / Score)' [pass]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: AndroidBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'AndroidBench'; no cost or tokens in the source (score only); footnotes: "Evaluated on the 95-task public subset, reporting avg@3 scores."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: AndroidWorld qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'AndroidWorld'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: Automation-Bench (Pass@1) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Automation-Bench (Pass@1)'; no cost or tokens in the source (score only); footnotes: "Evaluated on the 600-task public subset."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: BabyVision [with-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'BabyVision' [with-CI]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: BabyVision [without-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'BabyVision' [without-CI]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: CC-OCR-Bench-V2 qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'CC-OCR-Bench-V2'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: CharXiv (RQ) [with-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'CharXiv (RQ)' [with-CI]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: CharXiv (RQ) [without-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'CharXiv (RQ)' [without-CI]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: ClawEval-MM [average] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'ClawEval-MM' [average]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: COCO qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'COCO'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: CountQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'CountQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: CoWorkBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'CoWorkBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: DeepSWE 1.1 qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'DeepSWE 1.1'; no cost or tokens in the source (score only); footnotes: "We report the highest score among both harnesses"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: Dense200 qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'Dense200'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: EgoLife (w/ Mem.) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'EgoLife (w/ Mem.)'; no cost or tokens in the source (score only); footnotes: "Scores are evaluated using a memory system built with Qwen-MM-Plugins"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: ERQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'ERQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: FrontierSWE qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'FrontierSWE'; no cost or tokens in the source (score only); footnotes: "All other available MEAN@5 results are taken from the official FrontierSWE leaderboard" / "Evaluated with the Claude Code harness."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: HealthBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HealthBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: HiPhO qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'HiPhO'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: HLE qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HLE'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: HLE w/ tools qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'HLE w/ tools'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: IFBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'IFBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: JobBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'JobBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: LingoQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'LingoQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: LogicVista qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'LogicVista'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: LongBench v2 qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'LongBench v2'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: LVBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'LVBench'; no cost or tokens in the source (score only); footnotes: "Scores are evaluated using a memory system built with Qwen-MM-Plugins"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: LVBench (w/ Mem.) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'LVBench (w/ Mem.)'; no cost or tokens in the source (score only); footnotes: "Scores are evaluated using a memory system built with Qwen-MM-Plugins"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MADQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MADQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MathVision [with-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MathVision' [with-CI]; no cost or tokens in the source (score only); footnotes: "For other models, we report the higher score obtained from runs with and without the `\boxed{}` formatting requirement."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MathVision [without-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MathVision' [without-CI]; no cost or tokens in the source (score only); footnotes: "For other models, we report the higher score obtained from runs with and without the `\boxed{}` formatting requirement."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MedXpertQA-MM qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MedXpertQA-MM'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MLVU (M-Avg) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MLVU (M-Avg)'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: MMMU-Pro qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MMMU-Pro'; no cost or tokens in the source (score only); footnotes: "All other models are evaluated in-house." / "Results for Gemini3.1-Pro and GPT5.6-Sol are taken from official model reports or system cards."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MMStar qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MMStar'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MMVU qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MMVU'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MobileWorld qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MobileWorld'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: MRCR v2 256K (8-needle) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'MRCR v2 256K (8-needle)'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: MTVQA-Test qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'MTVQA-Test'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: NL2Repo-Bench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'NL2Repo-Bench'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: OCR-Bench-V2 (EN/ZH) [EN] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'OCR-Bench-V2 (EN/ZH)' [EN]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: OCR-Bench-V2 (EN/ZH) [ZH] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'OCR-Bench-V2 (EN/ZH)' [ZH]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: OmniDocBench 1.5 qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'OmniDocBench 1.5'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: OSWorld 2.0 [binary] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'OSWorld 2.0' [binary]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: OSWorld-Verified qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'OSWorld-Verified'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: PaperBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PaperBench'; no cost or tokens in the source (score only); footnotes: "Evaluated in the BasicAgent setting under Code-Dev mode, judged by Claude Opus 4.6, and averaged over 3 runs (max 12 hours per run)."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: Parametric CAD Bench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'Parametric CAD Bench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: PhyX qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'PhyX'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: PLawBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PLawBench'; no cost or tokens in the source (score only); footnotes: "Evaluated using gemini-3.1-pro-preview."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: PMC-VQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'PMC-VQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: PRBench-Finance qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PRBench-Finance'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: PRBench-Legal qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'PRBench-Legal'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: PresentBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'PresentBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: QwenBlenderBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'QwenBlenderBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: QwenQoderBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenQoderBench'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness. Reporting avg@5 with a 6-hour timeout"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: QwenReactBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenReactBench'; no cost or tokens in the source (score only); footnotes: "Inhouse React project building benchmark using Claude Code as the harness"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: QwenSVGBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenSVGBench'; no cost or tokens in the source (score only); footnotes: "Inhouse SVG code generation benchmark; bilingual (EN/CN), auto-render + multimodal judge; BT/Elo rating."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: QwenSWEBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'QwenSWEBench'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness. Reporting avg@3 with an 8-hour timeout"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: QwenVisualOffice qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'QwenVisualOffice'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: RealWorldQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'RealWorldQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: RecreationBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'RecreationBench'; no cost or tokens in the source (score only); footnotes: "An internal long-horizon application-recreation benchmark"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: RefAdv-S qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'RefAdv-S'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: ScreenSpot Pro qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'ScreenSpot Pro'; no cost or tokens in the source (score only); footnotes: "Scores for Opus4.8 and Fable5 are taken from official system cards." / "The Fable5 results refer to the corresponding Mythos Preview scores." / "All other models are evaluated in-house."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: SimpleVQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'SimpleVQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: SkillsBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'SkillsBench'; no cost or tokens in the source (score only); footnotes: "Opus 4.8 and Fable 5 are evaluated on Claude Code" / "GPT-5.6 Sol is evaluated on Codex" / "the Qwen-series are evaluated on OpenCode. All results are from our own testing."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: SLAKE qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'SLAKE'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: SURDS qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'SURDS'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: SWE-bench Pro qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'SWE-bench Pro'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness, temp=1.0, top_p=0.95, and a 256K context window." / "all baselines evaluated on the refined benchmark"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: Terminal Bench 2.1 qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Terminal Bench 2.1'; no cost or tokens in the source (score only); footnotes: "Claude Opus 4.8 and Claude Fable 5 with Terminus 2 from Artificial Analysis" / "GPT-5.6 Sol with Codex (https://openai.com/index/previewing-gpt-5-6-sol/)" / "For all other models, we report the best published score across harnesses" / "Evaluated with Claude Code (avg@10), using a 5-hour timeout and max_tokens=131,072."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: Toolathlon Verified (Pass@1) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'Toolathlon Verified (Pass@1)'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: TVBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'TVBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: VideoDR (w/ Search) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'VideoDR (w/ Search)'; no cost or tokens in the source (score only); footnotes: "Scores are evaluated with access to a search tool."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: VideoMME (w/ Sub.) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'VideoMME (w/ Sub.)'; no cost or tokens in the source (score only); footnotes: "Scores are evaluated with subtitles enabled."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| blog-qwen3.8: VideoMME v2 (w/ Sub.) qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'VideoMME v2 (w/ Sub.)'; no cost or tokens in the source (score only); footnotes: "Scores are evaluated with subtitles enabled."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: VideoMMMU qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'VideoMMMU'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: VisFactor qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'VisFactor'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: Vision2Web qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'Vision2Web'; no cost or tokens in the source (score only); footnotes: "using the Claude Code harness and gpt-5.4-2026-03-05 as the judge"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: VLMsAreBiased qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'VLMsAreBiased'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: WebArena-Verified qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'WebArena-Verified'; no cost or tokens in the source (score only); footnotes: "Scores are reported using the official WebArena grader within the OSWorld scaffold."; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: WideSearch qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'WideSearch'; no cost or tokens in the source (score only); footnotes: "Evaluated with the Claude Code harness for external models and the Qwen-Agent harness for ours"; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: WorkSpaceBench qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 1, benchmark 'WorkSpaceBench'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: WorldVQA qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'WorldVQA'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: ZeroBench (Pass@5) [with-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'ZeroBench (Pass@5)' [with-CI]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: ZeroBench (Pass@5) [without-CI] qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'ZeroBench (Pass@5)' [without-CI]; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| blog-qwen3.8: ZeroBench-Sub qwen-blog-qwen3.8 | launch blog (text table + multimodal table); saved jina-qwen3.8.md (README/blog text as fetched 2026-10-19; created 2026-08-03); table 2, benchmark 'ZeroBench-Sub'; no cost or tokens in the source (score only); footnotes: none attached; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| Box AI eval · finance box.com | Box blog 28 Sep · financial services subset, Sonnet 5.5 vs 5, score only ✓ | xmodel | Sonnet 5.5·default · Sonnet 5·default |
| Box AI eval · legal box.com | Box blog 28 Sep · legal subset, Sonnet 5.5 vs 5, score only ✓ | xmodel | Sonnet 5.5·default · Sonnet 5·default |
| Box AI eval · life sciences box.com | Box blog 28 Sep · life sciences subset, Sonnet 5.5 vs 5, score only ✓ | xmodel | Sonnet 5.5·default · Sonnet 5·default |
| Box AI eval · public sector box.com | Box blog 28 Sep · public sector subset, Sonnet 5.5 vs 5, score only ✓ | xmodel | Sonnet 5.5·default · Sonnet 5·default |
| Box AI eval (Sonnet 5.5) box.com | Box blog 28 Sep · internal enterprise eval, Sonnet 5.5 vs 5, score only ✓ | xmodel | Sonnet 5.5·default · Sonnet 5·default |
| Braintrust MathTutorBench (Opus 5.5 vs Fable 5.1) braintrust | 175 examples (7 families x 25); problems-solved over the 5 objectively scored families; writing-quality over all 7 via a GLM-5.3-Flash judge; thinking.type=adaptive, output_config.effort=high, max_tokens=8192; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| braintrust25 braintrust | config ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default · Sonnet 5·default |
| braintrust50 braintrust | config ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default · Sonnet 5·default |
| brinvik.com: CRM-cleanup + Q4 sales-forecast test, Sonnet 5.5 vs Opus 5.5 (one run each) brinvik.com | brinvik.com/en/journal/claude-sonnet-5-5-vs-opus-5-5-sales-forecast-test; synthetic 61-deal CRM export, 9 planted errors, build Q4 forecast + Excel workbook + 5 slides; both reached the exact pre-computed answer (DKK 6,529,724 from 27 deals) and caught all 9 planted errors; cost = author's own token-count list-price calculation, n=1 per model; read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| Brood War Bench broodwarbench | Claude Code CLI · round-robin StarCraft:Brood War tournament, 171 games, default effort only ✓ | xmodel | Fable 5.1·default · Opus 5·default · Sonnet 5·default · Haiku 4.5·default |
| BrowseComp (table) openai-gpt55-launch | BrowseComp; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| Browserbase HardBench stagehand.dev | Browserbase / Stagehand browser-agent evals, HardBench, Claude rows in Claude Code · accuracy in %, cost per task (USD); no effort stated (default); task and run counts not stated; read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Opus 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| Build quality /70 (Opus judge), complex builds, private codebase github-coleam00-kimi-k3-reliability | as simple builds but under-specified goals; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| Build quality /70 (Opus judge), simple builds, private codebase github-coleam00-kimi-k3-reliability | Plan+build workflow on a real private codebase, PR judged by Opus on 7-dimension rubric /70; image footnote "Scores are directional ... Treat as estimates"; simple tickets; repo last commit 2026-07-21; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| BullshitBench v1 github-cc-83510 | CC issue 83510 · v1 n=55 detect-strict + median output tokens, 7 couples ✓ | xmodel | Opus 4.8·nothink · Sonnet 4.6·high · Sonnet 5·low · Opus 4.7·max · Sonnet 5·max · Opus 5·xhigh · Opus 5·low · Sonnet 4.6·nothink |
| BVB arxiv-2609.15478 | arXiv 2609.15478 · Mini-BVB via LiteLLM, $3/scene cap, 5 Claude models at high (+ omitted-effort runs), LiteLLM list cost ✓ | xmodel | Opus 5·high · Sonnet 5·high · Sonnet 4.6·high · Opus 4.7·high · Opus 4.8·high · Sonnet 4.6·default · Sonnet 5·default · Opus 4.7·default · Opus 4.8·default · Opus 5.5·high · Opus 5.5·xhigh · Sonnet 5.5·xhigh |
| c0ze/agent-evaluation: 13 tasks, blind-graded (24-way pass of 13 Oct) c0ze-agent-evaluation | 13 self-contained tasks, 1 run per model, blind rubric grading by Opus 5.5 subagents; score = mean of 13 task totals /10; cost per 13-task run (credit-balance drop, whole $; Haiku API-equivalent); read 2026-10-15 ✓ | xmodel | Sonnet 5.5·xhigh · Fable 5.1·xhigh · Opus 5.5·high · Haiku 5.5·xhigh |
| CalorieBench github-caloriebench | Anthropic API · 200 food photos, effort=high sent without thinking → nothink regime, within-20 % score ✓ | xmodel | Fable 5.1·nothink · Opus 5.5·nothink · Sonnet 5·nothink · Haiku 4.5·think-8k |
| cauchye_asia Japanese text + vision, basic set (21 q x 2) zenn-cauchye_asia | paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·high |
| cauchye_asia Japanese text + vision, hard set (10 q x 2) zenn-cauchye_asia | paid API default settings (thinking default; Claude max_tokens=4000); string-match scoring; 14 Oct 2026; cost = tokens x PRICE table; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·high |
| CausalGame Agent mode (structured tool calling): mean survival rate over 14 experiments causalgame-2607.04293 | CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Agent mode (tool calls); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| CausalGame Prompting mode (Python code): mean survival rate over 14 experiments causalgame-2607.04293 | CausalGame (30 models, 14 game experiments with causal traps; antenna/deployment-zone/weather SCMs), Table 16 mean survival rate % over the 14 experiments (95% CI over runs), 3 runs, Prompting mode (Python code); the other mode is a separate group; arXiv 2607.04293 (Jul 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| CEO-Bench arxiv-2606.18543 | terminal-agent · Opus/Sonnet=MAX, Haiku=thinking ✓ | xmodel | Opus 4.8·max · Opus 4.7·max · Sonnet 4.6·max · Haiku 4.5·solo · Fable 5·max · Sonnet 5·max |
| cfg-bash computingforgeeks | config ✓ | xmodel | Opus 5·default · Opus 4.8·default |
| cfg-k8s computingforgeeks | config ✓ | xmodel | Opus 5·default · Opus 4.8·default |
| cfg-terraform computingforgeeks | config ✓ | xmodel | Opus 5·default · Opus 4.8·default |
| ChaosCypher context-extraction: carrier suite (N=21) github-chaoscypher | War & Peace fixture, mcp:claude-code harness, carrier suite, pass/fail verdicts; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Sonnet 5·high |
| ChaosCypher context-extraction: chat suite (N=80) github-chaoscypher | War & Peace fixture, mcp:claude-code harness, chat suite, pass/fail verdicts; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Sonnet 5·high |
| ChaosCypher context-extraction: native suite (N=65) github-chaoscypher | War & Peace fixture, mcp:claude-code harness, native suite, pass/fail verdicts; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Sonnet 5·high |
| Chartography (Surge AI) launch snapshot July 2026: cells absent from or different in the held board chartography-2608.10677 | Chartography launch leaderboard (arXiv 2608.10677, frozen July 2026 snapshot; 100 tasks x 20 trials, no tools, mean pass@1 % with 95% CI, Surge-hosted): only the cells that are NOT identical to the held surgehq.ai board (read 2026-10-15, group surge-chartography) are kept; join with that group for the max/xhigh peers; arXiv 2608.10677 (Aug 2026); read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.7·default · Sonnet 5·default · Opus 4.8·default |
| Chess puzzle accuracy, base prompt (imec APO paper) imec-2610.00416 | 559-puzzle test set, base prompt (no optimisation), mean of 3 runs, generative single-move answer, legal-move list given, no tools; metric puzzle accuracy %; cost = USD per 1,000 solver-position calls; arXiv 2610.00416 Table 2 (Oct 2026); read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo |
| ChronoBay watch reference identification from one photo (QutaibaAbualrob/chronobay_ai_bench) QutaibaAbualrob/chronobay_ai_bench | REPORT.md 2026-09-30, 100 images, 1 pass/image, same prompt (sha 4e521b10...); score is exact-reference-number match %; cost is per-image USD computed by the repo from its own token counts and a pinned price table (list-price-cost); read 2026-10-01 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·nothink · Haiku 4.5·solo |
| Claude Code Camp 12-task claudecodecamp | beehiiv newsletter · Claude Code, default effort, 12 tasks×3 reps, cost+tokens only ✓ | xmodel | Opus 5·default · Opus 5.5·default |
| Claude Code研究所: 3 paperwork tasks, Sonnet 5.5 vs Opus 5.5 (medium) note-claudecode_lab | note.com/claudecode_lab/n/n960c8fc026e6; table read from src/notes/cclab_table.png (image, opened and read directly); claude -p --model ... --effort medium --max-turns 20, Claude Code 2.1.286, costUSD=total_cost_usd (API list-price); n=1 per cell; cache-read % given per cell; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·medium · Opus 5.5·medium |
| claude-eco review baseline github-claude-code-eco | Claude Code 2.1.233 · one code-review task n=5, 5 models at CC default effort, grader re-run on raw answers (3 issues), CLI cost (Sonnet ×2/3) ✓ | xmodel | Opus 5·default · Sonnet 5·default · Opus 4.8·default · Fable 5·default · Haiku 4.5·solo |
| claude-haiku-terraform-benchmark: 3 Terraform tasks scored /100 (fmt, validate, tflint, checkov, coverage, conventions), Bedrock github-terraform-haiku-bench | moriryota62/claude-haiku-terraform-benchmark results/report.md (latest run, 11 Oct 2026); score = mean of 3 task scores; cost = total of the 3 generations at Bedrock regional price (list x1.1); read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Sonnet 5.5·default · Haiku 5.5·default |
| claude-maintain-rails-evals: frozen planner benchmark, Rails case (23 expected findings), 5 reps github-claude-maintain-rails-evals | andreimaxim/claude-maintain-rails-evals tests/results/20261012; planner plugin run in Amp threads; score = mean final_actionable findings (of 23) over the repetitions with a plan; cost = mean list-price estimate of recorded requests (complete estimates only); read 2026-10-15 ✓ | xmodel | Opus 5.5·high · Opus 5.5·low · Sonnet 5.5·medium · Haiku 5.5·medium |
| claude-maintain-rails-evals: planner variant v12 (release notes), Claude headless, 5 runs/config github-claude-maintain-rails-evals | claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high |
| claude-maintain-rails-evals: planner variant v16 (Argon clean-room procedure), Claude headless, 5 runs/config github-claude-maintain-rails-evals | claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·high |
| claude-maintain-rails-evals: risk-report variant v20 (changes-first), Opus 5.5 high vs low, 5 runs/config github-claude-maintain-rails-evals | claude -p --plugin-dir via scripts/run-worker.ts; Rails 8.1 upgrade case (23 planted findings); 5 runs per configuration; whole session on the stated model; graded by Claude Opus 5.5 high; median cost per run (grading cost excluded), mean found per run; NEW GROUP (held rails-planner-frozen is the frozen Amp+planner-plugin configuration of 20261012); read 2026-10-18 ✓ | xmodel | Opus 5.5·high · Opus 5.5·low |
| claude.dev hillclimb claude-dev-blog | claude.dev hillclimb post · 30 support tickets, audited prompt, Opus 5.5 vs Sonnet 5 at low, cost per ticket ✓ | xmodel | Opus 5.5·low · Sonnet 5·low |
| Claw-Bench V1 (Hermes) claw-bench.com | Claw-Bench V1, agent tasks in the Hermes harness, 153 tasks · reward in %; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| Claw-Eval-Live claw-eval-live.github.io | Claw-Eval-Live, 105 enterprise-workflow tasks · pass rate in %, average score and tokens in the flags; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Sonnet 4.6·default |
| clawarena arxiv-2606.31174 | config ✓ | xmodel | Fable 5·default · Sonnet 4.6·default |
| ClawProBench open dataset 102 tasks (OpenClaw) clawprobench-board | ClawProBench (suyoumo.github.io/bench), open dataset 102 tasks, OpenClaw agent runtime v3.2.6, deterministic grading, repeated trials; Final Score 0-100 (page default rank signal); cost = page cost_usd = (in + 0.5 x (cacheread + cachewrite)) x price_in + out x price_out; rows updated 2026-02..2026-07 (live board); one route kept per couple (earliest updated); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| Clerk evals (baseline) clerk.com | Clerk auth-integration evals, 17 framework × category cells, baseline · mean cell score in % (computed by us from the published cells), cost summed over the 17 cells (USD, the runner's tokens × list price), tokens; no reasoning option set (default); the 5.5 models ran on 28–29 Sep, the others on 3 Sep; read 2026-10-01 ✓ | xmodel | Fable 5.1·default · Fable 5·default · Opus 5.5·default · Opus 5·default · Sonnet 5·default · Sonnet 5.5·default |
| Clerk evals (with MCP) clerk.com | Clerk auth-integration evals, 17 framework × category cells, with MCP · mean cell score in % (computed by us from the published cells); no cost; no reasoning option set (default); the 5.5 models ran on 28–29 Sep, the others on 3 Sep; read 2026-10-01 ✓ | xmodel | Fable 5.1·default · Fable 5·default · Opus 5.5·default · Opus 5·default · Sonnet 5·default · Sonnet 5.5·default |
| Clerk evals (with skills) clerk.com | Clerk auth-integration evals, 17 framework × category cells, with skills · mean cell score in % (computed by us from the published cells); no cost; no reasoning option set (default); the 5.5 models ran on 28–29 Sep, the others on 3 Sep; read 2026-10-01 ✓ | xmodel | Fable 5.1·default · Fable 5·default · Opus 5.5·default · Opus 5·default · Sonnet 5·default · Sonnet 5.5·default |
| ClerkBench banded_meter_billing (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload zenodo-23048151 | Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Opus 5·high |
| ClerkBench fee_ledger_dropzero (Zenodo 23048151) — pass@1 at L=100, deterministic clerical workload zenodo-23048151 | Table 6 pass@1 at L=100, Table 2 $/attempt (L=100, mean list-price cost incl. failures); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Opus 5·high |
| cliphou.se Motion Graphics Benchmark v1.1: cost per run (Remotion videos) cliphou.se | 3 briefs x 2 runs via OpenRouter, reasoning high; cost = mean OpenRouter generation cost per run over the 6 runs (computed by us); no quality score published (human review pending); read 2026-10-15 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high |
| clopy TypeScript task: 6-file validateCents refactor zenn-clopy | claude -p --effort medium requested; Read/Edit/Write only; 3 runs each; independent host-side grading; 10 Oct 2026; subscription; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Sonnet 5.5·medium |
| clopy TypeScript task: add tests (6 fixed mutations) zenn-clopy | claude -p --effort medium requested; Read/Edit/Write only; 3 runs each; independent host-side grading; 10 Oct 2026; subscription; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Sonnet 5.5·medium |
| clopy TypeScript task: decimal amount parse fix zenn-clopy | claude -p --effort medium requested; Read/Edit/Write only; 3 runs each; independent host-side grading; 10 Oct 2026; subscription; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Sonnet 5.5·medium |
| Code-as-Policy / RoboDojo (2609.39018) — 5 simulated manipulation tasks, +URAI interface arxiv-2609.39018 | Table 1, avg success rate % over 5 tasks x 5 episodes, output tokens k incl. reasoning; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Fable 5.1·xhigh · Opus 5.5·xhigh |
| Code-as-Policy / RoboDojo (2609.39018) — 5 simulated manipulation tasks, native interface arxiv-2609.39018 | Table 1, avg success rate % over 5 tasks x 5 episodes, output tokens k incl. reasoning; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Fable 5.1·xhigh · Opus 5.5·xhigh |
| code-review arxiv-2606.15689 | VibeOps · temp 0.1, thinking NS ✓ | xmodel | Haiku 4.5·nothink · Sonnet 4.6·nothink |
| code-review (synthetic, n=100) arxiv-2606.15689 | arXiv 2606.15689 T4 · VibeOps prompt, score only ✓ | xmodel | Haiku 4.5·nothink · Sonnet 4.6·nothink |
| code-review (synthetic, n=25) arxiv-2606.15689 | arXiv 2606.15689 T4 · VibeOps prompt, score only ✓ | xmodel | Haiku 4.5·nothink · Sonnet 4.6·nothink |
| code-review-bench · /ce-code-review github-codereviewbench | kamui/code-review-bench 23716f7 · 12 OSS PRs × 3 trials, same skill snapshot, Claude Code 2.1.284 high, recall and cohort cost ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| code-review-bench · /thermo review github-codereviewbench | kamui/code-review-bench 23716f7 · 12 OSS PRs × 3 trials, same skill snapshot, Claude Code 2.1.284 high, recall and cohort cost ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| code-review-bench 5 selected tasks: Claude Code built-in /code-review, Opus 5.5 vs Fable 5.1 vs Sonnet 5.5 (high), 2026-10-02 github-codereviewbench | kamui/code-review-bench 2026-10-02 'selected' cohort: 5 PR tasks x 3 repetitions, claude-code 2.1.284, rubric v2, blinded Claude Opus 5.5 High grader; cost = cost_contemporaneous_usd (list-price-equivalent subscription usage, failed attempts included); read 2026-10-18 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Fable 5.1·high |
| code-review-bench 5 selected tasks: thermo-nuclear-code-quality-review, Opus 5.5 vs Sonnet 5.5 (high), 2026-10-02 github-codereviewbench | kamui/code-review-bench 2026-10-02 'selected' cohort: 5 PR tasks x 3 repetitions, claude-code 2.1.284, rubric v2, blinded Claude Opus 5.5 High grader; cost = cost_contemporaneous_usd (list-price-equivalent subscription usage, failed attempts included); read 2026-10-18 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high |
| code-review-bench: Claude 5.5 as claim labeller (12 batches, 493 claims), agreement with GPT-6.1 Sol labels github-codereviewbench | kamui/code-review-bench grader-model-test-2026-10-12; 12 sessions (12 batches of 53 reviews; 493 claims); Claude Code high; score = label agreement with the saved GPT-6.1 Sol labelling (not accuracy); read 2026-10-15 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high |
| Code4Scene public set (95 cases), overall S_model arxiv-2609.36777 | Code4Scene: 190 Unreal Engine cases (Text-to-Scene + Image-to-Scene), coding agents with 25 shared tools and a metering proxy; score in [0,1], S_model weights T2S and I2S equally; a second attempt only after a failed first; cost exists only in Figure 1(b) (not extracted: no cost); Sep 2026; public set (95 cases), overall S_model; read 2026-10-19 ✓ | xmodel | Fable 5.1·max · Opus 5·max |
| CodeActionBench, Reference harness (single code-generation loop) arxiv-2609.33807 | CodeActionBench (arXiv 2609.33807, 27 Sep 2026): 25 tasks x 3 seeds = 75 attempts per configuration, agentic Code-as-Policy for embodied manipulation; 'Reference' harness for all rows of this group; score = Success (/75 attempts); task coverage /25 is another summary; cost = Table 11 total API-equivalent USD over the 75 attempts (computed by the authors from tokens at list rates); read 2026-10-20 ✓ | xmodel | Opus 5·high · Sonnet 5·high |
| CodeActionBench, vendor harnesses (Codex CLI / Claude Code) arxiv-2609.33807 | CodeActionBench: same 75 attempts; vendor harness rows (GPT-6 Astra in Codex CLI, Claude Opus 5 in Claude Code); Success (/75); cost = Table 11 total API-equivalent USD; read 2026-10-20 ✓ | xmodel | Opus 5·high |
| CodeRabbit OSS-Aug (Sonnet 5.5) coderabbit | CodeRabbit blog 28 Sep · 44 PRs, Sonnet 5.5 vs Sonnet 5, cost only (judge pending) ✓ | xmodel | Sonnet 5.5·default · Sonnet 5·default |
| CodeRabbit review (Fable 5.1) coderabbit | CodeRabbit blog 1 Sep · 45 tasks/105 issues, Fable 5.1 low/high + Fable 5, Opus 5, score only ✓ | xmodel | Fable 5.1·low · Fable 5.1·high · Fable 5·default · Opus 5·default |
| CodeRabbit Signal coderabbit | CodeRabbit blogs 22 & 28 Sep · 13 hard patterns, frozen cassette · Opus 5.5 Standard/Max score only; Sonnet 5.5 thinking on/off and Sonnet 5 with measured cost ✓ | xmodel | Opus 5.5·default · Opus 5.5·max · Sonnet 5.5·default · Sonnet 5.5·nothink · Sonnet 5·default |
| CodeRouterBench (in-dist.) arxiv-2606.22902 | arXiv 2606.22902 T9 · single-turn, 9 dimensions, score only ✓ | xmodel | Sonnet 4.6·default |
| CodeSOTA codesota-openrouter | list-price blended (not a run) ✓ | xmodel | Sonnet 4.6·priceblend · Opus 4.8·priceblend |
| Codex Radar intelligence-efficiency board, Codex CLI harness codexradar | Codex Radar recurring repository-level coding eval (per-task pass over valid tasks; payload mode equal_latest_3: equal-weight latest three valid samples per task); harness codex; snapshot payload stamped 2026-10-01T23:00:27+08:00 read 2026-10-20; score=passed/valid_tasks (derived; payload gives passed, valid_tasks, iq=150*pass_rate); cost=average_price_usd per run (estimated USD-equivalent from token prices); valid_tasks differ per row (36..336); read 2026-10-20 ✓ | xmodel | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·xhigh · Opus 5.5·max |
| codrouter-ood arxiv-2606.22902 | config ✓ | xmodel | Sonnet 4.6·default |
| computingforgeeks: Claude Code bug-fix task (log parser) computingforgeeks | 4-bug Python log parser + 8-test pytest suite, headless Claude Code, 3 runs/model, default effort per model; score saturated (9/9 green) -> cost only; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·medium · Opus 5.5·medium · Sonnet 5·high |
| Conceptual Reasoning Index · DTBench conceptualreasoning.ai | Conceptual Reasoning Index DTBench sub-score (CRI-rescaled accuracy, x100), cost = average USD per response (site cost chart); effort from the csv effort column; blank = default; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·medium · Fable 5.1·max · Opus 5·xhigh · Fable 5.1·xhigh · Opus 5·high · Fable 5.1·high · Opus 5·max · Fable 5·max · Opus 5·medium · Fable 5.1·low · Fable 5·xhigh · Opus 5·low · Fable 5·high · Fable 5·medium · Fable 5·low · Opus 4.8·default · Opus 4.7·default · Sonnet 5·max · Sonnet 5·xhigh · Sonnet 5·high · Sonnet 4.6·default · Sonnet 5·medium · Sonnet 5·low · Haiku 4.5·solo · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Sonnet 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low · Haiku 5.5·nothink |
| Conceptual Reasoning Index · LMCA conceptualreasoning.ai | Conceptual Reasoning Index LMCA sub-score (Pearson-r based, x100), cost = average USD per response (site cost chart); effort from the csv effort column (max rows carry the base name); blank = default; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·medium · Fable 5.1·max · Opus 5·xhigh · Fable 5.1·xhigh · Opus 5·high · Fable 5.1·high · Opus 5·max · Fable 5·max · Opus 5·medium · Fable 5.1·low · Fable 5·xhigh · Opus 5·low · Fable 5·high · Fable 5·medium · Fable 5·low · Opus 4.8·default · Opus 4.7·default · Sonnet 5·max · Sonnet 5·xhigh · Sonnet 5·high · Sonnet 4.6·default · Sonnet 5·medium · Sonnet 5·low · Haiku 4.5·solo · Opus 5.5·xhigh · Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Sonnet 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5.5·low · Haiku 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·high · Haiku 5.5·medium · Haiku 5.5·low · Haiku 5.5·nothink |
| Context Arena 8-needle retrieval, 4K-128K bins (the five bins every model has) contextarena.ai | contextarena.ai API needles=8 mode=full, read 2026-10-20; score = sample-weighted mean of the five bins 4K-128K that all models share (models with more bins, up to 1M, are cut to these so the context range is the same); bins recomputed from the payload; reasoning_mode as the board labels it; read 2026-10-20 ✓ | xmodel | Opus 5·max · Opus 4.8·max · Sonnet 4.6·high · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 5·max · Sonnet 4.6·default · Haiku 4.5·solo · Opus 4.7·medium · Opus 4.7·low · Opus 4.7·xhigh · Opus 4.7·high · Opus 4.7·default |
| Convex evals (no guidelines, web research) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines, web research' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Opus 5.5·medium · Fable 5.1·medium · Opus 5·medium · Sonnet 5.5·medium · Opus 4.8·medium · Sonnet 5·medium |
| Convex evals (no guidelines) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'no guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Opus 5.5·medium · Fable 5·medium · Sonnet 5.5·medium · Opus 4.8·medium · Opus 5·medium · Fable 5.1·medium · Sonnet 5·medium |
| Convex evals (with guidelines) convex.dev | Convex coding evals, 112 evals in 8 categories, experiment 'with guidelines' · total score in % with its error bar, billed cost per full run (USD, OpenRouter); effort medium from the runner's OpenRouter request (HEAD config, assumed for the runs); read 2026-10-01 ✓ | xmodel | Sonnet 5.5·medium · Fable 5·medium · Fable 5.1·medium · Opus 5.5·medium · Opus 5·medium · Opus 4.8·medium · Sonnet 5·medium |
| Convex LLM leaderboard v92a60e0b · no guidelines convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines; read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Fable 5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Opus 4.8·medium · Haiku 5.5·medium · Sonnet 5·medium |
| Convex LLM leaderboard v92a60e0b · no guidelines + web convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, no guidelines + web; read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Fable 5·medium · Fable 5.1·medium · Sonnet 5.5·medium · Opus 4.8·medium · Opus 5·medium · Sonnet 5·medium · Haiku 5.5·medium |
| Convex LLM leaderboard v92a60e0b · with guidelines convex.dev | Convex evals (112), benchmark version 92a60e0b (guidelines compressed 7010→4146 tokens), OpenRouter, reasoning effort medium, with guidelines; read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium · Fable 5·medium · Fable 5.1·medium · Opus 5·medium · Opus 4.8·medium · Haiku 5.5·medium · Sonnet 5·medium |
| costrouter github-costrouter | config ✓ | xmodel | Opus 5·low · Haiku 4.5·solo |
| CRBugBench post-merge bugs (Fix track), recall@5 arxiv-2610.13668 | 500 review tasks per track from 98 OSS repos, copilot-cli 1.0.74 official /review prompt, default prompt; Table 1 of arXiv 2610.13668; read 2026-10-15 ✓ | xmodel | Opus 5·xhigh · Opus 4.8·xhigh · Sonnet 5·xhigh · Haiku 4.5·solo |
| CRBugBench pre-merge bugs (Cmt track), recall@5 arxiv-2610.13668 | 500 review tasks per track from 98 OSS repos, copilot-cli 1.0.74 official /review prompt, default prompt; Table 1 of arXiv 2610.13668; read 2026-10-15 ✓ | xmodel | Opus 5·xhigh · Opus 4.8·xhigh · Sonnet 5·xhigh · Haiku 4.5·solo |
| CritPt (GLM-5.2 card) zai-hf-glm-5.2 | CritPt, temperature=1.0 top_p=0.95 163,840 tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| ctala github-ctala | Claude Code CLI · reasoning not configured, temp 0.7 ✓ | xmodel | Opus 4.8·default · Opus 4.7·default · Sonnet 5·default · Fable 5·default · Opus 5·default · Sonnet 4.6·default · Haiku 4.5·solo · Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default |
| ctala Claude Code route github-ctala | ctala models.json · Claude Code subscription route, 7 models, unranked (exam incomplete), list-price per assumed call ✓ | xmodel | Sonnet 5·default · Fable 5·default · Haiku 4.5·solo · Opus 5·default · Opus 4.8·default · Sonnet 4.6·default · Opus 4.7·default |
| CTE-Bench core-v1 all-call VM, Answers revealed arxiv-2609.36647 | CTE-Bench core-v1 255 scenarios, condition Answers revealed; VM=exact response match over all future calls (Eff.VM/SM same runs not used); paper 2026-09-28; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| CTE-Bench core-v1 all-call VM, Free rollout arxiv-2609.36647 | CTE-Bench core-v1 255 scenarios, condition Free rollout; VM=exact response match over all future calls (Eff.VM/SM same runs not used); paper 2026-09-28; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| CTE-Bench core-v1 all-call VM, No feedback arxiv-2609.36647 | CTE-Bench core-v1 255 scenarios, condition No feedback; VM=exact response match over all future calls (Eff.VM/SM same runs not used); paper 2026-09-28; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| CTE-Bench reasoning-mode probe, 24 delayed-effect scenarios, answers revealed, all-call VM arxiv-2609.36647 | CTE-Bench 24 delayed-effect scenarios, answers revealed (Table 10); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default · Sonnet 4.6·solo |
| CTI-REALM unconstrained (300-message budget): overall reward ctirealm-2610.14180 | Unconstrained CTI-REALM (Averlon), Table 2: 50 tasks, one sample each, common harness, judges Gemma-4-31B thinking / DeepSeek-v4-flash / GLM-5.3-flash x3 trials, message budget 300 (14 models have samples run at 300 incl. 10 spliced, 12 carried from lower budgets with zero capped samples); overall weighted reward; checkpoints C0-C4 same runs (rule 8); cost = cost of the 50-task run at list prices; arXiv 2610.14180 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 5·default · Sonnet 5·default · Haiku 4.5·nothink |
| CUA-SWE devops code: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, devops domain (20 tasks), deterministic executable verifiers, access = code-only; frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default · Fable 5·default |
| CUA-SWE devops hybrid: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, devops domain (20 tasks), deterministic executable verifiers, access = hybrid CUA (screenshots + graphical actions); frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default · Fable 5·default |
| CUA-SWE game code: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, game domain (29 tasks), deterministic executable verifiers, access = code-only; frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default · Fable 5·default |
| CUA-SWE game hybrid: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, game domain (29 tasks), deterministic executable verifiers, access = hybrid CUA (screenshots + graphical actions); frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default · Fable 5·default |
| CUA-SWE mobile code: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, mobile domain (20 tasks), deterministic executable verifiers, access = code-only; frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 4.8·high · Sonnet 5·high · Fable 5·high |
| CUA-SWE mobile hybrid: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, mobile domain (20 tasks), deterministic executable verifiers, access = hybrid CUA (screenshots + graphical actions); frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 4.8·high · Sonnet 5·high · Fable 5·high |
| CUA-SWE web code: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, web domain (36 tasks), deterministic executable verifiers, access = code-only; frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default · Fable 5·default |
| CUA-SWE web hybrid: task success % cua-swe-2609.32600 | CUA-SWE Table 8: single attempt per task, web domain (36 tasks), deterministic executable verifiers, access = hybrid CUA (screenshots + graphical actions); frontier-model runs only (Codex+GPT-5.6-Sol and Claude Code+Opus-5 systems are another harness: excluded); pass@3 for 3 models = same runs; arXiv 2609.32600; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default · Fable 5·default |
| CWE-bench (120 vulnerability-repair tasks, 4 attempts) arxiv-2610.17020 | Claude Code, high reasoning effort, 4 attempts per task, 1 h limit; programmatic pass@1; cost = mean API cost per rollout; read 2026-10-15 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| CWE-Bench (Collinear) pass@1 vs average cost per rollout, Google Pareto chart (2 Sep 2026) google-blog-3-8-cyber | blog.google 3.8 Flash Cyber chart; pass@1 + cost per rollout; pass digitised except the 3.8 Cyber score (text 47.2) and Fable 5 (text 47.8 = leading frontier model); cost digitised; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| CyberGym (table) openai-gpt55-launch | CyberGym; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| CyberGym (table) openai-gpt56-launch | CyberGym; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| CyberGym pass@1 (GLM-5.3 card) zai-hf-glm-5.3 | CyberGym 1507 tasks single-run pass@1, Claude Code 2.1.207, no web tools, unlimited timeout; only GLM-5.3 stated as evaluated; GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| D2K-Bench (26 GPU-kernel tasks), with design guidance arxiv-2610.03226 | D2K-Bench (arXiv 2610.03226, 2 Oct 2026): 26 GPU-kernel tasks, agents in Claude Code CLI 2.1.197 (Opus 4.8, GLM-5.3, Kimi-K3) or Codex CLI 0.147.0 (GPT-6-Astra, GPT-5.6-Sol), with design guidance; score = S_perf (speedup score relative to Fable-5's first correct submission without guidance; Table 2); correct-task count is another column; no cost; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| D2K-Bench (26 GPU-kernel tasks), without design guidance arxiv-2610.03226 | D2K-Bench (arXiv 2610.03226, 2 Oct 2026): 26 GPU-kernel tasks, agents in Claude Code CLI 2.1.197 (Opus 4.8, GLM-5.3, Kimi-K3) or Codex CLI 0.147.0 (GPT-6-Astra, GPT-5.6-Sol), without design guidance; score = S_perf (speedup score relative to Fable-5's first correct submission without guidance; Table 2); correct-task count is another column; no cost; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| dahatake · spreadsheet engine qiita-dahatake | qiita 30 Sep · GitHub Copilot CLI, --reasoning-effort medium, spreadsheet engine, n=3, hidden tests, cost in Copilot credits ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| dahatake · SQL engine, prompt C1 qiita-dahatake | qiita 30 Sep · GitHub Copilot CLI, --reasoning-effort medium, SQL engine, prompt C1, n=3, hidden tests, cost in Copilot credits ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| dahatake · SQL engine, prompt C1S qiita-dahatake | qiita 30 Sep · GitHub Copilot CLI, --reasoning-effort medium, SQL engine, prompt C1S, n=3, hidden tests, cost in Copilot credits ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| Dante (Korean YouTube): 40 shopping-review classification (sentiment x topic) youtube-dante-ko | Claude Code default settings; recorded run (3 runs elsewhere); cost = Claude Code token counts x list price at 1 USD = 1,400 KRW; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo · Sonnet 5.5·default · Opus 5.5·default |
| Dante (Korean YouTube): bookkeeping program, 8 requirements, 22 hidden tests youtube-dante-ko | Claude Code default settings; recorded run; cost = Claude Code tokens x list price, KRW; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo · Sonnet 5.5·default · Opus 5.5·default |
| Data Science Tasks (Internal) (OpenAI GPT-6 Astra post chart) openai-gpt6-astra-launch | Data Science Tasks (Internal); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·max · Opus 5·max · Fable 5·max |
| DataSpace-Agent backbones: Task Accuracy (410 tasks) dataspace-2608.03451 | DataSpace Table 3/11: 410 tasks (documents/tables/video workspaces, tabular-output contract), fixed light ReAct DataSpace-Agent, backbones released Apr-Jul 2026 via Vercel AI Gateway (July 2026), 32,768-token output ceiling; cost per task = provider official pricing x tokens (mean over 410 tasks); harness-comparison table uses MiMo-V2.5 (out of scope) and is not taken; arXiv 2608.03451 (Aug 2026); read 2026-10-20 ✓ | xmodel | Sonnet 5·default |
| DAYJOB: 50 healthcare work-product tasks, agentic-judge-graded, Harbor arxiv-dayjob | arxiv.org/html/2610.01306v1 Table 3 (healthcare column) + text (Opus 5.5 cost only, $7.10/attempt, the only config on the cost frontier whose cost is itemized by model; others 'not on the cost frontier' / not itemized); strict pass@1 %, 5 attempts/task; effort-from-config:"adaptive/max" -> max; read 2026-10-05 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max |
| DAYJOB: 80 finance work-product tasks, agentic-judge-graded, Harbor arxiv-dayjob | arxiv.org/html/2610.01306v1 Table 3 (finance column) + text (Opus 5.5 cost only, $11.47/attempt); strict pass@1 %, 5 attempts/task; effort-from-config:"adaptive/max" -> max; read 2026-10-05 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max |
| Dealwatch defect review github-dealwatch | Claude Code orchestrator · 5 re-seeded bugs, Fable 5.1 vs Opus 5.5 @high, 1 run, list-price cost ✓ | xmodel | Fable 5.1·high · Opus 5.5·high |
| Dealwatch small feature github-dealwatch | same repo · 6-req feature, hidden tests saturated → cost only ✓ | xmodel | Fable 5.1·high · Opus 5.5·high |
| Debug Loop (agentic fix of a failing pytest, Opus 4.8 vs GLM-5.2) github-dvallslanaquera-glm5.2-benchmark | same repo results/debug_loop_latest.json generated 2026-06-21: real bug commit d11ae62, tools list_dir/read_file/grep/edit_file/run_tests, step budget 15, 5 trials/model; score = pass@1; cost = mean cost per trial (list price); n=5 per model; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| deepseek-v4-flash-0731-card: ale deepseek-v4-flash-0731-card | Agents' Last Exam; DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: not stated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| deepseek-v4-flash-0731-card: autobench deepseek-v4-flash-0731-card | AutomationBench (Public); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-flash-0731-card: cybergym deepseek-v4-flash-0731-card | CyberGym; DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| deepseek-v4-flash-0731-card: deepswe deepseek-v4-flash-0731-card | DeepSWE (v1.1 per later cards); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-flash-0731-card: dsbench-fs deepseek-v4-flash-0731-card | DSBench-FullStack (DeepSeek internal set); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: not stated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-flash-0731-card: dsbench-hard deepseek-v4-flash-0731-card | DSBench-Hard (DeepSeek internal set); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-flash-0731-card: nl2repo deepseek-v4-flash-0731-card | NL2Repo; DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| deepseek-v4-flash-0731-card: tb21 deepseek-v4-flash-0731-card | Terminal-Bench 2.1 (Pass@1); DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-flash-0731-card: toolathlon-v deepseek-v4-flash-0731-card | Toolathlon-Verified; DeepSeek model card of 2026-07-31; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-flash-vision-card: apexbench deepseek-v4-flash-vision-card | ApexBench Pass@1; DeepSeek model card of 2026-08-31; n/attempts and judge not stated; harness: not stated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| deepseek-v4-flash-vision-card: chartography deepseek-v4-flash-vision-card | Chartography; DeepSeek model card of 2026-08-31; n/attempts and judge not stated; harness: not stated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| deepseek-v4-flash-vision-card: zerobench deepseek-v4-flash-vision-card | ZeroBench Pass@5; DeepSeek model card of 2026-08-31; n/attempts and judge not stated; harness: not stated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| deepseek-v4-pro-0813-card: cybergym deepseek-v4-pro-0813-card | CyberGym; DeepSeek model card of 2026-08-13; n/attempts and judge not stated; harness: DeepSeek Harness minimal mode for DeepSeek rows of code-agent/text-agent tasks, others unstated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-pro-0813-card: hle deepseek-v4-pro-0813-card | HLE without tools; DeepSeek model card of 2026-08-13; n/attempts and judge not stated; harness: not stated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v4-pro-0813-card: hle-tools deepseek-v4-pro-0813-card | HLE with tools; DeepSeek model card of 2026-08-13; n/attempts and judge not stated; harness: not stated; card values at max reasoning effort only where the footnote says so; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| deepseek-v41-flash-1017: apexbench-v11 deepseek-v41-flash-1017 | Apex Bench v1.1; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·default |
| deepseek-v41-techreport: ale deepseek-v41-techreport | Agents' Last Exam (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold(ALE-CLI); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: autobench deepseek-v41-techreport | AutomationBench v1.0.6 public set (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness official-scaffold; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: babyvision deepseek-v41-techreport | BabyVision w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: chartography deepseek-v41-techreport | Chartography w/ tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: exploitgym deepseek-v41-techreport | ExploitGym (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: hle deepseek-v41-techreport | HLE (Pass@1), full set; DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness none(reasoning); all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: hle-tools deepseek-v41-techreport | HLE with tools (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness unstated; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: nl2repo deepseek-v41-techreport | NL2Repo-Bench (Score); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: programbench deepseek-v41-techreport | ProgramBench (Almost@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: tb30 deepseek-v41-techreport | Terminal-Bench 3.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: tb40 deepseek-v41-techreport | Terminal-Bench 4.0 (Pass@1); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness DeepSeek-Harness-minimal; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| deepseek-v41-techreport: zerobench deepseek-v41-techreport | ZeroBench-main w/ tools (Pass@5); DeepSeek V4.1-Flash tech report / model card; sampling stated by the card: temperature=1.0, top_p=0.95 (report 5.3.1: temperature and top-p 1.0 for reasoning benchmarks); N=8 samples per task for DeepSWE v1.1, N=3 for Terminal-Bench 2.1 (Table 4 note); judge not stated; harness Claude-Code; all columns at max effort for report/card E; 1017 card effort not stated; read 2026-10-19 ✓ | xmodel | Opus 5·max |
| DeepSWE (GLM-5.2 card, 2h timeout) zai-hf-glm-5.2 | DeepSWE official pier framework, mini-swe-agent, temperature=1.0 timeout=2h 400K context, 2 CPUs 8GB; GLM-5.2 card (other setup than GLM-5.3 card: 6h, temp 0.95); read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| DeepSWE (OpenAI GPT-6 Sol and Luna post) openai-gpt6-sol-luna-launch | DeepSWE; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| DeepSWE 1.0 (xAI Grok 4.5 post) xai-grok-4-5-launch | DeepSWE 1.0 pass@1 %; footnote Eval created by Datacurve, run with each model provider's harnesses by AA; xAI post 16 Jul 2026; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max · Opus 4.7·max |
| DeepSWE v1.1 (GLM-5.3 card) zai-hf-glm-5.3 | DeepSWE v1.1, mini-swe-agent, temperature=0.95 top_p=1.0 timeout=6h 400K context; GLM-5.3 card 2026-10-14; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| DeepSWE v1.1 (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | DeepSWE v1.1; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·nothink |
| DeepSWE v1.1 (xAI Grok 4.7 post) xai-grok-4-7-launch | DeepSWE v1.1, resolve rate %, harness/n/attempts not stated; xAI Grok 4.7 launch post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Fable 5.1·max |
| Deloitte code review anthropic-partner-deloitte | Opus 5.5 launch page partner quote · bugs caught, score only, n unstated ✓ | xmodel | Opus 5.5·low · Opus 5·high |
| dentsu-b dentsusoken | config ✓ | xmodel | Sonnet 5·high · Opus 5·high · Fable 5·high |
| dentsu-d dentsusoken | config ✓ | xmodel | Sonnet 5·high · Opus 5·high · Fable 5·high |
| Design Arena - all categories designarena | designarena.ai/leaderboard 'allcategories': crowd human-vote Elo on AI-generated designs/UI (1,817,484 votes, 180 models); effort not in the Claude slugs (other vendors' slugs carry -max/-high) so default; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Fable 5.1·default · Opus 5·default · Fable 5·default · Haiku 5.5·default · Sonnet 4.6·default · Sonnet 5·default · Opus 4.8·default · Haiku 4.5·solo |
| Design Arena - fullstack designarena | designarena.ai/leaderboard 'fullstack' category: crowd human-vote Elo on full-stack app generation (61,252 votes, 49 models); effort not stated; no cost; read 2026-10-15 ✓ | xmodel | Fable 5.1·default · Opus 5·default · Fable 5·default · Sonnet 5·default · Opus 4.8·default · Sonnet 4.6·default |
| Design Creativity Bench (UI generation; overall T-score, median cost per design) arxiv-2610.11539 | 80 UI design tasks / 160 prompts, one generation per design, highest effort (Opus 5.5 at xhigh); Tables 2 and 9 of arXiv 2610.11539; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Opus 5.5·xhigh |
| Design Tasks (Internal) (OpenAI GPT-6 Astra post chart) openai-gpt6-astra-launch | Design Tasks (Internal); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·max |
| DevelopersIO Project Euler 1012 (hard), Claude 5 family + Haiku 5.5 effort ladder classmethod | claude -p --effort medium (trio) / Haiku 5.5 five efforts x 2 runs; Bash/Read/Write/Edit/WebFetch; cost = Claude Code estimate; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·medium · Opus 5.5·medium · Fable 5.1·medium · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| DexHoldem (2605.18727v2) — 36 agentic perception problems, Claude Code, avg of 3 runs arxiv-2605.18727 | Table 2, overall strict % and per-category average column; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5.5·medium · Sonnet 5·medium · Opus 4.7·medium · Sonnet 4.6·medium · Haiku 4.5·solo |
| DigBench (agent) digbench.ai | DigBench, games to beat, DigBench's own agent harness · score in % with its interval and runs in the flags; no effort in this table (default); no cost; read 2026-09-30 ✓ | xmodel | Opus 5·default · Opus 4.8·default |
| DigBench (Claude Code) digbench.ai | DigBench, games to beat, Claude Code · score in % with its interval and runs in the flags; no effort in this table (default); no cost; read 2026-09-30 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| digirise/チャエン: 3 prompts (game/summary/earnings), total time/cost, n=1 digirise-chaen | digirise.ai/chaen-ai-lab/claude-opus-5-5/ (cites チャエン's X post 2026-09-23); Claude Code, 1 run per model per prompt, 3 prompts summed; effort-from-config: "Claudeの2モデルはClaude Code経由でeffortはxhigh"; read 2026-10-01 ✓ | xmodel | Opus 5.5·xhigh · Fable 5.1·xhigh |
| Dinobase per-source MCP accuracy (RevOps, 15 questions) dinobase | same questions/models, per-source MCP tools column; 15 per model; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| Dinobase SQL accuracy (RevOps, 15 questions) dinobase | kappa90/dinobase REPORT.md, SQL column, 15 RevOps questions (HubSpot+Stripe), max turns 15, temp 0, OpenRouter; 15 per model per REPORT.md (README says 75: conflict noted); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| DiscoverPhysics 11 worlds (ARA Labs runs) hf-agentnativeresearchlab | AgentNativeResearchLab HF datasets discoverphysics- | xmodel | Fable 5·default · Opus 4.8·max |
| Doc2Harness: pass rate of the harness a model builds arxiv-2610.20344 | Doc2Harness: the build model writes a candidate harness from a development document inside a fixed OpenCode 1.17.0 builder harness; 328 test cases; pass rate = mean over 3 scored builds (+- sd); 19 models; no effort setting and no cost reported; paper 14 Oct 2026; read 2026-10-19 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Opus 4.7·default |
| Double-pendulum job, Haiku 5.5 effort ladder + Sonnet 5.5 medium (AI Receipts video) youtube-aireceipts | 3 runs each; pass = energy drift < 0.1% at 2 simulated min; cost = exact token counts x list price; 2026-10-08; read 2026-10-18 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max · Sonnet 5.5·medium |
| driftproofhq Report 014 — code-review-and-quality (no skill), Haiku 5.5 vs Haiku 4.5 driftproofhq | driftproofhq.com/reports/014/, 3 runs averaged, grader Claude Opus 5 (claude-judge); 'code-review-and-quality' skill, baseline mode; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo |
| driftproofhq Report 014 — code-review-and-quality (with skill), Haiku 5.5 vs Haiku 4.5 driftproofhq | driftproofhq.com/reports/014/, 3 runs averaged, grader Claude Opus 5 (claude-judge); 'code-review-and-quality' skill, with_skill mode; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo |
| driftproofhq Report 014 — documentation-and-adrs (no skill), Haiku 5.5 vs Haiku 4.5 driftproofhq | driftproofhq.com/reports/014/, 3 runs averaged, grader Claude Opus 5 (claude-judge); 'documentation-and-adrs' skill, baseline mode; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo |
| driftproofhq Report 014 — documentation-and-adrs (with skill), Haiku 5.5 vs Haiku 4.5 driftproofhq | driftproofhq.com/reports/014/, 3 runs averaged, grader Claude Opus 5 (claude-judge); 'documentation-and-adrs' skill, with_skill mode; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo |
| driftproofhq Report 014 — git-workflow-and-versioning (no skill), Haiku 5.5 vs Haiku 4.5 driftproofhq | driftproofhq.com/reports/014/, 3 runs averaged, grader Claude Opus 5 (claude-judge); 'git-workflow-and-versioning' skill, baseline mode; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo |
| driftproofhq Report 014 — git-workflow-and-versioning (with skill), Haiku 5.5 vs Haiku 4.5 driftproofhq | driftproofhq.com/reports/014/, 3 runs averaged, grader Claude Opus 5 (claude-judge); 'git-workflow-and-versioning' skill, with_skill mode; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Haiku 4.5·solo |
| DrivingBench: progress along a 127 m real-car cone course (best attempt, %) drivingbench-2609.38948 | DrivingBench Table 2: VLM drives a real Toyota Corolla through 3 tools, one chat session per model = one trial (n=1), up to 3 attempts with retained context and a fixed reflection prompt; progress % = furthest point on the centreline (best attempt); each model in its native harness (Codex CLI 0.154.0-alpha.6.2 for GPT-6 Astra and GPT-5.6 Sol; Claude Code 2.1.274; Cursor); cost = tokens x public list prices as of Sep 2026 summed over all attempts; values typed by hand from the page-7 image (the text layer runs digits together); arXiv 2609.38948 (Sep 2026); read 2026-10-20 ✓ | xmodel | Fable 5.1·medium |
| DSEffi-Bench data-science code, Pass (correctness) % arxiv-2608.30248 | DSEffi-Bench 1000 StackOverflow-derived tasks, greedy T=0.0, tests; Pass=tasks solved (Beyond efficiency same runs not used); open models served via vLLM; paper 2026-08-31; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| Dyad app-builder · Deskhero github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Deskhero (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Fable 5·medium · Sonnet 5·medium |
| Dyad app-builder · Portalis github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Portalis (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Fable 5·medium · Sonnet 5·medium |
| Dyad app-builder · Relay CRM github-dyad | dyad-sh/dyad ba27f52 · local-agent mode, Relay CRM (3 milestones), medium effort, n=1, composite of Playwright CUJs, security probes and an LLM judge; list-price cost ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium · Fable 5.1·medium · Opus 5·medium · Fable 5·medium · Sonnet 5·medium |
| E-Bench (MCP tools): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench (MCP tools): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| E-Bench-Code (code execution): Avg@3 task success ebench-2607.23722 | E-Bench (Tencent-related authors; apps Honor of Kings / QQ Music / Tencent Meeting) E-Bench-Code (code execution): 323 tasks, 3 independent trials from fresh DB copies, MCP tool agents with shared harness, deterministic final-DB-diff grader; Avg@3 = mean trial success (Pass@3, Pass^3 same runs); cost = per-task USD computed by the authors from task-average tokens x list price (Table 5; price date not given); arXiv 2607.23722 (28 Jul 2026); read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| E-Commerce Bench (arXiv 2608.30730) arxiv-2608.30730 | E-Commerce Bench, simulated e-commerce store, 365-day task, 18 tools, 4000-turn cap, 5 independent episodes per model in one fixed world, harness = the paper's agent loop; score = mean end-of-year total assets (kilo-yuan) from a 100k stake, bankrupt episodes kept in the mean; no cost reported; arXiv v1 2026-08-31; read 2026-10-20 ✓ | xmodel | Fable 5·max · Opus 4.8·max · Opus 4.7·max |
| econ-eval leaderboard A (20 tasks, Opus-5 judge) github-deluair-econ-eval | deluair/econ-eval README at commit 77f9f3e (16 Oct 2026), leaderboard of 2026-09-10: 20 economics tasks (quantitative, reasoning, coding, writing) x 5 samples = 100 completions/model, numeric/exact/code-exec graders + rubric tasks judged by Claude Opus 5; score = mean over tasks of task means; cost = total for the 100 completions from token counts x list price (prices read from the OpenRouter catalog 2026-07-31 / vendor pages); per-track tables are the same completions (not extra groups); ten models ran 2026-08-01, deepseek-flash added 2026-09-10 same tasks/samples/judge; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| econ-eval leaderboard B (50 tasks, GPT-6 Astra judge) github-deluair-econ-eval | deluair/econ-eval README at commit 77f9f3e, leaderboard B: 50 tasks (the 20 above + 30 daily-work tasks added 2026-09-15), graded by GPT-6 Astra; 5 samples per task (Argon 1); runs dated 2026-09-15 / 2026-10-13 / 2026-10-15; cost only where an API price exists; different judge and task set from leaderboard A: separate group; read 2026-10-20 ✓ | xmodel | Fable 5.1·default |
| EditHero: long-horizon part-level 3D mesh editing, comparison subset (416 turns) arxiv-edithero | arxiv.org/html/2610.02298v1 Table 3; IF = instruction-following score [0,1] (headline metric); no effort token given for the Claude rows (Sol 6 and Astra are marked medium, Opus 5.5/Fable 5.1 are not) -> default; harness = Claude Code for both Claude models; single rollout per chain, no repeats stated; other metrics in the source (CC-prev, F_W, LPIPS_E/W, PSNR_W, and the Table 15/16 cond./held-out split) not ingested here; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default |
| EEBench (xAI Grok 4.7 post) xai-grok-4-7-launch | EEBench electrical engineering accuracy %, harness/n not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Fable 5.1·max |
| EffiBench IIV arxiv-2609.14839 | direct API · n=5/model, IIV-penalty abstention on optimal code ✓ | xmodel | Opus 4.7·default · Opus 4.8·default |
| EMB vals.ai | Vals AI · bash+execute, 10 Claude models at max (payload compute_effort) — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo · Sonnet 5.5·max · Haiku 5.5·max |
| Encrypted-skill attack (caesar CSI): task completion rate, Haiku 5.5 vs Sonnet 5.5 arxiv-2610.18262 | arXiv 2610.18262 (13 Oct 2026); caesar CSI; TCR on ~100 non-destructive tasks, skill injected; read 2026-10-18 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default |
| Encrypted-skill attack (layered CSI): task completion rate, Haiku 5.5 vs Sonnet 5.5 arxiv-2610.18262 | arXiv 2610.18262 (13 Oct 2026); layered CSI; TCR on ~100 non-destructive tasks, skill injected; read 2026-10-18 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default |
| EnigmaForge (2609.30144) — 600 procedurally generated problem-discovery instances, exact-match arxiv-2609.30144 | Table 1, fact F1 [95% CI] over 600 instances, one configuration per model; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Haiku 4.5·solo · Sonnet 5·default · Fable 5·default · Opus 5·default · Fable 5.1·default |
| ENSO stochastic-model building (SciExam), final composite score arxiv-2610.10513 | agent builds an ENSO model within 6 h eligible time, one run per system, Claude Code on subscription; Table 4/6 of arXiv 2610.10513; read 2026-10-15 ✓ | xmodel | Fable 5.1·default · Opus 5·default |
| Epoch · chess puzzles epoch.ai | Epoch AI's own evaluation (chess_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Sonnet 4.6·max · Haiku 4.5·think-59k · Haiku 4.5·think-32k · Sonnet 4.6·high · Sonnet 4.6·medium · Opus 4.7·max · Opus 4.7·xhigh · Opus 4.7·low · Opus 4.8·max · Opus 4.8·low · Opus 4.8·nothink · Fable 5·max · Fable 5·high · Fable 5·low · Sonnet 5·max · Sonnet 5·xhigh · Opus 5·max · Opus 5·low · Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Haiku 5.5·low · Opus 5·default · Sonnet 4.6·think-32k |
| Epoch · ebr bench epoch.ai | Epoch AI's own evaluation (ebr_bench), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Opus 4.8·max · Fable 5·max · Opus 4.7·max · Haiku 5.5·max |
| Epoch · frontiermath epoch.ai | Epoch AI's own evaluation (frontiermath), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh · Sonnet 4.6·think-16k · Haiku 4.5·think-32k · Haiku 4.5·solo |
| Epoch · frontiermath erdos epoch.ai | Epoch AI's own evaluation (frontiermath_erdos), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Fable 5.1·max · Fable 5·max · Sonnet 5.5·max · Opus 5.5·max |
| Epoch · frontiermath tier 4 epoch.ai | Epoch AI's own evaluation (frontiermath_tier_4), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh · Sonnet 4.6·think-16k · Haiku 4.5·think-32k |
| Epoch · frontiermath tier 4 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tier_4_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max · Fable 5·max · Haiku 5.5·max |
| Epoch · frontiermath tiers 1 3 v2 epoch.ai | Epoch AI's own evaluation (frontiermath_tiers_1_3_v2), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max · Fable 5·max · Haiku 5.5·max |
| Epoch · furniture assembly epoch.ai | Epoch AI's own evaluation (furniture_assembly), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Opus 4.7·max · Haiku 5.5·max |
| Epoch · gpqa diamond epoch.ai | Epoch AI's own evaluation (gpqa_diamond), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Sonnet 5·max · Sonnet 4.6·max · Opus 5·default · Opus 5·low · Opus 4.8·nothink · Opus 4.8·low · Opus 4.7·max · Fable 5·max · Fable 5·low · Fable 5·high · Opus 5·max · Sonnet 4.6·high · Sonnet 4.6·medium · Sonnet 5·xhigh · Opus 4.8·max · Opus 4.7·xhigh · Sonnet 4.6·think-32k · Haiku 4.5·think-32k · Haiku 4.5·solo · Haiku 5.5·max |
| Epoch · math level 5 epoch.ai | Epoch AI's own evaluation (math_level_5), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Haiku 4.5·think-32k · Haiku 4.5·solo |
| Epoch · mirrorcode epoch.ai | Epoch AI's own evaluation (mirrorcode), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Opus 5.5·max · Fable 5.1·high · Opus 4.7·high · Fable 5·high |
| Epoch · mystery game puzzles epoch.ai | Epoch AI's own evaluation (mystery_game_puzzles), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Sonnet 4.6·default · Sonnet 4.6·low · Opus 5·default · Opus 4.7·default · Sonnet 5·default · Fable 5·max · Sonnet 5·max · Opus 4.8·xhigh · Opus 4.8·max · Opus 5·max · Opus 4.7·max · Haiku 5.5·max |
| Epoch · otis mock aime 2024 2025 epoch.ai | Epoch AI's own evaluation (otis_mock_aime_2024_2025), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5·max · Opus 5·low · Sonnet 4.6·max · Opus 5·default · Opus 4.8·nothink · Opus 4.8·low · Opus 4.7·max · Fable 5·low · Fable 5·high · Opus 5·max · Sonnet 4.6·high · Sonnet 4.6·medium · Sonnet 5·xhigh · Fable 5·max · Opus 4.8·max · Opus 4.7·xhigh · Sonnet 4.6·think-32k · Haiku 4.5·think-32k · Haiku 4.5·solo · Haiku 5.5·xhigh · Haiku 5.5·high |
| Epoch · simpleqa verified epoch.ai | Epoch AI's own evaluation (simpleqa_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5·xhigh · Haiku 4.5·think-32k · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·xhigh · Opus 4.8·max · Opus 5·max · Fable 5·xhigh · Sonnet 4.6·high · Haiku 4.5·solo · Haiku 5.5·max |
| Epoch · swe bench verified epoch.ai | Epoch AI's own evaluation (swe_bench_verified), benchmarking hub archive read 30 Sep 2026; effort from the model-version suffix, mean score ✓ | xmodel | Opus 4.7·max · Sonnet 4.6·default |
| Epoch AI HLE external table (accuracy %, 2026-10-15 zip) epoch.ai | epoch.ai/data/benchmark_data.zip (hle_external.csv; zip read 2026-10-15; copy gc/c4/src/epoch/2026-10-15); Humanity's Last Exam, accuracy with standard error; Notes column empty (provenance unstated); values differ from every Scale HLE capture (not a restatement); read 2026-10-20 ✓ | xmodel | Fable 5.1·xhigh · Sonnet 5.5·xhigh · Opus 4.7·default · Haiku 5.5·default |
| EQ-Bench 4 eqbench.com | EQ-Bench 4, emotional-intelligence conversations judged pairwise by an LLM panel · Elo (soft Bradley-Terry) with its interval and scenario count in the flags; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Opus 5·default · Fable 5·default · Opus 4.7·default · Opus 4.8·default · Sonnet 5·default · Sonnet 4.6·default · Haiku 4.5·solo |
| EQ-Bench creative-writing-longform (EQ-bench/eqbench-leaderboard-results) EQ-bench/eqbench-leaderboard-results (GitHub) | GitHub repo EQ-bench/eqbench-leaderboard-results, pinned commit 266853f2ac7d10c31430cb3bd00d64c758beb581, dir creative-writing-longform/results_sonnet-4.6/, 12/12 multi-chapter longform tasks, judge claude-sonnet-4-6 (flag claude-judge), 1 iteration (n=1); distinct board from the already-held eqbench4 (eqbench.com static roleplay-Elo data, different judge/metric/source) — not a restatement.; read 2026-10-08 ✓ | xmodel | Haiku 4.5·solo · Opus 4.7·default · Sonnet 4.6·default |
| EvalResearchBench: quality of evaluation suites designed by the model (overall Score, 14 targets) arxiv-2610.04184 | one research episode per model, frozen assessment on 14 targets; Table 2 of arXiv 2610.04184; read 2026-10-15 ✓ | xmodel | Fable 5.1·default · Opus 5·default |
| eveagent-base github-vercel-eve | config ✓ | xmodel | Fable 5.1·default · Opus 5·default · Sonnet 5·default |
| eveagent-guided github-vercel-eve | config ✓ | xmodel | Fable 5.1·default · Opus 5·default · Sonnet 5·default |
| evg-baserm arxiv-2604.26180 | config ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| evg-opt arxiv-2604.26180 | config ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| evg-rag arxiv-2604.26180 | config ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| evg-rlm arxiv-2604.26180 | config ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| evg-unopt arxiv-2604.26180 | config ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| EvoClaw evo-claw.com | EvoClaw, long-horizon repository evolution in Claude Code, current board v1.0.2 · overall score in %, total cost of the run (USD) and output tokens; effort from the board's label (Opus 4.8 max 1M, Opus 4.7 xhigh; none stated otherwise); the Opus 4.7 1M-context and OpenHands rows left out (another configuration); read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh · Sonnet 4.6·default |
| EvoRiskBench task success rate (Claude Code harness) arxiv-2610.03153 | EvoRiskBench (arXiv 2610.03153, 2 Oct 2026) Table 4: task success rate (TSR %) of 450 frozen workspace-agent cases with embedded indirect-prompt-injection attacks; harness Claude Code (versions Claude Code 2.1.268 / Codex CLI 0.154.0 / OpenClaw 2026.9.4); utility under attack, not ASR; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| EvoRiskBench task success rate (Codex harness) arxiv-2610.03153 | EvoRiskBench (arXiv 2610.03153, 2 Oct 2026) Table 4: task success rate (TSR %) of 450 frozen workspace-agent cases with embedded indirect-prompt-injection attacks; harness Codex (versions Claude Code 2.1.268 / Codex CLI 0.154.0 / OpenClaw 2026.9.4); utility under attack, not ASR; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| EvoRiskBench task success rate (OpenClaw harness) arxiv-2610.03153 | EvoRiskBench (arXiv 2610.03153, 2 Oct 2026) Table 4: task success rate (TSR %) of 450 frozen workspace-agent cases with embedded indirect-prompt-injection attacks; harness OpenClaw (versions Claude Code 2.1.268 / Codex CLI 0.154.0 / OpenClaw 2026.9.4); utility under attack, not ASR; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| Exploit Gym (table) openai-gpt6-astra-launch | Exploit Gym; OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Fable 5·default · Opus 5·default |
| ExploitBench (OpenAI GPT-5.6 post chart) openai-gpt56-launch | ExploitBench cap percent, API harness, 5 seeds, reasoning continuity (fn1); web chart JSON, not the system-card image; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| ExploitBench (table) openai-gpt6-astra-launch | ExploitBench; OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Opus 5·default |
| ExploitBench coverage (GLM-5.3 card) zai-hf-glm-5.3 | ExploitBench 41 tasks x 3 revisions, union coverage score, Claude Code 2.1.207, 300 rounds; only GLM-5.3 stated as evaluated; GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| ExploitGym 2h budget, tasks solved of 869 (GLM-5.3 card) zai-hf-glm-5.3 | ExploitGym 869 tasks single-run pass@1 (count solved), Claude Code 2.1.207 no web tools, 2h timeout rescaled by per-model TPS (AA); GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| ExploitGym 6h budget, tasks solved of 869 (GLM-5.3 card) zai-hf-glm-5.3 | ExploitGym 869 tasks single-run pass@1 (count solved), Claude Code 2.1.207 no web tools, 6h timeout rescaled by per-model TPS (AA); GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| EXTRA retroboard-antigravity github-retrospective-board-eval | new group made by verifier v6 from the g6 report (rows flagged extra-group; no held Claude rows unless stated); read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default |
| fable-mode injected github-fable-mode | fable-mode RESULTS · 6 tasks with fable-mode prompt, Opus 4.8 vs Haiku 4.5, tokens only ✓ | xmodel | Opus 4.8·default · Haiku 4.5·solo |
| Fable-workflow-skill: no-skill baseline (2026-10-02 rerun) github-fable-workflow | one-shot under-specified rate-limiter spec, no skill injected, scored Thinking/50+Answer/50 by a blind non-contestant judge; read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| Fable-workflow-skill: with-skill (2026-10-02 rerun) github-fable-workflow | same task + the fable-workflow skill's one-shot unknowns-list instruction, scored identically; read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| fablemode github-fable-mode | config ✓ | xmodel | Fable 5·default · Sonnet 5·default |
| fablemode-ctl github-fable-mode | config ✓ | xmodel | Opus 4.8·default · Haiku 4.5·solo |
| fcodev1ext cognition | config ✓ | xmodel | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| FeatureBench Lite split · %PASSED · scaffold OpenHands featurebench-github | FeatureBench README leaderboard (Dataset V1.0, Lite split), column %PASSED, rows whose scaffold is OpenHands; efforts not stated; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| FigCodeBench LaTeX figure-to-code, MMOS (mean of 3 MLLM judges) arxiv-2610.10066 | FigCodeBench LaTeX subset, reproduce figure from image; MMOS = mean model opinion score of judges GPT-5.1/GPT-5.6-Luna/Gemini-3.5-Flash (Exec rate, PSNR/SSIM same runs not used); cost = approximate $ per query; paper 2026-10-07; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| FigCodeBench Matlab figure-to-code, MMOS (mean of 3 MLLM judges) arxiv-2610.10066 | FigCodeBench Matlab subset, reproduce figure from image; MMOS = mean model opinion score of judges GPT-5.1/GPT-5.6-Luna/Gemini-3.5-Flash (Exec rate, PSNR/SSIM same runs not used); cost = approximate $ per query; paper 2026-10-07; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| FigCodeBench Python figure-to-code, MMOS (mean of 3 MLLM judges) arxiv-2610.10066 | FigCodeBench Python subset, reproduce figure from image; MMOS = mean model opinion score of judges GPT-5.1/GPT-5.6-Luna/Gemini-3.5-Flash (Exec rate, PSNR/SSIM same runs not used); cost = approximate $ per query; paper 2026-10-07; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| FigCodeBench R figure-to-code, MMOS (mean of 3 MLLM judges) arxiv-2610.10066 | FigCodeBench R subset, reproduce figure from image; MMOS = mean model opinion score of judges GPT-5.1/GPT-5.6-Luna/Gemini-3.5-Flash (Exec rate, PSNR/SSIM same runs not used); cost = approximate $ per query; paper 2026-10-07; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| FinanceAgent v1.1 (table) openai-gpt55-launch | FinanceAgent v1.1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| Financial Touchstone v1.2 accuracy (annual-report QA, shared top-5 retrieval) arxiv-2608.08634 | arXiv 2608.08634 (9 Aug 2026) Financial Touchstone v1.2: 2,967 question-context-answer items on 495 annual reports, same top-5 retrieved context for every model, output cap 5,000 tokens, LLM judge Claude Opus 4.6 (16 models) / Claude Opus 4.8 (Kimi K2.6, GLM-4.7, Qwen3-Max, ERNIE 5.1); score = accuracy excluding retriever errors (Fig. 1); hallucination rate = same runs, other metric (not a group); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| FinancialAuditBench (2609.32835) — 45 audit workpaper tasks, bash-only agent, 8 runs/task arxiv-2609.32835 | Table 8/Table 9 main row, strict pass % and mean cost $/trajectory, 360 trajectories/model; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5.5·medium · Fable 5.1·medium |
| FinIndices Single-Index with formula hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; with formula hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Single-Index with formula hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; with formula hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Single-Index with formula hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; with formula hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Single-Index without hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; without hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Single-Index without hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; without hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Single-Index without hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Single-Index queries; without hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Table-Index with formula hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; with formula hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Table-Index with formula hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; with formula hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Table-Index with formula hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; with formula hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Table-Index without hint, dev arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; without hint; split dev; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Table-Index without hint, test arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; without hint; split test; end-to-end accuracy %; mean of 3 runs ± sd (Dev/Test); read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| FinIndices Table-Index without hint, train arxiv-2607.28661 | FinIndices (Chinese listed-company financial statements, long context ~16k tokens); Table-Index queries; without hint; split train; end-to-end accuracy %; single run on the training split; read 2026-10-19 ✓ | xmodel | Opus 4.8·solo |
| Firecrawl 57-run firecrawl | claude -p in sandbox-exec · 3 models at matched high, measured usage; 7/7 all → cost only ✓ | xmodel | Fable 5·high · Fable 5.1·high · Opus 5·high |
| Firefox 4-line plan, Claude as the 'hand' choosing numbered options (mappy55) zenn-mappy55 | same 4-line plan, prompt, judge code; Haiku 5.5 6 runs, Sonnet 5.5 3 runs; cost converted at 150 JPY/USD; read 2026-10-18 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default |
| FlavourBench Score (mean of 3 families x 178 tasks) flavourbench-2608.20574 | FlavourBench Table 3 leaderboard, 534 tasks, FB Score = equal-family mean, 2 collection panels pooled, OpenRouter fixed provider (GLM 5.3 direct); arXiv 2608.20574 (20 Aug 2026); family columns Table 10 are same runs (rule 8); read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 5·default · Sonnet 5·default |
| ForecastBench preliminary leaderboard: dataset-question Brier Index, web search + code execution, 128k, 984 questions forecastbench | ForecastBench preliminary leaderboard; dataset score = difficulty-adjusted Brier Index; N=984 resolved dataset questions; cost absent; read 2026-10-18 ✓ | xmodel | Opus 5·max · Fable 5·max · Fable 5·high |
| forge Phase 2 github-forge-benchmark | Claude Code + Repomix · 3 tasks, Haiku/Sonnet 4.6/Opus 4.6 × medium/max, saturated → tokens only ✓ | xmodel | Sonnet 4.6·medium · Sonnet 4.6·max · Haiku 4.5·req-medium · Haiku 4.5·req-max |
| FrontierChallenge (Claude Code scaffold): Pass Rate overall frontierchallenge-2608.24979 | FrontierChallenge (Apodex Team; vendor of Apodex 1.1, not one of the eight providers) Table 2: 97 scientific-workflow tasks (ORCA/CP2K/LAMMPS/AmberTools/PLUMED), whole artifact bundle graded against a task contract, Judge = GPT-5.6 Sol x3 passes; Pass Rate (full completion) overall; scaffold Claude Code; Avg Score and HDS6 are secondary; Apodex rows are not in the eight providers; arXiv 2608.24979 (Aug 2026); read 2026-10-20 ✓ | xmodel | Opus 5·default |
| FrontierCode (OpenAI GPT-6 Sol and Luna post) openai-gpt6-sol-luna-launch | FrontierCode; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| FrontierCode 1.1 Extended (OpenAI GPT-6 Astra post chart, code mode) openai-gpt6-astra-launch | FrontierCode 1.1 Extended; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = code; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max |
| FrontierCode 1.1 Extended (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | FrontierCode 1.1 Extended; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| FrontierCode v1.1 Extended (xAI Grok 4.6 post) xai-grok-4-6-launch | FrontierCode v1.1 (Extended) %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| FrontierFinance T4 arxiv-2608.11683 | arXiv 2608.11683 Table 4 · effort omitted, Fable 5 vs Opus 4.8 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| FrontierMath Tier 4 (v2) (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | FrontierMath Tier 4 (v2); OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh |
| FrontierMath Tier 4 (v2) (table) openai-gpt56-launch | FrontierMath Tier 4 (v2); OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default |
| FrontierSWE dominance score (GLM-5.2 card, as of 2026/06/16) zai-hf-glm-5.2 | FrontierSWE, evaluated by Proximal, 1M context 128K output, dominance score as of 2026/06/16; held group frontierswe is mean-reward@5 (other metric and date); read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| FrontierSWE dominance score (GLM-5.3 card, as of 2026/08/14) zai-hf-glm-5.3 | FrontierSWE, evaluated by Proximal, 1M context, 128K output, dominance score as of 2026/08/14; held group frontierswe is mean-reward@5 (glm-5.3 30.18 there vs 78.1 here: different metric); read 2026-10-19 ✓ | xmodel | Opus 4.8·max · Fable 5·max |
| FrontierSWE v2 frontierswe | Proximal frontierswe.com + Sonnet 5.5 card · 34 ultra-long tasks, max effort, proximus harness, mean reward@5; cost basis undocumented ✓ | xmodel | Fable 5.1·max · Opus 5·max · Fable 5·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max |
| GameASG-Bench: strict task success arxiv-2609.21293 | GameASG-Bench (arXiv 2609.21293, 18 Sep 2026); 47 browser game-generation tasks; each (stack, task) pair run once from clean workspace; headless Chromium; Claude Code 2.1.206 or Codex CLI 0.153.4; full tools, nominal 120 turns; strict success = all L1+L2 checks; cost = reported mean resource use (Table 4); read 2026-10-20 ✓ | xmodel | Opus 5·max |
| GameLogicBench arxiv-2609.21562 | arXiv 2609.21562 · 72 Godot game-logic tasks in Claude Code, effort=high, Opus 5 vs Sonnet 5, tokens × vendor list ✓ | xmodel | Opus 5·high · Sonnet 5·high |
| garrytan/gbrain-evals N2 judge replay (6 Oct 2026): 741 judged pairs, contradiction detection prompt v4; recall saturated 150/150 for all three, cost-only github-gbrain-evals | docs/benchmarks/2026-10-06-n2-judges; cost from the repo budget ledger (741 paid requests); no effort control; read 2026-10-18 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default |
| gbstox agronomy LLM benchmark: text-based Overall Score (425 questions), live README 2026-10-17 gbstox-agronomy | gbstox/agronomy_llm_benchmarking README leaderboard (pushed 2026-10-17); Overall Score = text categories (image columns excluded); per-model Date Tested 2026-06..10; OpenRouter, quality-first quantization routing; no reasoning setting stated; read 2026-10-20 ✓ | xmodel | Opus 5.5·default · Haiku 5.5·default · Opus 4.8·default · Sonnet 5·default · Sonnet 5.5·default · Fable 5·default |
| gdp.pdf (table) openai-gpt56-launch | gdp.pdf; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| GDPval (Elo) (xAI Grok 4.7 post) xai-grok-4-7-launch | GDPval Elo (Artificial Analysis GDPval-AA family by Elo scale); chart only; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Fable 5.1·max |
| GDPval (wins or ties) (table) openai-gpt55-launch | GDPval (wins or ties); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| GDPval-AA v2 (OpenAI GPT-5.6 post chart) openai-gpt56-launch | GDPval-AA v2; AA scoring restated in the OpenAI post chart (cells not identical to held artificialanalysis cells), per effort; cost = OpenAI-simulated, scope unstated; snapshot = post date; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| GDPval-AA v2 (table) openai-gpt56-launch | GDPval-AA v2; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| GDPVal-AA v2 (xAI Grok 4.6 post) xai-grok-4-6-launch | GDPVal-AA v2 Elo; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| GDPval-AA v2 Elo (GLM-5.3 card) zai-hf-glm-5.3 | GDPval-AA v2, 'Models are evaluated by Artificial Analysis'; GLM-5.3 card 2026-10-14; step 1; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| GDPval-AA v2 Elo (GLM-5.3-Flash chart) zai-hf-glm-5.3-flash | GDPval-AA v2 evaluated by Artificial Analysis; Flash chart 2026-10-15; glm-5.2 1504 and opus-4.8 1582 differ from the GLM-5.3 card (1508, 1588) -> own group; step 1; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| GeneBench Pro (OpenAI GPT-5.6 post chart) openai-gpt56-launch | GeneBench Pro, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| Gert Labs GBENCH agentic coding (models write code run as agents) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode agentic_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Opus 5·default · Sonnet 5.5·default · Fable 5·default · Haiku 5.5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default · Sonnet 5·default |
| Gert Labs GBENCH decision making (models play the games) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode decision_making, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Opus 5·default · Sonnet 5.5·default · Fable 5·default · Haiku 5.5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default · Sonnet 5·default |
| Gert Labs GBENCH one-shot coding (models write code in one shot) gertlabs-gbench | Gert Labs GBENCH global rankings API (gertlabs.com/api/v1/global/rankings?limit=300), mode oneshot_coding, gscore 0-1 (board composite of percentile / success / response time), cost = avg_cost_usd per submission as printed by the payload (definition not documented); models' own games set varies by model; live board, payload 2026-10-20T05:55Z; read 2026-10-20 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Opus 5·default · Sonnet 5.5·default · Fable 5·default · Haiku 5.5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default · Sonnet 5·default |
| GM-Bench 2.0 agentic lane (hockey-GM 5-season episode, 32 private seeds, Claude Code) github-gmbench | nedcut/gm-bench README (rows published 12-15 Oct 2026); Claude Code harness default effort; mean score over 32-seed private panel, reference pick-trader heuristic = 249.2; rows flagged unpinned by the source; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Haiku 4.5·solo · Sonnet 5·default |
| GM-Bench 2.1: 5-season hockey GM inside Claude Code, 32-seed private panel, score-v2 (released 2026-10-16) github-gmbench | nedcut/gm-bench results/agentic_v21 claude-2.1.281 panel-32x5; reference pick-trader 249.618 (2.1 sim-v4-league21); NEW GROUP: 2.1 is a new simulator/panel, held gm-bench2-agentic is 2.0 (pick-trader 250.068); read 2026-10-18 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Haiku 5.5·default |
| GMMLU (S55) anthropic-syscard | Sonnet 5.5 card p140 · 4 models at max, 1 trial, score-only ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5·max |
| Google model cards: Agent's Last Exam (pass rate) google-gemini-3.7-flash-card | Agent's Last Exam, binary pass rate; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: Agent's Last Exam (pass rate) google-gemini-4-argon-card | Agent's Last Exam, binary pass rate; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Opus 5.5·default |
| Google model cards: BrowseComp (search + python + browse) google-gemini-3.1-pro-card | BrowseComp, Gemini via Deep Research with search, python and browsing; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·max |
| Google model cards: CharXiv Reasoning (1000 val questions) google-gemini-3.1-flash-lite-card | CharXiv Reasoning, 1000 reasoning questions of the validation split; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: CharXiv Reasoning no tools google-gemini-3.5-flash-lite-card | CharXiv Reasoning no tools; Google model-card table(s) 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: CharXiv Reasoning no tools google-gemini-3.8-flash-card | CharXiv Reasoning no tools; Google model-card table(s) 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| Google model cards: DeepSWE v1.1 google-gemini-3.8-flash-card | DeepSWE v1.1 (Datacurve), mini-swe-agent harness; Google model-card table(s) 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Opus 5·default |
| Google model cards: DeepSWE v1.1 google-gemini-4-argon-card | DeepSWE v1.1 (Datacurve), mini-swe-agent harness; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default |
| Google model cards: GDP.PDF all-pass google-gemini-3.7-flash-card | GDP.PDF all pass rate; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: GDP.PDF all-pass google-gemini-3.8-flash-card | GDP.PDF all pass rate; Google model-card table(s) 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Opus 5·default |
| Google model cards: GDPval-AA (Elo) google-gemini-3.1-pro-card | GDPval-AA Elo (Artificial Analysis); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·max |
| Google model cards: GDPval-AA (Elo) google-gemini-3.5-flash-card | GDPval-AA Elo (Artificial Analysis); Google model-card table(s) 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| Google model cards: GDPVal-AA v2 (Elo) google-gemini-3.6-flash-card | GDPVal-AA v2 Elo (Artificial Analysis); Google model-card table(s) 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: GDPVal-AA v2 (Elo) google-gemini-3.5-flash-lite-card | GDPVal-AA v2 Elo (Artificial Analysis); Google model-card table(s) 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: GDPVal-AA v2 (Elo) google-gemini-3.7-flash-card | GDPVal-AA v2 Elo (Artificial Analysis); Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: GDPVal-AA v2 (Elo) google-gemini-3.8-flash-card | GDPVal-AA v2 Elo (Artificial Analysis); Google model-card table(s) 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric elo; read 2026-10-19 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| Google model cards: GPQA Diamond (no tools) google-gemini-3.1-flash-lite-card | GPQA Diamond no tools; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: GraphWalks 256k-1M BFS F1 google-gemini-4-argon-card | GraphWalks BFS F1, 256k to 1M, 200 items; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default |
| Google model cards: GraphWalks up to 128k BFS F1 google-gemini-4-argon-card | GraphWalks BFS F1, up to 128k, 650 items; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default |
| Google model cards: LABBench2 google-gemini-4-argon-card | LABBench2, Linux terminal with bioinfo tools, Python, R and internet; macro-average of 11 sub-tasks; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default |
| Google model cards: LiveCodeBench (175 UI problems) google-gemini-3.1-flash-lite-card | LiveCodeBench code generation, 175 UI problems 1 Jan-1 May 2025; Google model-card table(s) 3 Mar 2026 (google-gemini-3.1-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: LVBench google-gemini-4-argon-card | LVBench long video understanding, no tools; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default |
| Google model cards: MCP Atlas (public set, Turing) google-gemini-3.1-pro-card | MCP Atlas public set sourced from Turing; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·max |
| Google model cards: MCP Atlas (Scale AI board) google-gemini-3.5-flash-card | MCP Atlas, Scale AI leaderboard; Google model-card table(s) 19 May 2026 (google-gemini-3.5-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default |
| Google model cards: MLE-Bench Partial 30 (position score) google-gemini-3.6-flash-card | MLE-bench Partial 30 subset, Average Position Score; Google model-card table(s) 21 Jul 2026 (google-gemini-3.6-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: MRCR v2 8-needle 128k (average) google-gemini-3.7-flash-card | MRCR v2 8-needle, 128k cumulative average; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: OSWorld-2.0 (partial score) google-gemini-3.8-flash-card | OSWorld 2.0 partial score; Google model-card table(s) 2 Sep 2026 (google-gemini-3.8-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: OSWorld-Verified google-gemini-3.5-flash-lite-card | OSWorld-Verified, 100 max steps, pyautogui; Google model-card table(s) 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: PostTrainBench v1.1 google-gemini-4-argon-card | PostTrainBench v1.1, OpenCode harness, 10 h on one H100, weighted aggregate of 4 base models x 7 benchmarks; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default |
| Google model cards: SciCode google-gemini-3.1-pro-card | SciCode (Artificial Analysis); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·max |
| Google model cards: SWE-Bench Pro (Public) google-gemini-3.5-flash-lite-card | SWE-Bench Pro public set; Google model-card table(s) 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: tau2-bench Retail google-gemini-3.1-pro-card | tau2-bench Retail (Airline excluded); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·max |
| Google model cards: tau2-bench Telecom google-gemini-3.1-pro-card | tau2-bench Telecom (Airline excluded); Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·max |
| Google model cards: Terminal-Bench 2.0 (Terminus-2) google-gemini-3.1-pro-card | Terminal-Bench 2.0, default agent harness Terminus 2; Google model-card table(s) 19 Feb 2026 (google-gemini-3.1-pro-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·max |
| Google model cards: Terminal-Bench 2.1 (Terminus-2) google-gemini-3.5-flash-lite-card | Terminal-Bench 2.1, default agent harness Terminus 2 only; Google model-card table(s) 21 Jul 2026 (google-gemini-3.5-flash-lite-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo |
| Google model cards: Terminal-Bench 3.0 google-gemini-3.7-flash-card | Terminal-Bench 3.0; Google model-card table(s) 13 Aug 2026 (google-gemini-3.7-flash-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Sonnet 5·default |
| Google model cards: Terminal-Bench Science 0.1 google-gemini-4-argon-card | Terminal-Bench Science 0.1; Google model-card table(s) 13 Oct 2026 (google-gemini-4-argon-card); benchmark pages as printed; score metric pct; read 2026-10-19 ✓ | xmodel | Opus 5.5·default |
| GPQA Diamond (table) openai-gpt55-launch | GPQA Diamond; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| GPQA Diamond (table) openai-gpt56-launch | GPQA Diamond; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| GPQA Diamond (table) openai-gpt6-astra-launch | GPQA Diamond; OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5·default |
| GPQA-Diamond (GLM-5.2 card) zai-hf-glm-5.2 | GPQA-Diamond, temperature=1.0 top_p=0.95 163,840 tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| GraphWalks BFS 1mil f1 (table) openai-gpt56-launch | GraphWalks BFS 1mil f1; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| Graphwalks BFS 256k f1 (table) openai-gpt55-launch | Graphwalks BFS 256k f1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| GraphWalks BFS 256k f1 (table) openai-gpt56-launch | GraphWalks BFS 256k f1; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| Graphwalks parents 256k f1 (table) openai-gpt55-launch | Graphwalks parents 256k f1; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| GRIP closed-loop accuracy (E1, mean over Q1-Q5), spatial and physical reasoning for VLMs grip-2610.16170 | GRIP Table 3 Mean (E1): 34 domains, procedurally generated images with closed-form answers, 5 questions per image in fixed order, fresh single-turn per item, 8,500 queries per hosted model, routed API; hosted max output 256-2048 tokens; Q1-Q5 columns are the same runs (rule 8); arXiv 2610.16170 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 5·low · Sonnet 5·low |
| GSO (OpenHands), runs dated 2026-04-27..07-11 gso-bench | GSO: 102 tasks, 10 codebases, 5 languages; make code as fast as the expert patch; scaffold OpenHands; Opt@1 = fraction of tasks where one attempt gets >=95% of human speedup and passes tests (hack-adjusted column is another summary); board last_updated 2026-09-27; this group = runs dated 2026-04-27..07-11 (2x max_iterations=200; reasoning_effort for Claude) per the page changelog; read 2026-10-20; read 2026-10-20 ✓ | xmodel | Opus 4.7·high |
| GSO (OpenHands), runs dated 2026-07-12..09-26 gso-bench | GSO: 102 tasks, 10 codebases, 5 languages; make code as fast as the expert patch; scaffold OpenHands; Opt@1 = fraction of tasks where one attempt gets >=95% of human speedup and passes tests (hack-adjusted column is another summary); board last_updated 2026-09-27; this group = runs dated 2026-07-12..09-26 (task hardened: network filters, MCP disabled) per the page changelog; read 2026-10-20; read 2026-10-20 ✓ | xmodel | Opus 4.8·xhigh · Sonnet 5·xhigh |
| GSO (OpenHands), runs dated 2026-09-27 gso-bench | GSO: 102 tasks, 10 codebases, 5 languages; make code as fast as the expert patch; scaffold OpenHands; Opt@1 = fraction of tasks where one attempt gets >=95% of human speedup and passes tests (hack-adjusted column is another summary); board last_updated 2026-09-27; this group = runs dated 2026-09-27 (improved elicitation: keep measuring and improving after the first speedup) per the page changelog; read 2026-10-20; read 2026-10-20 ✓ | xmodel | Fable 5.1·xhigh · Fable 5·xhigh |
| GVS5H paper: LiveCodeBench release_v6 hard-split LCB-100 pass@1 and cost per pass, single call (no manager scaffold) arxiv-2608.26480 | LCB-100 (100 most recent hard LiveCodeBench v6 problems), pass@1 mean of 5 passes, 128k max_tokens, reasoning on (Table 1 header), pinned backends, single call at temperature 0.2, corrected evaluator (App. B); cost = list rate x tokens of one pass over the 100 problems (Table 2, source-computed, retried attempts counted); manager-scaffold arms not rows; read 2026-10-20 ✓ | xmodel | Fable 5·high |
| Hack-Log: 1 harder task (Markdown table->list, 9 hidden tests), Sonnet 5.5 vs Opus 5.5, mean of 3 runs note-hacklog_stealth | note.com/hacklog_stealth/n/n07e2f50a43b9; same setup as hacklog-setA, harder single task, 3 runs/model; report gives means directly (not computed by us): mean cost, mean time, mean output tokens; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| Hack-Log: 3 easy tasks (13 hidden tests), Sonnet 5.5 vs Opus 5.5, 2 runs each note-hacklog_stealth | note.com/hacklog_stealth/n/n07e2f50a43b9; Claude Code 2.1.281, claude -p --permission-mode bypassPermissions; cost = author's own recomputation from token counts x official list price (Claude Code 2.1.281 mis-displays Sonnet 5.5 cost at Opus prices, costBasis=unknown, flagged by the author); effort-from-config: "effort は既定のまま"; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| Haiku 5.5 launch page: Chartography (no tools), benchmark-grid table anthropic-chart | anthropic.com/claude-haiku-5-5 benchmarkGrid table, no per-effort chart for this benchmark on this page; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 launch page: FrontierCode 1.1 (Main) benchmark-grid table anthropic-chart | anthropic.com/claude-haiku-5-5 benchmarkGrid table, no per-effort chart for this benchmark on this page; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5.5·xhigh |
| Haiku 5.5 launch page: Terminal-Bench 4.0, cache-read price-cut illustration (7 Oct 2026) anthropic-chart | anthropic.com/claude-haiku-5-5 launch page, 'Further updates' section, embedded chart CSV (exact); read 2026-10-08 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max |
| Haiku 5.5 system card: AA-Briefcase v1.1, max effort, xmodel anthropic-syscard | Table 8.1.A row, run independently by Artificial Analysis; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: BenchCAD Vision2Code, no tools, max effort, voxel IoU anthropic-syscard | 1,000-file random subset of 17,900 Vision2Code files, 5 runs averaged; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: BenchCAD Vision2Code, with tools, max effort, voxel IoU anthropic-syscard | same subset, with container + image cropping tool; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: biomedical image analysis, max effort anthropic-syscard | CT/MRI/histopathology/dermoscopy/retinal/ultrasound/microscopy imaging, scaled to best public-competition result; 1 attempt/problem (exception to the chapter default); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max |
| Haiku 5.5 system card: Chartography, no tools, max effort anthropic-syscard | Surge AI benchmark, Gemini 3.5 Flash grader, 5 runs averaged; cost digitised from Figure 8.9.1.B (p125, rightmost/highest-cost marker of each dashed 'without tools' curve, anchored by matching its y-pixel to this exact text score, see CHARTCOST note above); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: Chartography, with tools, max effort anthropic-syscard | with tools = container with image file, standard libs, image cropping tool; new vs launch page (which only gave the no-tools row); cost digitised from Figure 8.9.1.B (p125, rightmost marker of each solid 'with tools' curve); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: Claude Code dual-use & benign request success rate, xmodel anthropic-syscard | share of dual-use/benign coding requests the model assisted with (higher = better, same sense as a 100-refusal% score); Claude Mythos 5.1 present in evidence but excluded (not in MODELS set); read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Haiku 4.5·solo |
| Haiku 5.5 system card: CyScenarioBench solve rate, xmodel anthropic-syscard | 10-challenge subset, multi-stage cyber operation orchestration; Claude Mythos 5.1 present in evidence but excluded (not in MODELS set); read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5·default · Haiku 4.5·solo · Sonnet 5.5·default · Opus 5.5·default |
| Haiku 5.5 system card: ExploitBench AutoNudge capability-flag capture rate, xmodel anthropic-syscard | 41 V8 environments, AutoNudge arm, cap rate = % of capability flags captured; cyber safeguards off, escape classifiers on; Claude Mythos 5.1 present in evidence but excluded (not in MODELS set); read 2026-10-08 ✓ | xmodel | Haiku 4.5·solo · Opus 5.5·default · Sonnet 5.5·default · Haiku 5.5·default |
| Haiku 5.5 system card: ExploitGym, 2-hour wall-clock budget, xmodel anthropic-syscard | successful exploits (unauthorized code execution + secret-flag read) out of 869 vulnerability instances; cyber safeguards off; ASLR/stack-canaries/KASLR/(non-sandbox-escape) V8-heap-sandbox disabled, kernel targets keep SMEP/SMAP enabled; read 2026-10-08 ✓ | xmodel | Sonnet 5·default · Opus 5·default · Opus 5.5·default · Sonnet 5.5·default · Haiku 5.5·default |
| Haiku 5.5 system card: ExploitGym, 6-hour wall-clock budget, xmodel anthropic-syscard | same setup as the 2-hour group, longer wall-clock budget; read 2026-10-08 ✓ | xmodel | Sonnet 5·default · Opus 5·default · Opus 5.5·default · Sonnet 5.5·default · Haiku 5.5·default |
| Haiku 5.5 system card: FrontierCode 1.1 (Main), Table 8.1.A anthropic-syscard | DISCREPANCY-FIXED vs h3 report section 2.1: raw text order gives Haiku-5.5 TWO points (max, xhigh), not Haiku-4.5 one point; Haiku 4.5 is a dash (no data) in this row. Corroborated by h3's own section-2.2 image read (xhigh=45.8%, max=46.4% for Haiku 5.5 on the same benchmark's effort chart); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 5.5·xhigh · Sonnet 5.5·max · Sonnet 5.5·xhigh |
| Haiku 5.5 system card: FrontierSWE v2, Proximal's own harness, max effort anthropic-syscard | 34 ultra-long-horizon tasks, 20h trial budget, mean of 5 trials/task, Proximal's own harness; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Fable 5·max |
| Haiku 5.5 system card: Global MMLU average accuracy across 42 languages, max effort anthropic-syscard | single trial, safety classifiers enabled, blocked/unanswered questions excluded from average; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max |
| Haiku 5.5 system card: HealthBench Professional (length-adjusted), max effort, xmodel anthropic-syscard | Table 8.1.A row; same benchmark as the 8.11.3 per-effort sweep group below (shhealthbenchpro-eff), this is the cross-model max-effort snapshot; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: Humanity's Last Exam (with tools) score+cost by effort, 5 efforts anthropic-syscard | chart image, y-axis printed score labels per point; x-axis (cost, log scale) has printed tick labels ($0.01,$0.02,$0.03,$0.05,$0.07,$0.10,$0.20,$0.30,$0.50,$0.70,$1,$2) but no per-point numeric cost label, so cost is digitised from the marker pixel position (calibration in notes.md: x=1685.88+709.69*log10(cost), fit from the $0.01 and $1 ticks, cross-checked exact at $0.1); Haiku 4.5's single point's x could not be confidently isolated from the crowded pink-diamond legend swatch vs. data point in the time available, left without cost; read 2026-10-08 ✓ | xmodel | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo |
| Haiku 5.5 system card: LatchBio Bioinformatics, SingleCellBench, max effort anthropic-syscard | single-cell RNA sequencing analysis; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max |
| Haiku 5.5 system card: LatchBio Bioinformatics, SpatialBench Verified, max effort anthropic-syscard | spatial transcriptomics / chromatin-accessibility analysis; bash tool, file editor, prespecified packages; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max |
| Haiku 5.5 system card: medicinal chemistry (ADME), max effort, xmodel anthropic-syscard | 504 questions, drug-like molecule structures, ADME property comparison; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max · Opus 5.5·max · Sonnet 5.5·max |
| Haiku 5.5 system card: MILU average accuracy across 11 languages, max effort anthropic-syscard | averaged over 5 trials, safety classifiers enabled; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max |
| Haiku 5.5 system card: morphology-to-molecule matching (Axiom Bio), max effort anthropic-syscard | matches 4 multi-channel cell-imaging sets to 4 shuffled molecule-dose perturbations, human liver in-vitro assay; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5.5·max |
| Haiku 5.5 system card: Multimodal virology (VCT) accuracy, xmodel anthropic-syscard | 350-question multimodal virology troubleshooting set; accuracy stored x100 (source scale 0-1) to match the score% convention used elsewhere in this data file; Claude Mythos 5.1 present in evidence but excluded (not in this salvo's MODELS set); read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5·default · Opus 5·default · Opus 5.5·default · Sonnet 5.5·default |
| Haiku 5.5 system card: PhysicianBench, max effort, xmodel anthropic-syscard | 100 physician EHR tasks, 5 attempts/task, pass rate; GPT-6 Luna run by Anthropic (not OpenAI's own figure); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5.5·max · Opus 5.5·max |
| Haiku 5.5 system card: ProgramBench, 166-task hidden-test subset anthropic-syscard | mini-swe-agent harness, no six-hour time limit, hidden hidden-test pass rate; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5.5·default · Opus 5.5·default |
| Haiku 5.5 system card: protein design, Library Ranking, max effort anthropic-syscard | prioritize protein designs for experimental testing; 3 attempts/problem (exception to the chapter default); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max · Sonnet 5.5·max |
| Haiku 5.5 system card: protein design, Sequence Generation, max effort anthropic-syscard | novel protein sequence generation under design constraints; 1 attempt/problem (exception to the chapter's 5-attempt default, stated explicitly); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max · Sonnet 5.5·max |
| Haiku 5.5 system card: Protocols, Troubleshooting, max effort anthropic-syscard | detect and fix issues in molecular biology lab protocols; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max · Sonnet 5.5·max |
| Haiku 5.5 system card: Protocols, Understanding (V2, Benchling), max effort anthropic-syscard | real lab protocol document + practical application question; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5·max · Sonnet 5.5·max |
| Haiku 5.5 system card: single-turn benign-request refusal (child safety), stored as 100-refusal% -- condition: API, without a system prompt anthropic-syscard | Table 4.2.A; condition quoted from the table header: "API, without a system prompt"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (child safety), stored as 100-refusal% -- condition: Claude.ai anthropic-syscard | Table 4.2.A; condition quoted from the table header: "Claude.ai"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (disordered eating), stored as 100-refusal% -- condition: API, without a system prompt anthropic-syscard | Table 4.3.2.A; condition quoted from the table header: "API, without a system prompt"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (disordered eating), stored as 100-refusal% -- condition: Claude.ai anthropic-syscard | Table 4.3.2.A; condition quoted from the table header: "Claude.ai"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (election integrity), stored as 100-refusal%; Haiku 5.5 is the WORST here (refuses most) -- condition: API, without a system prompt anthropic-syscard | Table 4.4.3.A; NOTE: here Haiku 5.5 over-refuses MORE than every other model incl. Haiku 4.5 (inverse of the general trend in the other four refusal tables) -- genuine data, not an error; condition quoted from the table header: "API, without a system prompt"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (election integrity), stored as 100-refusal%; Haiku 5.5 is the WORST here (refuses most) -- condition: Claude.ai anthropic-syscard | Table 4.4.3.A; NOTE: here Haiku 5.5 over-refuses MORE than every other model incl. Haiku 4.5 (inverse of the general trend in the other four refusal tables) -- genuine data, not an error; condition quoted from the table header: "Claude.ai"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (general, 16 policy areas x 7 languages), stored as 100-refusal% -- condition: API, without a system prompt anthropic-syscard | Table 4.1.2.A, over-refusal rate on benign-but-sensitive-subject prompts; condition quoted from the table header: "API, without a system prompt"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (general, 16 policy areas x 7 languages), stored as 100-refusal% -- condition: Claude.ai anthropic-syscard | Table 4.1.2.A, over-refusal rate on benign-but-sensitive-subject prompts; condition quoted from the table header: "Claude.ai"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (suicide/self-harm), stored as 100-refusal% -- condition: API, without a system prompt anthropic-syscard | Table 4.3.1.A; condition quoted from the table header: "API, without a system prompt"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: single-turn benign-request refusal (suicide/self-harm), stored as 100-refusal% -- condition: Claude.ai anthropic-syscard | Table 4.3.1.A; condition quoted from the table header: "Claude.ai"; read 2026-10-08 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 4.5·solo |
| Haiku 5.5 system card: SWE-bench Multilingual, max effort anthropic-syscard | Table 8.1.A row; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: SWE-bench Multimodal, max effort anthropic-syscard | Table 8.1.A row; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Haiku 5.5 system card: SWE-Bench Pro, max effort anthropic-syscard | Table 8.1.A row; Haiku 5.5 standard config = adaptive thinking at max effort, 5 trials; Haiku 4.5 has no entry in this row (dash); read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Sonnet 5.5·max |
| Haiku 5.5 system card: Terminal-Bench 4.0, safeguards-on, each model's own best-effort config anthropic-syscard | Claude Code --bare mode; Haiku 5.5/Haiku 4.5 10 trials/task (660), Sonnet/Opus 5.5 5 trials/task (330); safeguard-flagged trials counted as failures; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max · Opus 5.5·xhigh |
| Haiku 5.5 system card: Terminal-Bench-Science 0.1, safeguards-on, max effort anthropic-syscard | Claude Code --bare mode, max thinking effort for all except Haiku 4.5 (fixed budget); 700 trials for Haiku 5.5/4.5; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·max · Opus 5.5·max |
| Haiku 5.5 system card: WANDR soft F1 score+cost by effort, 5 efforts anthropic-syscard | chart image, y-axis printed score labels per point; x-axis (cost, log scale) ticks $0.005,$0.01,$0.02,$0.05,$0.10,$0.20,$0.50,$1,$2,$5,$10,$20 give calibration x=1230.9+457.94*log10(cost) (fit from $0.005 and $20 ticks, cross-checked exact at $1 and $0.10); several Opus/Sonnet 'Med'/'High' markers were partly occluded by a crossing line of another series and their centroid is a merge of fragment pixels (lower confidence, noted in notes.md); Haiku 4.5's point not confidently isolated, left without cost; read 2026-10-08 ✓ | xmodel | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo |
| hama-jp: coastal Viking village (voxel) youtube-hama-jp | Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·high · Sonnet 5.5·high |
| hama-jp: five-story pagoda garden (three.js) youtube-hama-jp | Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·high · Sonnet 5.5·high |
| hama-jp: model railway with interlocking (3h cap) youtube-hama-jp | Claude Code with Playwright MCP; same prompt; 1 run each; effort high (Haiku 4.5 n/a); cost = Claude Code converted amount; subscription; 8 Oct 2026; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·high · Sonnet 5.5·high |
| Hanabi convention-inference accuracy (all teammates), clone teams of one model arxiv-2610.19051 | 250 fresh Hanabi games, 4 samples per model per game, temperature 1; Table 2 of arXiv 2610.19051; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Sonnet 5.5·default · Opus 5.5·default |
| Harbor-Index 1.0 (82 tasks), Terminus-2 harness for all models arxiv-2609.04298 | Harbor-Index 1.0: 82 tasks selected from 54 Harbor benchmarks, 3 trials, harness Terminus-2 for every model; pass rate % and approximate per-run cost (USD) of one full Index run; provider default settings (paper: reasoning effort at provider default); models other than GPT-5.5/Opus 4.8/Gemini 3.1 Pro served via OpenRouter; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| Harbor-Index 1.0 (82 tasks), vendor-native harness (Codex / Claude Code / Gemini CLI; Claude Code for open models) arxiv-2609.04298 | Harbor-Index 1.0: 82 tasks, 3 trials; harness = vendor-native (Codex for GPT, Claude Code for Claude, Gemini CLI for Gemini, Claude Code for six OpenRouter-served models); pass rate % and approximate per-run cost (USD); read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| Harbor-Index adapter AA-LCR, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AA-LCR evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter AA-LCR, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AA-LCR evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Aider Polyglot, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Aider Polyglot evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Aider Polyglot, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Aider Polyglot evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter AIME 24 & 25, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AIME 24 & 25 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter AIME 24 & 25, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AIME 24 & 25 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter AlgoTune, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AlgoTune evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter AlgoTune, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter AlgoTune evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter ARC-AGI-2, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ARC-AGI-2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter ARC-AGI-2, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ARC-AGI-2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter BFCL, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BFCL evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter BFCL, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BFCL evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter BigCodeBench-Hard, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BigCodeBench-Hard evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter BigCodeBench-Hard, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BigCodeBench-Hard evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter BIX-Bench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BIX-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter BIX-Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter BIX-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CodePDE, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CodePDE evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CodePDE, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CodePDE evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CompileBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CompileBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CompileBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CompileBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CRUST-Bench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CRUST-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CRUST-Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CRUST-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CyberGym, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CyberGym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter CyberGym, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter CyberGym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter DA-Code, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter DA-Code evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter DA-Code, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter DA-Code evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter DeepSynth, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter DeepSynth evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter DeepSynth, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter DeepSynth evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter FeatureBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter FeatureBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter FeatureBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter FeatureBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter FinanceAgent, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter FinanceAgent evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter FinanceAgent, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter FinanceAgent evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GAIA, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GAIA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GAIA, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GAIA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GAIA2, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GAIA2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GAIA2, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GAIA2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GPQA Diamond, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GPQA Diamond evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GPQA Diamond, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GPQA Diamond evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GSO, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GSO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter GSO, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter GSO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter HumanEvalFix, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter HumanEvalFix evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter HumanEvalFix, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter HumanEvalFix evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Humanity’s Last Exam, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Humanity’s Last Exam evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Humanity’s Last Exam, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Humanity’s Last Exam evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter IneqMath, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter IneqMath evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter IneqMath, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter IneqMath evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter KUMO, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter KUMO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter KUMO, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter KUMO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter LAB-Bench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LAB-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter LAB-Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LAB-Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter LawBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LawBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter LawBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LawBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter LiveCodeBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LiveCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter LiveCodeBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter LiveCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter MedAgentBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MedAgentBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter MedAgentBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MedAgentBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter MMAU, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MMAU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter MMAU, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MMAU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter MMMLU, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MMMLU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter MMMLU, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter MMMLU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Omni-Math, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Omni-Math evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Omni-Math, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Omni-Math evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter PIXIU, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter PIXIU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter PIXIU, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter PIXIU evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter QCircuitBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter QCircuitBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter QCircuitBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter QCircuitBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter QuixBugs, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter QuixBugs evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter QuixBugs, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter QuixBugs evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Reasoning Gym, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Reasoning Gym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Reasoning Gym, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Reasoning Gym evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter ReplicationBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ReplicationBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter ReplicationBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ReplicationBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter ResearchCodeBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ResearchCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter ResearchCodeBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter ResearchCodeBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SciCode, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SciCode evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SciCode, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SciCode evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Seal-0, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Seal-0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Seal-0, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Seal-0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SimpleQA, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SimpleQA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SimpleQA, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SimpleQA evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SkillsBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SkillsBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SkillsBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SkillsBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SLDBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SLDBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SLDBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SLDBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Spider 2, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Spider 2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Spider 2, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Spider 2 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SpreadsheetBench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SpreadsheetBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SpreadsheetBench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SpreadsheetBench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter StrongReject, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter StrongReject evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter StrongReject, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter StrongReject evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-Bench Pro, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Bench Pro evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-Bench Pro, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Bench Pro evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-bench Verified, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-bench Verified evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-bench Verified, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-bench Verified evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-Bench-Multilingual, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Bench-Multilingual evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-Bench-Multilingual, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Bench-Multilingual evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-Lancer, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Lancer evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-Lancer, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-Lancer evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-smith, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-smith evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWE-smith, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWE-smith evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWT Bench, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWT Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter SWT Bench, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter SWT Bench evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Terminal-Bench 2.0, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Terminal-Bench 2.0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter Terminal-Bench 2.0, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter Terminal-Bench 2.0 evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter USACO, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter USACO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter USACO, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter USACO evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter WideSearch, harness Claude Code (3 trials) arxiv-2609.04298 | Harbor benchmark adapter WideSearch evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Claude Code, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harbor-Index adapter WideSearch, harness Terminus-2 (3 trials) arxiv-2609.04298 | Harbor benchmark adapter WideSearch evaluated for the Harbor-Index paper (arXiv 2609.04298 Tables 6-8): harness Terminus-2, 3 trials per task, provider default settings; pass fraction x100; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Harness or Model? solve rate on 80 private tasks (corrected costs) arxiv-2609.11987 | Private contamination-controlled suite, frozen pool of 80 tasks, k=2 (C5/C6 k=1), Docker oracle; solve rate = task-mean over pass/fail verdicts (Table 3); cell = harness x model (claude-agent-sdk / codex-sdk / deepagents); cost = corrected list-price cost PER SOLVED TASK (Table 8, revised version 8 Sep 2026); 1200 s ceiling per run; cells differ in harness by design (harness-confound); read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| Harvey LAB-AA (Artificial Analysis), criteria-pass rate, re-scored October 2026 artificialanalysis | AA eval page harvey-lab-aa, harveyLabBreakdown.criteriaPass; live 15 Oct 2026 (re-scored after group aaharvey); read 2026-10-15 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max |
| HealthBench Professional (length-adjusted) (table) openai-gpt6-astra-launch-0929 | HealthBench Professional (length-adjusted); OpenAI GPT-6 Astra post, LIVE copy (Wayback 20261015182235) carrying the Sep 22/29 update banners; later snapshot of the same tables as the Sep-3 groups (cells changed vs Sep 3 in these columns); 'Evaluation scores are the maximum at any effort'; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Fable 5·default · Opus 5·default |
| HealthBench Professional (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | HealthBench Professional; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5·xhigh · Opus 5·max |
| HealthBench Professional (table) openai-gpt56-launch | HealthBench Professional; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| Hebbia citation recall anthropic-partner-hebbia | Opus 5.5 launch page partner quote · rubric coverage, score only ✓ | xmodel | Opus 5.5·default · Opus 5·default |
| hehee9 Maze Bench Tier 1 (30 mazes) hehee9maze-bcd609c | Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 30 mazes (6 sizes x 5, 4x4..18x18), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status completed); read 2026-10-20 ✓ | xmodel | Fable 5·medium · Fable 5.1·medium · Haiku 4.5·req-8k · Haiku 5.5·medium · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium |
| hehee9 Maze Bench Tier 2 (20 mazes) hehee9maze-bcd609c | Maze image (2048x2048) -> single-line move string, per-maze score = 100 x progress x efficiency, 20 mazes (15x15..24x24, 5 each), one run per model-maze, single call, image input, no tools, medium effort / 8K budget when supported; metric official mean score 0-100; cost = dashboard calculateCost = (input tokens x input rate + output tokens x output rate)/1e6 from rates stored per model in the JSON (before cache discounts); live board snapshot of 20 Oct 2026 (github.com/hehee9/maze-bench commit bcd609c, status running); read 2026-10-20 ✓ | xmodel | Fable 5.1·medium · Opus 5.5·medium · Sonnet 5.5·medium |
| Hex DataBench hex.tech | Hex DataBench, 100 analytics tasks in the Hex workspace, rubric-judged · score in %, average cost per task (USD), tokens per task (one total); effort from the row label; Opus 5, Fable 5.1, Fable 5, Opus 4.8, Sonnet 5 at every listed effort (Opus 5.5 in its own group); read 2026-10-01 ✓ | xmodel | Fable 5.1·max · Fable 5.1·xhigh · Fable 5.1·medium · Fable 5.1·high · Opus 5·max · Opus 4.8·xhigh · Opus 5·high · Fable 5·max · Opus 5·xhigh · Opus 4.8·medium · Opus 5·medium · Fable 5.1·low · Fable 5·high · Opus 5·low · Opus 4.8·max · Fable 5·low · Fable 5·medium · Sonnet 5·xhigh · Opus 4.8·high · Sonnet 5·high · Sonnet 5·medium · Sonnet 5·max · Opus 4.8·low · Sonnet 5·low |
| HLE with tools (GLM-5.3 card) zai-hf-glm-5.3 | HLE w/ tools, temperature=1.0 top_p=0.95, 163,840 gen tokens, 300K context with context management, judge GPT-5.6-luna (medium); GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| HMMT Feb 2026 (GLM-5.2 card) zai-hf-glm-5.2 | HMMT Feb 2026, temperature=1.0 top_p=0.95 163,840 tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| HMMT Nov 2025 (GLM-5.2 card) zai-hf-glm-5.2 | HMMT Nov 2025, temperature=1.0 top_p=0.95 163,840 tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| HouseholdBench: household-economics prediction, macro-F1 on categorical targets arxiv-2610.07563 | 32 household tasks, pre-cutoff test split, default reasoning effort; Table 3 of arXiv 2610.07563; read 2026-10-15 ✓ | xmodel | Fable 5.1·default · Opus 5·default · Opus 4.8·default · Sonnet 5·default |
| HSS 388-task subset, Claude Code agent arxiv-2610.08966 | Claude Code harness on the 388-task subset, pass@1; tokens_out = mean thinking tokens per task inside the harness; read 2026-10-15 ✓ | xmodel | Opus 5·default · Fable 5.1·default |
| Humanity's Last Exam (no tools) (table) openai-gpt55-launch | Humanity's Last Exam (no tools); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| Humanity's Last Exam (w/ tools) (table) openai-gpt6-astra-launch | Humanity's Last Exam (w/ tools); OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Fable 5·default · Opus 5·default |
| Humanity's Last Exam (with tools) (table) openai-gpt55-launch | Humanity's Last Exam (with tools); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| Humanity's Sixth Sense (HSS), single-pass API, pass@1 arxiv-2610.08966 | 522 image/video tasks, pass@1 = every rubric criterion met, effort = highest honoured per Table 5; tokens_out = mean thinking + answer tokens per task; read 2026-10-15 ✓ | xmodel | Opus 5.5·xhigh · Fable 5.1·xhigh · Fable 5·max · Opus 5·max · Sonnet 5·max · Opus 4.8·max |
| HumanWill cybersecurity: 100 - false-refusal rate (424 scenarios) hf-humanwill-cybersecurity | source metric is false-refusal rate (%) on 424 cybersecurity scenarios, lower is better; stored here as 100-rate so higher=better like every other score% in this dataset (see confound for the original rate); partial/full withholding counted; 'not everyday refusal rates or an overall safety score' per source; model settings/routes/run-dates vary historically; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·high · Opus 5·default · Fable 5.1·default · Sonnet 4.6·default |
| Hy-MultiTurn (Chinese deep multi-turn dialogue understanding): Importance score hymultiturn-2607.29196 | Hy-MultiTurn leaderboard (Tencent Hunyuan authors; Hy3/Doubao rows are not in the eight providers): six dimensions, final-turn response scored against weighted rubric items; Importance (weighted correct share) = primary metric, mean of three independent rolls; arXiv 2607.29196 (31 Jul 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·high |
| Hypothesis Frontier Challenge64 (N=64): Round-1 direct validity R1 hypfrontier-2608.10843 | Hypothesis Frontier (Batzoglou, independent) Table 1: first-order rule induction with planted formulas, (N=64); R1 = direct Round-1 validity (single prompt, no feedback/repair/simplification; identical in HF and no-symbolic rows); HF Valid/Acc/H-exact are search-method results (not used, rule 8); recovery: one retry at the original reasoning setting then lower; arXiv 2608.10843 (Aug 2026); read 2026-10-20 ✓ | xmodel | Fable 5·medium |
| iam_lima: weekly-report-from-notes task, mean cost/time of 3 runs note-iam_lima | note.com/iam_lima/n/ncfed4a80e005; Claude Code 2.1.284, work-notes with a planted weekday error and a planted arithmetic error, 3 runs per setting, means reported directly by the source; effort-from-config: "effortはAPIの既定にあたるhighにそろえ、Sonnet 5.5だけはClaude Codeの既定のmediumでも回しました"; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Sonnet 5.5·medium · Sonnet 5·high |
| IAmFabian (YouTube, German): arcade of 4 small games, Opus 5.5 alone vs Haiku 5.5 alone youtube-ichbinfabian | Claude Code on Max plan; API price from real tokens; 1 run per team; effort not stated; read 2026-10-15 ✓ | xmodel | Opus 5.5·default · Haiku 5.5·default |
| IdentityBench v1: identity-governance decisions (680 test cases), Claude CLI baseline prompt github-identitybench | rlwillen0121/identity-eval-harness README leaderboard; frozen test split, baseline prompt, provider claude_cli; no effort set; score = grounded accuracy %; cost = API-equivalent USD per 1,000 decisions; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default · Fable 5.1·default · Opus 5.5·default |
| IdentityBench v3-Full: 360 multi-turn stateful IGA episodes (max 12 turns), claude_cli provider, 17 Oct 2026 github-identitybench | rlwillen0121/identity-eval-harness README fad1304 22-model v3_full table; experiments/v3-full.yaml; NEW GROUP (held identitybench-v1 is the 680-case single-prompt v1 suite); read 2026-10-18 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Fable 5.1·default · Haiku 5.5·default |
| IdentityBench v4.1 v41_lite github-identitybench | IdentityBench v4.1 v41_lite: 60 enterprise IGA scenarios x 6 governance categories = 360 category tasks, 1,628 required actions, 12-turn cap (T0 single monolith), seed 20261019, 1 run per model, README of 2026-10-20 (commit 56591d0), published 2026-10-19; metric tasks solved of 360; cost = run cost of the whole suite (billed or estimated); read 2026-10-20 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium · Haiku 5.5·medium |
| IH-Benchmark instruction-hierarchy compliance, conflict scenarios (overall %) ih-benchmark-2607.25987 | IH-Benchmark Table 16 Overall: 2,336 conflict scenarios (S>U and U>T tracks), programmatic checks + 189 LLM-judge scenarios (GPT-5-mini judge), once per model, LiteLLM, max_completion_tokens 8192, provider-default sampling, Feb-May 2026; HiddenLayer; arXiv 2607.25987 (27 Jul 2026); Tables 15/17 and track/domain columns = same runs (rule 8); cost = B.3 representative cost per full-benchmark evaluation; read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| IMOAnswerBench (GLM-5.2 card) zai-hf-glm-5.2 | IMOAnswerBench, temperature=1.0 top_p=0.95 163,840 tokens, judge GPT-5.5 (medium); GLM-5.2 card 2026-09-01; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| Imperial Dragon Harness tournament, Pi runner, 10 project tickets (own draw per arm) github-imperialdragon-1253 | GitHub PR MinhHaDuong/ImperialDragonHarness#1253 (open, head 2905c18b4f on branch results/haiku55-arena-n) adds Haiku 5.5 arm n to the main-branch snapshot (base ea4136d830); 3-judge panel (Gemini 3.1 Pro Preview, Grok 4.3, MiniMax M2.5) scores 0-30; score only ingested (see notes.md: cost scopes are not comparable between arm n and arms d/g); read 2026-10-08 ✓ | xmodel | Haiku 5.5·medium · Opus 5.5·medium · Opus 5.5·low · Sonnet 5.5·medium |
| IMProofBench improofbench.math.ethz.ch | IMProofBench (ETH Zürich), research-level proof writing, 47 questions · score in %; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Opus 4.7·default |
| Incident-Arena: Pass@1 on production incident repair (averaged over all efforts) incident-arena-2610.00648 | Incident-Arena Table 2: 20 tasks (Kubernetes incident repair with planted faults), each agent in its native vendor CLI harness (Codex 0.153.4, Claude Code 2.1.274/2.1.280, Grok Build 1.0.41; GLM-5.3 through Claude Code), 3 trials x each available reasoning level (150 trials per task; Grok 4.7 120), binary reward = outcome + safety gates; Pass@1 AVERAGED over all efforts of a model; cost = mean model API cost per episode at list prices; arXiv 2610.00648 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Opus 5·default |
| IndustryBench Chinese, Final (SV), non-thinking vs thinking arxiv-2605.10267 | IndustryBench (arXiv 2605.10267 v3): 2,049 Chinese industrial-procurement QA items, zero-shot closed-book, raw 0-3 by Qwen3-Max judge minus safety-violation penalty (Final SV, 0-3); outputs collected Feb 2026; Table 7 = default non-reasoning run, Table 8 = same items/judge with thinking enabled; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·nothink |
| InferenceBench (prompt v1.0.1 (†)) inferencebench.ai | InferenceBench, agents tuning LLM inference serving in Claude Code · geometric-mean speed-up over the PyTorch eager baseline across four scenarios, ±SEM over seed-pair runs in the flags; runs with the benchmark prompt v1.0.1 (†) only; effort from the model name; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·default · Fable 5.1·max · Opus 5·default · Fable 5·low · Fable 5·default · Sonnet 5·default |
| InferenceBench (prompt v1.0) inferencebench.ai | InferenceBench, agents tuning LLM inference serving in Claude Code · geometric-mean speed-up over the PyTorch eager baseline across four scenarios, ±SEM over seed-pair runs in the flags; runs with the benchmark prompt v1.0 only; effort from the model name; no cost; read 2026-09-30 ✓ | xmodel | Opus 4.7·default · Opus 4.8·default · Opus 4.8·xhigh · Sonnet 4.6·default · Fable 5·default · Haiku 4.5·solo · Fable 5·low |
| IntelligentDocumentProcessing: 250 synthetic German documents, invoice routing correct (single-model systems) github-idp-invoices | DanielHuette/IntelligentDocumentProcessing README, measured 8 Oct 2026; one shared prompt; no extended thinking for Claude; cost = tokens billed x list price per 1,000 documents; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·nothink · Haiku 5.5·nothink |
| Internal Database Migration Tasks (table) openai-gpt6-astra-launch | Internal Database Migration Tasks; OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Fable 5·default |
| ITBench-AA (SRE) artificialanalysis | Artificial Analysis ITBench-AA (SRE); score only from the model-page payload; effort from payload effort.slug; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max |
| Japanese Stroke LLM Eval arxiv-2609.16739 | multi-turn Japanese conversation · Fable5/Opus4.7 default, score-only ✓ | xmodel | Fable 5·high · Opus 4.7·high |
| JobBench job-bench.github.io | JobBench, professional tasks graded by rubric, every model in OpenCode · overall score in %; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Opus 5·default · Fable 5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default · Opus 5.5·default · Fable 5.1·default |
| JockeyBench Program track (324 races, daily revision agent) arxiv-2610.19589 | Claude Code with MCP bridge; log gain in nats relative to uniform, mean over runs; no model cost (fees are betting score-yen); read 2026-10-15 ✓ | xmodel | Opus 5·high · Fable 5.1·high · Opus 5.5·high |
| K-Bench overall (178 scientific agent requests) kbench-2608.21601 | K-Bench Table 4/18: 178 verbatim requests, pi 0.84.0 harness in Modal sandbox, rubric v1.0, 3 judges (gpt-5.6-sol, qwen3.8-max, grok-4.5) blinded, OpenRouter; campaign 6-12 Aug 2026; arXiv 2608.21601; cost = mean inference $ per task (Table 18); read 2026-10-20 ✓ | xmodel | Opus 5·default |
| K3 report Fig.13b: BrowseComp score vs cost per task arxiv-2607.24653 | BrowseComp; K3 with context compaction at 300K tokens; GPT-5.6 Sol per effort medium/high/max (points read from the vector drawing of Fig.13b); cost USD per task as plotted (source of costs not stated); K3 tech report p.32; read 2026-10-19 ✓ | xmodel | Fable 5·req-max-1m · Fable 5·req-max-3m |
| K3 report Table 3: 24/7 ClawBench 2.0 arxiv-2607.24653 | 24/7 ClawBench 2.0 (Moonshot in-house); harness OpenClaw; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: Agent Behavior Bench arxiv-2607.24653 | Agent Behavior Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: Agentic Vision Bench arxiv-2607.24653 | Agentic Vision Bench (Moonshot in-house); harness Kimi Code; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: Chat All-in-One Bench arxiv-2607.24653 | Chat All-in-One Bench (Moonshot in-house); harness Kimi Work; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: CLIF Bench arxiv-2607.24653 | CLIF Bench (Moonshot in-house); harness Kimi Code; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| K3 report Table 3: Coding Experience [Claude Code harness] arxiv-2607.24653 | Coding Experience (Moonshot in-house); harness Claude Code; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: DECK Bench arxiv-2607.24653 | DECK Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: Deep Research Bench arxiv-2607.24653 | Deep Research Bench (Moonshot in-house); harness Kimi Agent; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| K3 report Table 3: Faithfulness arxiv-2607.24653 | Faithfulness (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| K3 report Table 3: Finance Bench arxiv-2607.24653 | Finance Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| K3 report Table 3: KAET arxiv-2607.24653 | KAET (Moonshot in-house); harness Kimi Code; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| K3 report Table 3: KWV Bench arxiv-2607.24653 | KWV Bench (Moonshot in-house); harness N/A; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: MIRA Bench arxiv-2607.24653 | MIRA Bench (Moonshot in-house); harness MIRA; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: Online Experience arxiv-2607.24653 | Online Experience (Moonshot in-house); harness Kimi Agent; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| K3 report Table 3: Swarm Bench arxiv-2607.24653 | Swarm Bench (Moonshot in-house); harness Kimi Agent; max reasoning effort (GPT-5.5 xhigh); K3 tech report Table 3 (p.29), not on the model card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kaggle Game Arena Chess Opening (Elo) kaggle-game-arena-2609.31473 | Kaggle Game Arena (Google DeepMind / Kaggle) Appendix Table 1 leaderboard, Kaggle Game Arena Chess Opening (Elo): full round-robin among the top-10 models of five labs (Feb 2026), public API endpoints with default sampling settings and generation limits, no tools; platform run by Google (Gemini rows vendor-reported:Google, others run-by-google); arXiv 2609.31473 (Sep 2026); read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo |
| Kaggle Game Arena Chess Text Input (Elo) kaggle-game-arena-2609.31473 | Kaggle Game Arena (Google DeepMind / Kaggle) Appendix Table 1 leaderboard, Kaggle Game Arena Chess Text Input (Elo): full round-robin among the top-10 models of five labs (Feb 2026), public API endpoints with default sampling settings and generation limits, no tools; platform run by Google (Gemini rows vendor-reported:Google, others run-by-google); arXiv 2609.31473 (Sep 2026); read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo |
| Kagi LLM reasoning benchmark (116 own tasks) kagi-llm-benchmark | Kagi help page 'Last task list revision: February 13th, 2026', Tasks 116, Input Tokens 15256; unpublished non-English/noisy-context/instruction tasks; no tools; cost = 'total cost required to parse the entire benchmark suite' in USD, rounded to 0.1; tokens column; live table read 2026-10-19 (rows run at different dates); read 2026-10-19 ✓ | xmodel | Fable 5·solo · Fable 5·default · Opus 4.8·solo · Opus 4.7·solo · Opus 4.7·default |
| kai_kou Haiku 5.5 effort sweep + Haiku 4.5 — yes/no reading check (38,082-tok doc) qiita-kaikou | claude -p --effort low/medium/high (Haiku 5.5) vs Claude Code default (Haiku 4.5); same 38,082-token Japanese document, same yes/no question (correct answer いいえ); score = correct% of trials run at that effort, trial count n given per row in confound (low n=2, others n=1); read 2026-10-08 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 4.5·solo |
| kaloyan.blog scenes-cli: rebuild a photo in Blender, Opus 5.5 vs Sonnet 5.5 kaloyan-blog | model writes/runs Blender Python to rebuild 3 photos, bench renders the saved scene and scores calibrated silhouette/colour/detail/mesh (0-100), 1 run(s)/image, created_at=2026-10-01T07:27:51Z; https://kaloyan.blog/content/eleven-models-in-blender/assets/benchmark-results-newest.json, src/kaloyan_results.json; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default |
| kamiumi_ai: 5 SVG prompts x3 runs, total cost/time, Sonnet 5.5 vs Opus 5.5 note-kamiumi_ai | note.com/kamiumi_ai/n/nb7bdc208736e; Claude Code 2.1.284 non-interactive (claude -p-equivalent), no tools, 4-at-a-time parallel execution (so time only comparable as ratio); effort not specified -> Claude Code default (quoted: "effortは指定しておらず、Claude Codeの既定（medium）で動いています"); cost = Claude Code execution record at API list price, totals over 5 tasks x3 runs = 15 runs per model; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| kamiumi_ai: blind-judged SVG quality by category, Sonnet 5.5 vs Opus 5.5 note-kamiumi_ai | note.com/kamiumi_ai/n/nb7bdc208736e; same 15-runs-per-model experiment as kamiumi-svg-cost; blind grading (model name hidden), two judges (Opus 5.5 and Sonnet 5.5); score given is the Opus-5.5-judge score (primary column in report), Sonnet-5.5-judge score quoted in confound; judge=Claude grading Claude -> claude-judge; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| KC-Bench: success score, mean of 3 domains arxiv-2609.03588 | KC-Bench (arXiv 2609.03588 v1 3 Sep 2026) Table 2: knowledge-conflict agent benchmark, domains Region / Retail / Personal Assistant, agent + task-driven user simulator + assertion model, greedy decoding temperature 0; cell = automatic LLM evaluation / manual verification; score = mean over 3 domains of the AUTOMATIC value (computed by the verifier; the table prints no overall); read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo |
| kenji_harada Sonnet 5.5 article: 6 small Node.js tasks x 3, effort high qiita-kenji-harada | claude -p --effort high (2.1.283/2.1.285) on 28 and 30 Sep; hidden tests; median per run; cost = API list-price conversion; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·high · Sonnet 5·high · Opus 5.5·high · Opus 5·high · Fable 5.1·high |
| kenji_harada: 4 hard coding tasks, Opus 5.5 sweep + Fable 5.1 (Claude Code) qiita-kenji-harada | Claude Code 2.1.283 -p, macOS, 4 hard Node.js tasks x3 runs, median time/cost at API list price; qiita.com/kenji_harada/items/7d8be259208cd3a5843f; read 2026-10-05 ✓ | xmodel | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Fable 5.1·high |
| kenji_harada: 6 small coding tasks, Opus 5.5/Opus 5/Fable 5.1 (Claude Code) qiita-kenji-harada | Claude Code 2.1.283 -p, macOS, 6 small Node.js tasks x3 runs, median time/cost at API list price; qiita.com/kenji_harada/items/7d8be259208cd3a5843f; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Opus 5.5·medium · Opus 5.5·low · Opus 5·high · Fable 5.1·high |
| KernelBench CUDA 01 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 01, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·max |
| KernelBench CUDA 01 (OpenRouter) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 01, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Fable 5·default · Fable 5.1·max · Opus 5·max |
| KernelBench CUDA 02 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 02, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·max |
| KernelBench CUDA 02 (OpenRouter) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 02, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Fable 5·default · Fable 5.1·max · Opus 5·max |
| KernelBench CUDA 03 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 03, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·max |
| KernelBench CUDA 03 (OpenRouter) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 03, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Fable 5·default · Fable 5.1·max · Opus 5·max |
| KernelBench CUDA 04 (Claude Code) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 04, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Opus 4.8·max · Opus 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·max |
| KernelBench CUDA 04 (OpenRouter) github-kernelbench | Infatoshi/kernelbench.com 817ff9c · CUDA kernel optimisation problem 04, RTX PRO 6000, fraction of hardware peak, effort from the board's label, n=1, no cost ✓ | xmodel | Fable 5·default · Fable 5.1·max · Opus 5·max |
| KernelBench hard 01_fp8_gemm github-kernelbench | kernelbench.com hard board problem 01_fp8_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Fable 5·max |
| KernelBench hard 03_paged_attention github-kernelbench | kernelbench.com hard board problem 03_paged_attention; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| KernelBench hard 05_topk_bitonic github-kernelbench | kernelbench.com hard board problem 05_topk_bitonic; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 5·max |
| KernelBench hard 06_sonic_moe_swiglu github-kernelbench | kernelbench.com hard board problem 06_sonic_moe_swiglu; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Fable 5·default |
| KernelBench hard 07_w4a16_gemm github-kernelbench | kernelbench.com hard board problem 07_w4a16_gemm; RTX PRO 6000 Blackwell; v2_containerized; 1 run per model; agentic CUDA/kernel optimisation; metric peak_fraction of hardware peak; snapshot master 71f7c3db, 2026-10-18; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 5·max · Opus 4.8·max |
| KernelBench MEGA (Claude Code) github-kernelbench | 02_kimi_linear_decode megakernel on RTX PRO 6000 Blackwell · harness=claude (Claude Code) · n=1 per cell, runs months apart (Jun-Sep 2026); NOT the board's CUDA task (that is the held kernelbench-cc/or group); read 2026-10-01 ✓ | xmodel | Opus 5.5·xhigh · Sonnet 5.5·max · Fable 5·max · Fable 5.1·max · Opus 4.8·default · Sonnet 5·max |
| KernelBench MEGA (or-fable, OpenRouter) github-kernelbench | 02_kimi_linear_decode megakernel on RTX PRO 6000 Blackwell · harness=or-fable (OpenRouter agent, different harness from Claude Code) · n=1 per cell, not comparable to the claude-harness rows; read 2026-10-01 ✓ | xmodel | Fable 5·default · Fable 5.1·max |
| kikuziro inquiry classifier (6 prompts x 10 runs), Haiku 5.5 vs Haiku 4.5 qiita-kikuziro | API max_tokens=1 (5.5) / 2 (4.5), thinking disabled, number-only answer; accuracy over 60 runs; read 2026-10-15 ✓ | xmodel | Haiku 5.5·nothink · Haiku 4.5·solo |
| KillSwitch-Bench killswitch-bench | Claude Code, esoteric adversarial language · 5 tasks × 2 attempts, $5 cap, default effort, cost = tokens × list ✓ | xmodel | Sonnet 5.5·default · Sonnet 5·default · Opus 5.5·default · Opus 5·default · Fable 5.1·default |
| kilobench kilo | config ✓ | xmodel | Opus 5·default · Fable 5·default · Opus 4.7·default · Opus 4.8·default · Sonnet 5·default · Sonnet 4.6·default · Fable 5.1·default |
| Kimi Code Bench 2.0 arxiv-2607.24653 | arXiv 2607.24653 Fig. 13a · Moonshot in-house suite via Claude Code, Opus 4.8 vs Fable 5 at max, measured cost (vector-read) ✓ | xmodel | Opus 4.8·max · Fable 5·max |
| Kimi K2.7 Code card: Kimi Claw 24/7 Bench moonshot-hf-kimi-k2.7-code | Kimi Claw 24/7 Bench; in-house; 17 scenarios/610 evaluation points; OpenClaw harness; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Opus 4.8·xhigh |
| Kimi K2.7 Code card: Kimi Code Bench v2 moonshot-hf-kimi-k2.7-code | Kimi Code Bench v2; in-house; 10+ languages; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Opus 4.8·xhigh |
| Kimi K2.7 Code card: MCP Atlas moonshot-hf-kimi-k2.7-code | MCP Atlas; 100 tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Opus 4.8·xhigh |
| Kimi K2.7 Code card: MCP Mark Verified moonshot-hf-kimi-k2.7-code | MCP Mark Verified; human-verified MCPMark; 100-step tool-call budget; 32k max tokens per step; avg of 3 runs; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Opus 4.8·xhigh |
| Kimi K2.7 Code card: MLS Bench Lite moonshot-hf-kimi-k2.7-code | MLS Bench Lite; 30-task subset; 5 h per task; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kimi K2.7 Code card: Program Bench moonshot-hf-kimi-k2.7-code | Program Bench; 200 tasks; 248k fuzz tests; general setting: temperature 1.0, top-p 0.95, 262,144 context; Kimi K2.7 Code model card (HF created 2026-06-11); read 2026-10-19 ✓ | xmodel | Opus 4.8·xhigh |
| Kimi K3 card: AA-Briefcase (Elo) moonshot-hf-kimi-k3 | AA-Briefcase Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: AA-LCR moonshot-hf-kimi-k3 | AA-LCR; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: Agents' Last Exam moonshot-hf-kimi-k3 | Agents' Last Exam; official leaderboard as of 2026-07-23, primary pass-rate metric; per-model harness from the leaderboard; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·xhigh |
| Kimi K3 card: APEX-Agents moonshot-hf-kimi-k3 | APEX-Agents; Mercor leaderboard as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: AutomationBench moonshot-hf-kimi-k3 | AutomationBench; 600-task public subset; official GitHub setup; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: BabyVision w/ python moonshot-hf-kimi-k3 | BabyVision with Python; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: CharXiv (RQ) (no tools) moonshot-hf-kimi-k3 | CharXiv reasoning questions; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: FrontierSWE moonshot-hf-kimi-k3 | FrontierSWE; dominance scores recomputed from raw scores, current as of 2026-07-16; K3 Kimi Code, Sol Codex; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: GDPval-AA v2 (Elo) moonshot-hf-kimi-k3 | GDPval-AA v2 Elo; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: GPQA Diamond moonshot-hf-kimi-k3 | GPQA Diamond; no tools; top-p 0.95; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kimi K3 card: HLE-Full (no tools) moonshot-hf-kimi-k3 | HLE full set; top-p 0.95; with-tools = general tools; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| Kimi K3 card: MathVision (no tools) moonshot-hf-kimi-k3 | MathVision; avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: MathVision (with python) moonshot-hf-kimi-k3 | MathVision; avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 0.95 for single-step tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: MCP-Atlas moonshot-hf-kimi-k3 | MCP-Atlas; 500-task public subset; 100-turn limit; judge Gemini 3.1 Pro; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: MCPMark-Verified moonshot-hf-kimi-k3 | MCPMark-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| Kimi K3 card: MLS-Bench-Lite moonshot-hf-kimi-k3 | MLS-Bench-Lite (30-task subset); Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| Kimi K3 card: MMMU-Pro (no tools) moonshot-hf-kimi-k3 | MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kimi K3 card: MMMU-Pro (with python) moonshot-hf-kimi-k3 | MMMU-Pro official protocol (input order kept; images first); avg of 3 runs; with python; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: MMVU moonshot-hf-kimi-k3 | MMVU; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kimi K3 card: OfficeQA Pro moonshot-hf-kimi-k3 | OfficeQA Pro; entire PDF corpus given as images, no machine-readable text; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: OmniDocBench moonshot-hf-kimi-k3 | OmniDocBench; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: PerceptionBench moonshot-hf-kimi-k3 | PerceptionBench (Moonshot in-house benchmark); avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: PostTrainBench moonshot-hf-kimi-k3 | PostTrainBench; K3/Fable/Sol official Harbor implementation at max effort, avg of 3 runs on H20 (official: H100); GLM/GPT-5.5/Opus from the official board; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: ProgramBench moonshot-hf-kimi-k3 | ProgramBench; K3 with Kimi Code harness; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: ResearchRubrics moonshot-hf-kimi-k3 | ResearchRubrics; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kimi K3 card: SaaS-Bench moonshot-hf-kimi-k3 | SaaS-Bench; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kimi K3 card: SciCode moonshot-hf-kimi-k3 | SciCode; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: SpreadsheetBench 2 moonshot-hf-kimi-k3 | SpreadsheetBench 2; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: SWE-Marathon moonshot-hf-kimi-k3 | SWE-Marathon, H20-calibrated branch (card: official tasks as of 2026-07-09 prior to final v1.1; blog/report: v1.1 tasks); K3/Opus/Fable Claude Code, Sol Codex; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: Toolathlon-Verified moonshot-hf-kimi-k3 | Toolathlon-Verified; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: Video-MME (w. sub) moonshot-hf-kimi-k3 | Video-MME with subtitles; avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Kimi K3 card: WorldVQA ForceAnswer moonshot-hf-kimi-k3 | WorldVQA ForceAnswer variant; vision avg of 3 runs; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kimi K3 card: ZeroBench (pass@5) (no tools) moonshot-hf-kimi-k3 | ZeroBench main set; pass@5 (run 5 times, official setting); no tools; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| Kimi K3 card: τ³-Banking moonshot-hf-kimi-k3 | tau3-Banking; board snapshot as of 2026-07-23; Moonshot Kimi K3 model card table 2026-07 (card snapshot read; HF lastModified 2026-09-02); reasoning effort max, temperature 1.0, top-p 1.0 for agentic tasks; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max |
| Kingy 30-case kingy.ai | Vercel AI Gateway pinned to Anthropic · 30 identical cases, measured ledger; quality saturated → cost only ✓ | xmodel | Fable 5.1·low · Opus 5·high · Sonnet 5·high |
| KJL waterslide game github-kjlkurt | Claude Code xhigh · one-shot game build, Sonnet 5.5 vs Opus 5.5, cost only ✓ | xmodel | Sonnet 5.5·xhigh · Opus 5.5·xhigh |
| KlinikeBench (2609.38480) — 333 clinical tasks, Harbor/Terminus-2 ReAct, GPT-5.4-mini patient+judge arxiv-2609.38480 | Table 7 strict pass@1 and est. $/task (text for Opus 5/5.5, Fig. 10 digitized for the rest); verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5·medium · Opus 5.5·medium · Opus 4.8·medium · Sonnet 5·medium · Sonnet 4.6·medium · Haiku 4.5·solo |
| Knows When to Stop (KWTS) healthcare admin dataset: 90 cases, Anthropic tool-use runner github-kwts-healthcare | tylergibbs1/kwts-healthcare-dataset results.json (9 Oct 2026); 90 cases (30 finish/stop/distractor triplets), 8 responses & 12 actions cap, 2048 output tokens per response; one repetition per model; score = passed cases %; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·high · Opus 5.5·medium · Haiku 4.5·solo |
| krauq.ai SCAN 2026 CTF qualifier (55 flags, 1 attempt each) krauq.ai | Claude Code (builds 2.1.273/2.1.284/2.1.293), 55 blockchain-investigation CTF flags, one attempt per flag, 20-min limit; score = flags solved of 55; cost = API-equivalent total of the 55 attempts (sum of per-attempt cost_usd; estimated attempts flagged); read 2026-10-15 ✓ | xmodel | Haiku 4.5·req-low · Haiku 4.5·req-medium · Haiku 4.5·req-high · Haiku 4.5·req-xhigh · Haiku 4.5·req-max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5·low · Fable 5·medium · Fable 5.1·low · Fable 5.1·medium |
| Labelbox · Implicit Intelligence labelbox.com | Labelbox leaderboards, Implicit Intelligence · score as printed (integer), top five models only; effort from the label; read 2026-09-30 ✓ | xmodel | Opus 4.7·high |
| LabOPBench multimodal laboratory decision making: overall normalised score labopbench-2610.19780 | LabOPBench Table 2: eight multimodal models, normalised scores (%) across categories (Preparation, Monitoring, Postprocessing, Characterization, Materials Science) and Overall; category columns = same runs (rule 8); arXiv 2610.19780 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| latitude easy tier github-latitude | ai-sdk · 12 single-file bug tasks ×3, 3 Claude models at default effort, list price ✓ | xmodel | Sonnet 5·high · Opus 4.8·default · Fable 5·high |
| latitude-cross6 github-latitude | config ✓ | xmodel | Sonnet 5·high · Opus 4.8·default · Fable 5·high |
| latitude-sweep github-latitude | config ✓ | xmodel | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| Law and Order: TY2025 tax-return autoformalization, method = direct translation arxiv-lawandorder | arxiv.org/html/2610.02792v1 Table 1; correct returns (strict) %; no effort token given for the Claude rows (only GPT-5.5 is marked Medium) -> default; small task set (n~40, strict deltas in multiples of 2.5pp); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default |
| Law and Order: TY2025 tax-return autoformalization, method = Law&Order (localized execution feedback) arxiv-lawandorder | arxiv.org/html/2610.02792v1 Table 1; correct returns (strict) %; no effort token given for the Claude rows (only GPT-5.5 is marked Medium) -> default; small task set (n~40, strict deltas in multiples of 2.5pp); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default |
| Law and Order: TY2025 tax-return autoformalization, method = self-debugging arxiv-lawandorder | arxiv.org/html/2610.02792v1 Table 1; correct returns (strict) %; no effort token given for the Claude rows (only GPT-5.5 is marked Medium) -> default; small task set (n~40, strict deltas in multiples of 2.5pp); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default |
| Law and Order: TY2025 tax-return autoformalization, method = w/o formalization arxiv-lawandorder | arxiv.org/html/2610.02792v1 Table 1; correct returns (strict) %; no effort token given for the Claude rows (only GPT-5.5 is marked Medium) -> default; small task set (n~40, strict deltas in multiples of 2.5pp); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default |
| Law and Order: TY2025 tax-return autoformalization, method = w/o formalization + web search arxiv-lawandorder | arxiv.org/html/2610.02792v1 Table 1; correct returns (strict) %; no effort token given for the Claude rows (only GPT-5.5 is marked Medium) -> default; small task set (n~40, strict deltas in multiples of 2.5pp); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default |
| lcb-baseline github-llmcost2026 | config ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| lcb-batch github-llmcost2026 | config ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| lechmazur Adversarial LLM Debate Benchmark — Bradley-Terry rating lechmazur-debate | lechmazur/debate, side-swapped adversarial debates, multi-model judge panel (incl. a Claude judge), Bradley-Terry rating with match counts; read 2026-10-05 ✓ | xmodel | Fable 5.1·high · Fable 5·high · Opus 5·high · Opus 5.5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high · Sonnet 4.6·high · Sonnet 4.6·nothink |
| lechmazur Creative Writing Benchmark — comparison score lechmazur-writing | lechmazur/writing, matched creative-brief stories judged pairwise by rotating LLM evaluators, Thurstone/BT-style comparison score centered at 0, bootstrap 95% CI; read 2026-10-05 ✓ | xmodel | Fable 5.1·high · Opus 5·xhigh · Opus 5.5·high · Opus 5·high · Fable 5·high · Opus 4.7·high · Sonnet 4.6·think-16k · Opus 4.8·xhigh · Opus 4.8·high |
| lechmazur NYT Connections Extended Benchmark — full 940-puzzle board lechmazur-nyt | lechmazur/nyt-connections, 940 puzzles, quadratic-v1 scoring (g/4)^2 per puzzle, objective (not model-judged); read 2026-10-05 ✓ | xmodel | Opus 5·xhigh · Fable 5·high · Opus 5·high · Opus 4.8·xhigh · Fable 5.1·high · Opus 5.5·high · Opus 4.8·high · Opus 5.5·medium · Sonnet 4.6·high · Sonnet 4.6·think-32k · Sonnet 5·high · Sonnet 4.6·think-16k · Sonnet 4.6·nothink · Opus 4.7·high · Haiku 4.5·solo · Opus 4.7·nothink · Sonnet 5.5·high · Haiku 5.5·high |
| lechmazur NYT Connections Extended Benchmark — harder 100-puzzle subset lechmazur-nyt | lechmazur/nyt-connections, harder 100-puzzle subset, same quadratic-v1 scoring, objective; read 2026-10-05 ✓ | xmodel | Opus 5·xhigh · Fable 5·high · Opus 4.8·high · Opus 5·high · Opus 4.8·xhigh · Opus 5.5·high · Fable 5.1·high · Opus 5.5·medium · Sonnet 4.6·high · Sonnet 5·high · Sonnet 4.6·think-32k · Sonnet 4.6·think-16k · Sonnet 5.5·high · Haiku 5.5·high |
| LegalForecastBench, motion-to-dismiss claim forecasting (johnhughes3/LegalForecastBench) johnhughes3/LegalForecastBench | 91 federal MTD cases / 387 claim-defendant units, agentic document access, no web; score is micro-Brier (lower is better, flagged accordingly); costs are heterogeneous across models per the paper's own caveat (cost_usd_estimate column of figure1_brier_ranked.csv, whole-run not per-case) and the Opus 5.5 cost is unavailable ("basis": "unavailable" in both exp_opus55.json and the source export used for the sonnet row); read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Fable 5.1·default · Opus 5·default · Sonnet 5·default |
| LHTB v1 (46 tasks): mean reward lhtb | Long-Horizon-Terminal-Bench v1, 46 tasks, 9 categories; Terminus-2 harness via Harbor identical for all models; one 90-minute attempt per task; hidden verifiers, errors = 0; cost = per-task spend in USD; July 2026 data; page saved 2026-10-19. Separate from LHTB 2.0 (v2 groups); read 2026-10-19 ✓ | xmodel | Sonnet 5·default · Opus 4.8·default · Fable 5·default · Sonnet 4.6·default |
| LIBERO-Agent (30-task embodied-manipulation suite), general-purpose coding agents arxiv-2609.39507 | LIBERO-Agent (arXiv 2609.39507, 30 Sep 2026): 30-task suite, 3 rollouts per task, full observation profile, no demonstrations, 1800 s per rollout; each model in its own agent harness (Table 5); score = Overall Score (Table 1, 0-100); judgement accuracy, short/long-horizon SR and SC are other columns of the same runs; avg time and tokens reported, no dollar cost; read 2026-10-20 ✓ | xmodel | Opus 5·high · Fable 5.1·high |
| LibraryDesignBench (2609.36730) — 15 library-design problems, 3 implementer models solve them arxiv-2609.36730 | Table 1 designers: score (correctness x simplicity composite), $/problem; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high |
| LifeSciBench (OpenAI GPT-5.6 post chart) openai-gpt56-launch | LifeSciBench, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| Lighthouse SVG one-shot (Reddit) reddit-lighthouse | one prompt, Copilot AIU, Opus capped at stop, subjective rank ✓ | xmodel | Opus 5.5·xhigh · Fable 5.1·xhigh |
| LiveBench livebench | new-livebench CSVs · 2026-06-25 release, 13 Claude configs (max/xhigh/high/medium), measured tokens × rates, global = mean of 7 category means ✓ | xmodel | Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 5.5·max · Opus 5.5·xhigh · Opus 4.8·max · Sonnet 5·xhigh · Opus 4.7·xhigh · Sonnet 4.6·medium · Sonnet 5.5·max · Sonnet 5.5·xhigh · Haiku 5.5·xhigh · Haiku 5.5·max |
| LiveBench 2.0 · Browser Use suite livebench | LiveBench 2.0 Browser Use suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Haiku 4.5·req-max · Haiku 5.5·max · Opus 4.8·max · Opus 5.5·max · Opus 5·max · Sonnet 4.6·max · Sonnet 5.5·max · Sonnet 5·max |
| LiveBench 2.0 · Coding suite livebench | LiveBench 2.0 Coding suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Haiku 4.5·req-max · Haiku 5.5·max · Opus 4.8·max · Opus 5.5·max · Opus 5·max · Sonnet 4.6·max · Sonnet 5.5·max · Sonnet 5·max |
| LiveBench 2.0 · Data Analysis suite livebench | LiveBench 2.0 Data Analysis suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Haiku 4.5·req-max · Haiku 5.5·max · Opus 4.8·max · Opus 5.5·max · Opus 5·max · Sonnet 4.6·max · Sonnet 5.5·max · Sonnet 5·max |
| LiveBench 2.0 · Design suite livebench | LiveBench 2.0 Design suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Haiku 4.5·req-max · Haiku 5.5·max · Opus 4.8·max · Opus 5.5·max · Opus 5·max · Sonnet 4.6·max · Sonnet 5.5·max · Sonnet 5·max |
| LiveBench 2.0 · Long-Horizon suite livebench | LiveBench 2.0 Long-Horizon suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Haiku 4.5·req-max · Haiku 5.5·max · Opus 4.8·max · Opus 5.5·max · Opus 5·max · Sonnet 4.6·max · Sonnet 5.5·max · Sonnet 5·max |
| LiveBench 2.0 · Reasoning suite livebench | LiveBench 2.0 Reasoning suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Haiku 4.5·req-max · Haiku 5.5·max · Opus 4.8·max · Opus 5.5·max · Opus 5·max · Sonnet 4.6·max · Sonnet 5.5·max · Sonnet 5·max |
| LiveBench 2.0 · Research suite livebench | LiveBench 2.0 Research suite, single run, every model at its highest effort (max), cost = suite total at list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Haiku 4.5·req-max · Haiku 5.5·max · Opus 4.8·max · Opus 5.5·max · Opus 5·max · Sonnet 4.6·max · Sonnet 5.5·max · Sonnet 5·max |
| livedoc v3 (collaborative editor): Haiku 5.5 vs Opus 5.5 in Claude Code github-esagduyu-livedoc | Claude Code 2.1.300, 3 runs per arm, 51 hidden checks (all arms 51/51: cost-only), cost = CLI total_cost_usd per run; read 2026-10-18 ✓ | xmodel | Haiku 5.5·default · Opus 5.5·default |
| LLM A-share trading simulation (mean return) github-llm-quant-trading | CollinsLiu615/llm-quant-trading (GitHub, commits 30 Sep 2026): simulated A-share market (GARCH, T+1), 15 stocks, 180 trading days, 100,000 CNY, 2 seeds (Claude Sonnet 5: 1 seed); score = mean total return %; individual hobby benchmark; read 2026-10-20 ✓ | xmodel | Sonnet 5·default |
| LLM cost bench compression github-llmcost2026 | raw API · 80 prompts, prompt-compression lever, Haiku 4.5 vs Sonnet 4.6 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| LLM cost bench output cap github-llmcost2026 | raw API · 80 prompts, output-cap lever ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| LLM judge accuracy on the alignment set (439 DPO pairs), original prompts arxiv-2610.18122 | pairwise verdict accuracy vs dataset label, CALM prompt, reasoning off; cost = US$ per 1,000 requests over the full 15,195-request design; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·nothink · Haiku 5.5·nothink |
| LLM judge accuracy on the fact-related set, original prompts arxiv-2610.18122 | pairwise verdict accuracy vs correct answer, CALM prompt, reasoning off; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·nothink · Haiku 5.5·nothink |
| llm_chess — aggregated Elo vs Random + Dragon engine (all Claude models) llm_chess | maxim-saplin/llm_chess, LLM plays Black vs RANDOM_PLAYER and Dragon engine, Elo anchored to Dragon ladder, 95% CI, average_game_cost as cost per game; read 2026-10-05 ✓ | xmodel | Opus 5·high · Sonnet 5·high · Opus 5.5·high · Sonnet 4.6·high · Opus 4.8·high · Haiku 4.5·req-16k · Sonnet 5·default · Haiku 4.5·nothink · Sonnet 4.6·default · Opus 4.7·high · Sonnet 5.5·high · Haiku 5.5·high |
| llm_chess — single 10-game match vs RANDOM_PLAYER (Black win rate) llm_chess | maxim-saplin/llm_chess raw per-match _aggregate_results.json, n=10 games per model, win% computed by us from black_wins/total_games; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high |
| LLM-generated GP parent-selection operators, test benchmarks, mean over mean arxiv-2607.23505 | zero-shot GP parent-selection operator written by each LLM, N=200 G=100, test set, mean-over-mean score of 1; paper 2026-07-26; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| LLM-generated GP parent-selection operators, train benchmarks, mean over mean arxiv-2607.23505 | zero-shot GP parent-selection operator written by each LLM, N=200 G=100, train set, mean-over-mean score of 1; paper 2026-07-26; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| llm-pixel-duel: one platformer prompt (cost and tokens) github-llm-pixel-duel | neurostack-lab/llm-pixel-duel (GitHub, pushed 17 Oct 2026): one prompt (8-bit platformer in one HTML file, autopilot), one run, max_tokens 64000, thinking medium; cost only (no quality score: both games load and the autopilot plays); read 2026-10-20 ✓ | xmodel | Opus 5.5·medium |
| llm-regression product suite 2026-10-03 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-03, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Opus 5.5·high |
| llm-regression product suite 2026-10-04 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-04, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Opus 5.5·high |
| llm-regression product suite 2026-10-05 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-05, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Opus 5.5·high |
| llm-regression product suite 2026-10-06 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-06, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Opus 5.5·high |
| llm-regression product suite 2026-10-07 (quality /10) nikoamoretti-llm-regression | nikoamoretti/llm-regression results branch 1661fbfc (data.json 2026-10-07T10:17Z): daily product suite 2026-10-07, 7-8 private coding tasks (same task list on both tracks), 1 trial, hidden tests, quality-v1 judged by claude-sonnet-5-5; Claude via Claude Code, Grok via Cursor; read 2026-10-20 ✓ | xmodel | Opus 5.5·high |
| llm-stats.com arena: agents category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), agents category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Fable 5.1·default · Haiku 5.5·default |
| llm-stats.com arena: code category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), code category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Haiku 5.5·default · Fable 5.1·default |
| llm-stats.com arena: general category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), general category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Fable 5.1·default · Haiku 5.5·default |
| llm-stats.com arena: multimodal category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), multimodal category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 5.5·default |
| llm-stats.com arena: reasoning category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), reasoning category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Fable 5.1·default · Haiku 5.5·default |
| llm-stats.com arena: tool_calling category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), tool_calling category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 5.5·default |
| llm-stats.com arena: vision category (Bradley-Terry/TrueSkill-style, own-run) llmstats.com | llm-stats.com's own per-category arena rating (mu/sigma/CI/games_played), vision category, methodology not fully disclosed on homepage; read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Haiku 5.5·default |
| LLMConfBench arxiv-2609.20666 | arXiv 2609.20666 · conformer energy ranking, 27 molecules × 3 prompts via OpenRouter, Opus 5 and Sonnet 5 at high; cost summed from the repo's per-call logs ✓ | xmodel | Opus 5·high · Sonnet 5·high |
| LMArena Agent lmarena.ai | Agent Mode real sessions (Code/Chat/Work) · cost=median $/task (14d window), score=Net Improvement composite signal, effort read from model-variant label; live board, read 27 Sep 2026 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max · Opus 5·max · Fable 5·high · Sonnet 5·high · Haiku 5.5·high |
| LMArena Document lmarena.ai | Document arena (Overall, the page's default board: no style control) · blind pairwise votes on answers about uploaded documents, Elo rating, no cost; vote cutoff 13 Sep 2026, so no 5.5 model yet; effort read from the arena's model key (-thinking, -high, -max); live board, read 30 Sep 2026 ✓ | xmodel | Opus 5·high · Fable 5.1·max · Fable 5·default · Opus 4.7·nothink · Opus 4.7·high · Sonnet 4.6·default · Opus 4.8·high · Sonnet 5·high · Opus 4.8·nothink · Haiku 4.5·nothink |
| LMArena Search lmarena.ai | Search arena (Overall, the page's default board: no style control) · blind pairwise votes on answers with web search, Elo rating, no cost; each model runs with the arena's search tool (key suffix -search); vote cutoff 24 Aug 2026, no 5.5 model; effort not in the key (default), Fable 5 at high per its display name; live board, read 30 Sep 2026 ✓ | xmodel | Fable 5.1·high · Opus 5·high · Opus 4.7·default · Fable 5·high · Sonnet 4.6·default · Opus 5.5·medium · Opus 4.8·default · Sonnet 5·default |
| LMArena Text lmarena.ai | Text arena (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-opus-5.5-high-vertex); live board, read 30 Sep 2026 ✓ | xmodel | Opus 5.5·high · Fable 5.1·max · Opus 5·max · Opus 5·high · Fable 5·high · Opus 4.7·high · Opus 4.7·nothink · Sonnet 5.5·xhigh · Opus 4.8·high · Sonnet 4.6·default · Opus 4.8·nothink · Sonnet 5·high · Haiku 4.5·nothink |
| LMArena Vision lmarena.ai | Vision arena (Overall, the page's default board: style control) · blind pairwise votes on image-input prompts, Elo rating, no cost; vote cutoff 28 Sep 2026, no 5.5 model yet; effort read from the arena's model key (-thinking, -thinking-16k/32k, -high, -max); live board, read 30 Sep 2026 ✓ | xmodel | Fable 5·high · Opus 5.5·high · Opus 5·high · Fable 5.1·max · Opus 4.7·nothink · Opus 4.7·high · Opus 4.8·high · Sonnet 5.5·xhigh · Opus 4.8·nothink · Sonnet 4.6·default · Sonnet 5·high |
| LMArena WebDev lmarena.ai | Code arena, WebDev board (Overall) · blind pairwise votes, Elo rating, no cost; effort read from the arena's model key (e.g. claude-sonnet-5-5-high); live board, read 30 Sep 2026 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·xhigh · Fable 5.1·max · Sonnet 5.5·high · Opus 5·max · Opus 5·high · Fable 5·high · Haiku 5.5·high · Opus 4.7·high · Opus 4.7·nothink · Opus 4.8·high · Sonnet 5·high · Opus 4.8·nothink · Sonnet 4.6·default · Haiku 4.5·nothink |
| LMArena WebDev — HTML sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-html, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·high · Fable 5.1·max · Opus 5·max · Fable 5·high · Opus 5·high · Haiku 5.5·high · Opus 4.7·nothink · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high · Sonnet 4.6·default · Opus 4.8·nothink · Haiku 4.5·nothink |
| LMArena WebDev — React sub-category lmarena.ai | lmarena-ai/leaderboard-dataset, configs/webdev, category=webdev-react, latest split, publish 2026-10-02; read 2026-10-05 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·xhigh · Fable 5.1·max · Sonnet 5.5·high · Opus 5·max · Opus 5·high · Fable 5·high · Haiku 5.5·high · Opus 4.7·high · Opus 4.8·high · Opus 4.7·nothink · Sonnet 5·high · Opus 4.8·nothink · Sonnet 4.6·default · Haiku 4.5·nothink |
| LMBuild: buildable/functional 3D structures, mean of four level scores arxiv-2610.04292 | 200-object core set, Baseline Interaction Protocol, 30 systems; Table 6 of arXiv 2610.04292; read 2026-10-15 ✓ | xmodel | Opus 5·default · Fable 5.1·default · Sonnet 5·default · Haiku 4.5·solo |
| log(N)-Questions (68 games, wins/68) logn-2609.19113 | log(N)-Questions Table 1: 68 games/model on identical document sets, N=4..1024, win = target found; cost = arm cost USD from usage normalised across providers; arXiv 2609.19113v3 (22 Sep 2026); per-N columns same runs (rule 8); read 2026-10-20 ✓ | xmodel | Opus 5·default |
| LSP vs grep (LSP arm) arxiv-2608.13568 | arXiv 2608.13568 §6.2 · 6 SWE-bench-Lite localization tasks, tokens-to-success ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default · Haiku 4.5·solo |
| lsp-t2s arxiv-2608.13568 | config ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default · Haiku 4.5·solo |
| M-GATE grammar error detection, MCC mean over 30 languages (temperature 0) mgate-2608.03803 | M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.1; 53 models / 82 configurations March-July 2026; linguist-crafted adversarial grammar items, 30 languages, temperature 0, one run; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓ | xmodel | Fable 5·default · Haiku 4.5·nothink · Haiku 4.5·solo · Opus 4.7·default · Opus 4.7·req-adaptive · Opus 4.8·default · Opus 4.8·req-adaptive · Sonnet 4.6·default · Sonnet 4.6·req-adaptive · Sonnet 4.6·req-enabled · Sonnet 5·default |
| M-GATE live board 2026-10-20: grammar MCC overall (mean over 30 languages, temperature-zero results) mgate-live-rws-20261020 | M-GATE (RWS TrainAI) live leaderboard m-gate.ai / rws.com/.../m-gate/leaderboard read 2026-10-20 via r.jina.ai text view (direct fetch is bot-challenged); 90 models, grammar MCC Overall = average across 30 languages, default view (variant rows 'Show variants' toggle not retrievable); a newer snapshot of the board whose frozen July 2026 table is group mgate-grammar-mcc (paper arXiv 2608.03803); effort of each row not printed on the live page; read 2026-10-20 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Fable 5·default · Opus 5·default · Sonnet 5.5·default · Opus 4.8·default · Sonnet 5·default · Opus 4.7·default · Sonnet 4.6·default · Haiku 5.5·default · Haiku 4.5·solo |
| M-GATE round-trip translation, hard subset, normalised score mean over 29 targets mgate-2608.03803 | M-GATE (RWS TrainAI) July 2026 reporting snapshot, Appendix J J.2; 53 models / 82 configurations March-July 2026; 100 English sources round-tripped into 29 languages, hard subset of 50 sentences, judges GPT-5.4 Mini + Claude Haiku 4.5 + Gemini 3.1 Flash Lite at temp 0; list prices only (no run spend: no cost); arXiv 2608.03803 (4 Aug 2026); default-temperature 3-run and per-language columns are other summaries of the same task (rule 8); read 2026-10-20 ✓ | xmodel | Fable 5·default · Haiku 4.5·nothink · Haiku 4.5·solo · Opus 4.7·default · Opus 4.7·req-adaptive · Opus 4.8·default · Opus 4.8·req-adaptive · Sonnet 4.6·default · Sonnet 4.6·req-adaptive · Sonnet 4.6·req-enabled · Sonnet 5·default |
| MA-Bench mission analysis, benchmark condition, weighted score (GPT-5.6 judge), Opus 5 / Fable 5.1 / Sonnet 5 arxiv-2610.19138 | arXiv 2610.19138 Table 4; production judge GPT-5.6; 3000-word cap; 216 briefs (n=30/50/30 per model); read 2026-10-18 ✓ | xmodel | Opus 5·medium · Fable 5.1·medium · Sonnet 5·medium |
| Management Consulting Tasks (Internal) (table) openai-gpt56-launch | Management Consulting Tasks (Internal); OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| MapleTechLabs/maple: tool-calling eval (50 tasks, k=3 x 2 runs pooled), pass@1 github-maple-evals | MapleTechLabs/maple apps/ai/src/evals/BASELINE.md, section 'Tools suite: 2026-10-08, six models, two runs pooled'; 50 tasks, production chat turn, eval world; ceiling-effect suite ('saturated' per source); score = pass@1 %; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default |
| MathArena · aime 2026 matharena.ai | MathArena, aime--aime_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · apex 2025 matharena.ai | MathArena, apex--apex_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · april matharena.ai | MathArena, arxiv--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · april matharena.ai | MathArena, arxiv_false--april: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Fable 5·max · Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · arxivmath matharena.ai | MathArena, overall--arxivmath: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 5.5·high · Fable 5.1·low |
| MathArena · august matharena.ai | MathArena, arxiv--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Fable 5.1·high · Opus 5.5·high · Fable 5.1·low |
| MathArena · august matharena.ai | MathArena, arxiv_false--august: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Fable 5.1·high · Fable 5.1·low · Opus 5.5·high |
| MathArena · brokenarxiv matharena.ai | MathArena, overall--brokenarxiv: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 5.5·high · Fable 5.1·low |
| MathArena · february matharena.ai | MathArena, arxiv--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · february matharena.ai | MathArena, arxiv_false--february: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · final answer comps matharena.ai | MathArena, overall--final_answer_comps: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · hmmt feb 2026 matharena.ai | MathArena, hmmt--hmmt_feb_2026: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · january matharena.ai | MathArena, arxiv--january: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.7·xhigh |
| MathArena · june matharena.ai | MathArena, arxiv--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Opus 5.5·high · Opus 5·max · Opus 4.8·max · Fable 5.1·low |
| MathArena · june matharena.ai | MathArena, arxiv_false--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 5.5·high · Opus 5·max · Fable 5.1·low · Fable 5·max · Opus 4.8·max |
| MathArena · june matharena.ai | MathArena, arxivlean--june: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 5·max · Opus 4.8·max · Fable 5·max |
| MathArena · kangaroo 2025 1-2 matharena.ai | MathArena, kangaroo--kangaroo_2025_1-2: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max |
| MathArena · kangaroo 2025 11-12 matharena.ai | MathArena, kangaroo--kangaroo_2025_11-12: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max |
| MathArena · kangaroo 2025 3-4 matharena.ai | MathArena, kangaroo--kangaroo_2025_3-4: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max |
| MathArena · kangaroo 2025 5-6 matharena.ai | MathArena, kangaroo--kangaroo_2025_5-6: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max |
| MathArena · kangaroo 2025 7-8 matharena.ai | MathArena, kangaroo--kangaroo_2025_7-8: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max |
| MathArena · kangaroo 2025 9-10 matharena.ai | MathArena, kangaroo--kangaroo_2025_9-10: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max |
| MathArena · march matharena.ai | MathArena, arxiv--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · march matharena.ai | MathArena, arxiv_false--march: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · may matharena.ai | MathArena, arxiv--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Fable 5·max · Fable 5.1·max · Opus 5.5·high · Opus 4.8·max · Fable 5.1·low |
| MathArena · may matharena.ai | MathArena, arxiv_false--may: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 5.5·high · Fable 5.1·low · Fable 5·max · Opus 4.8·max |
| MathArena · shortlist 2025 matharena.ai | MathArena, apex--shortlist_2025: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max · Opus 4.7·xhigh |
| MathArena · visual math matharena.ai | MathArena, overall--visual_math: each model run 4 times on each problem · accuracy in % with its 95 % CI in the flags, cost = the average cost of one run on one problem (USD), mean input/output tokens; effort from the model name; ⚠️ (model released after the competition) kept as a flag; read 2026-09-30 ✓ | xmodel | Opus 4.8·max |
| MathConstraint full (329 instances), no tools: verifier acceptance % mathconstraint-2605.08498 | MathConstraint (generated verified combinatorial instances; independent academics): MathConstraint full (329 instances), no tools; OpenRouter, temperature 0, max_tokens 16,384, one run per cell, accuracy = verifier acceptance of the submitted witness (SAT polarity accuracy = same runs); cost = dollars per instance from provider-reported billing; arXiv 2605.08498 (May 2026; pricing as of 2026-04-30); read 2026-10-20 ✓ | xmodel | Opus 4.7·think-8k · Sonnet 4.6·default |
| MathConstraint full (329 instances), Python sandbox tools (<=8 rounds): verifier acceptance % mathconstraint-2605.08498 | MathConstraint (generated verified combinatorial instances; independent academics): MathConstraint full (329 instances), Python sandbox tools (<=8 rounds); OpenRouter, temperature 0, max_tokens 16,384, one run per cell, accuracy = verifier acceptance of the submitted witness (SAT polarity accuracy = same runs); cost = dollars per instance from provider-reported billing; arXiv 2605.08498 (May 2026; pricing as of 2026-04-30); read 2026-10-20 ✓ | xmodel | Opus 4.7·think-8k · Sonnet 4.6·default |
| MathConstraint-Easy (266), no tools: verifier acceptance % mathconstraint-2605.08498 | MathConstraint (generated verified combinatorial instances; independent academics): MathConstraint-Easy (266), no tools; OpenRouter, temperature 0, max_tokens 16,384, one run per cell, accuracy = verifier acceptance of the submitted witness (SAT polarity accuracy = same runs); cost = dollars per instance from provider-reported billing; arXiv 2605.08498 (May 2026; pricing as of 2026-04-30); read 2026-10-20 ✓ | xmodel | Opus 4.7·think-8k · Sonnet 4.6·default |
| MathConstraint-Easy (266), Python sandbox tools: verifier acceptance % mathconstraint-2605.08498 | MathConstraint (generated verified combinatorial instances; independent academics): MathConstraint-Easy (266), Python sandbox tools; OpenRouter, temperature 0, max_tokens 16,384, one run per cell, accuracy = verifier acceptance of the submitted witness (SAT polarity accuracy = same runs); cost = dollars per instance from provider-reported billing; arXiv 2605.08498 (May 2026; pricing as of 2026-04-30); read 2026-10-20 ✓ | xmodel | Opus 4.7·think-8k · Sonnet 4.6·default |
| MathVis: mathematical figure generation as TikZ/SVG programs, pass@1 arxiv-2610.12954 | 512 official questions, single-round code generation, 3 rollouts; Table 6 of arXiv 2610.12954; read 2026-10-15 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Sonnet 4.6·default · Haiku 4.5·solo |
| MazeBench ASCII no tools mazebench | MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, no Python tools; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Fable 5·max · Sonnet 5.5·max · Opus 5·max · Haiku 5.5·max · Opus 4.8·max · Sonnet 4.6·high · Sonnet 5·max · Haiku 4.5·solo |
| MazeBench ASCII with Python tools mazebench | MazeBench AI leaderboard (mazebench.com/leaderboard/ai?mode=ascii), ASCII observations, Python tools enabled; score = gems collected /100 (the board's default bar), 1 run per (model, harness), snapshot of updated_at 2026-09-14 / generated 2026-10-08; cost = recorded API cost per run (basis per row); read 2026-10-20 ✓ | xmodel | Opus 5.5·max · Opus 5·max · Sonnet 5.5·max · Fable 5·max · Fable 5.1·max · Haiku 5.5·max · Sonnet 5·max |
| MCP Atlas (table) openai-gpt55-launch | MCP Atlas; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| MCP-Atlas public set (GLM-5.2 card) zai-hf-glm-5.2 | MCP-Atlas 500-task public set, think mode, 10 min timeout, judge Gemini-3.0-Pro; table value 76.8 (chart shows 77.0); read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| mcptox github-mcptox | config ✓ | xmodel | Haiku 4.5·solo · Sonnet 5·high · Opus 5·default · Opus 4.8·nothink |
| MedChem (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| MediCrab Lab notes #2: 24 planted errors in one pediatric septic-shock trial medicrab | same prompt, no web search, 2-3 runs each, graded blind; score = mean errors found of 24; read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Fable 5.1·medium · Sonnet 5.5·medium · Haiku 5.5·high |
| Melvynx medium duo github-melvynx | Claude Code · 2 apps, Fable 5 vs Sonnet 5 medium, cost only ✓ | xmodel | Fable 5·medium · Sonnet 5·medium |
| Melvynx xhigh trio github-melvynx | Claude Code · 3 apps, Fable 5 vs Sonnet 5 xhigh, same batch, verified runs, cost only ✓ | xmodel | Fable 5·xhigh · Sonnet 5·xhigh |
| Melvynx/benchmarks: web-app build, Claude Code, Opus 5.5 vs Sonnet 5.5 medium, 2 tasks without cursor continuation github-melvynx | Melvynx/benchmarks runs 2026-09-28; harness cc (Claude Code); tasks timezone-checker + video-generation summed; cost = accounting.json costUsd (harness cost, list basis); no quality score in repo (harness-verified only); read 2026-10-18 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| merc-bench hard tier T13 (per-task, adopted as core at 5e09655) github-merc-bench | github.com/sirneb/merc-bench, task T13, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓ | xmodel | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 4.5·req-high · Haiku 4.5·req-low · Haiku 4.5·req-medium · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| merc-bench hard tier T14 (per-task, adopted as core at 5e09655) github-merc-bench | github.com/sirneb/merc-bench, task T14, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓ | xmodel | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 4.5·req-high · Haiku 4.5·req-low · Haiku 4.5·req-medium · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·max · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| merc-bench hard tier T15 (per-task, adopted as core at 5e09655) github-merc-bench | github.com/sirneb/merc-bench, task T15, hard tier adopted from tournament candidates since #8 (5e09655); read 2026-10-05 ✓ | xmodel | Fable 5·high · Fable 5·low · Fable 5·max · Fable 5·medium · Fable 5·xhigh · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Haiku 4.5·req-high · Haiku 4.5·req-low · Haiku 4.5·req-medium · Opus 4.8·high · Opus 4.8·low · Opus 4.8·xhigh · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Opus 5.5·high · Opus 5.5·low · Opus 5.5·max · Opus 5.5·medium · Opus 5.5·xhigh · Sonnet 5·high · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5.5·high · Sonnet 5.5·low · Sonnet 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| Mercor · AdvancedIF mercor.com | Mercor, AdvancedIF: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·high · Fable 5.1·max |
| Mercor · AIME 2025 mercor.com | Mercor, AIME 2025: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5·max · Fable 5·max |
| Mercor · BrowseComp Extended mercor.com | Mercor, BrowseComp Extended: the public benchmark extended with new expert-built Mercor tasks (100 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Sonnet 5.5·max · Fable 5.1·max · Opus 5.5·max |
| Mercor · CharXiv mercor.com | Mercor, CharXiv: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max |
| Mercor · CharXiv Extended mercor.com | Mercor, CharXiv Extended: the public benchmark extended with new expert-built Mercor tasks (150 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max |
| Mercor · CVDP mercor.com | Mercor, CVDP: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5.5·max |
| Mercor · DeepResearch Bench II mercor.com | Mercor, DeepResearch Bench II: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5·high · Fable 5.1·max · Sonnet 4.6·high · Opus 5.5·max |
| Mercor · DeepSWE v1.1 mercor.com | Mercor, DeepSWE v1.1: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Opus 5·max |
| Mercor · FrontierSWE v2 mercor.com | Mercor, FrontierSWE v2: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Fable 5.1·high · Opus 5·high |
| Mercor · GDPval mercor.com | Mercor, GDPval: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Sonnet 5.5·max |
| Mercor · GDPval Extended mercor.com | Mercor, GDPval Extended: the public benchmark extended with new expert-built Mercor tasks (132 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Sonnet 5.5·max · Haiku 5.5·max |
| Mercor · Harvey LAB mercor.com | Mercor, Harvey LAB: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5·max · Fable 5.1·max · Fable 5·max |
| Mercor · HLE mercor.com | Mercor, HLE: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·high |
| Mercor · HLE Extended mercor.com | Mercor, HLE Extended: the public benchmark extended with new expert-built Mercor tasks (447 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5·high · Fable 5·high |
| Mercor · Long Context Reasoning (AA-LCR v1.1) mercor.com | Mercor, Long Context Reasoning (AA-LCR v1.1): the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Fable 5.1·max · Fable 5·max · Opus 5.5·max |
| Mercor · Long Context Reasoning (AA-LCR) Extended mercor.com | Mercor, Long Context Reasoning (AA-LCR) Extended: the public benchmark extended with new expert-built Mercor tasks (89 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·high |
| Mercor · MedXpertQA MM mercor.com | Mercor, MedXpertQA MM: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max |
| Mercor · MedXpertQA MM Extended mercor.com | Mercor, MedXpertQA MM Extended: the public benchmark extended with new expert-built Mercor tasks (51 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·high |
| Mercor · MMLU-Pro mercor.com | Mercor, MMLU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·high · Fable 5.1·max |
| Mercor · MMMU-Pro mercor.com | Mercor, MMMU-Pro: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max |
| Mercor · MMMU-Pro Extended mercor.com | Mercor, MMMU-Pro Extended: the public benchmark extended with new expert-built Mercor tasks (440 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max |
| Mercor · ProgramBench mercor.com | Mercor, ProgramBench: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·high · Opus 5·high |
| Mercor · SciCode mercor.com | Mercor, SciCode: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max · Fable 5·max |
| Mercor · SUPERChem mercor.com | Mercor, SUPERChem: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5.5·max |
| Mercor · SWE Atlas - Codebase QnA mercor.com | Mercor, SWE Atlas - Codebase QnA: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5·max · Sonnet 5.5·max · Opus 5.5·max |
| Mercor · SWE Atlas - Codebase QnA Extended mercor.com | Mercor, SWE Atlas - Codebase QnA Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Opus 5·max · Fable 5.1·max |
| Mercor · SWE-bench Multilingual mercor.com | Mercor, SWE-bench Multilingual: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5·max · Opus 5.5·max |
| Mercor · SWE-bench Verified mercor.com | Mercor, SWE-bench Verified: the open-source benchmark independently evaluated by Mercor · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max |
| Mercor · SWE-bench Verified Extended mercor.com | Mercor, SWE-bench Verified Extended: the public benchmark extended with new expert-built Mercor tasks (50 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5·max · Fable 5·max |
| Mercor · Terminal-Bench 2.1 Extended mercor.com | Mercor, Terminal-Bench 2.1 Extended: the public benchmark extended with new expert-built Mercor tasks (99 Mercor tasks) · score in %, ± kept in the flags; the card shows its top 5 models only; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max |
| Mercor APEX-Accounting mercor.com | Mercor's own board, APEX-Accounting, harness loop_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Opus 5·max · Haiku 5.5·max · Fable 5·max · Opus 4.7·max · Sonnet 5·max |
| Mercor APEX-Agents mercor.com | Mercor's own board, APEX-Agents, harness loop_truncated_tools_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max · Fable 5.1·high · Sonnet 4.6·high · Opus 5.5·medium · Opus 5.5·max · Sonnet 5.5·medium · Sonnet 5.5·max · Haiku 5.5·max |
| Mercor APEX-SWE (inspect) mercor.com | Mercor's own board, APEX-SWE, harness inspect · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Fable 5·max · Opus 4.7·high · Opus 4.8·high · Sonnet 5·high · Opus 5·max |
| Mercor APEX-SWE (terminus-2) mercor.com | Mercor's own board, APEX-SWE, harness terminus-2 · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5·max · Opus 4.7·max · Sonnet 5·max · Opus 4.8·max · Sonnet 4.6·high · Opus 5.5·medium · Sonnet 5.5·medium · Opus 5·max |
| Mercor APEX-v1 (The AI Productivity Index) mercor.com | Mercor's own board, APEX-v1 (The AI Productivity Index), harness singleshot_agent · pass@1 in %, Mercor's ± half-width kept in the flags, mean score beside it; effort from the board's per-model field; no cost; read 2026-09-30 ✓ | xmodel | Fable 5.1·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Fable 5·max · Opus 4.7·max · Sonnet 5·max · Opus 4.8·max · Sonnet 4.6·high · Opus 5·xhigh |
| Mercor open-source · GeneBench-Pro mercor.com | Mercor open-source evaluation, effort max (board field); read 2026-10-15 ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max |
| Mercor open-source · Terminal-Bench 4.0 mercor.com | Mercor open-source evaluation, effort max (board field); read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max |
| Mideind Icelandic LLM leaderboard, Case Government (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_case_marking@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Completion (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_cloze@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Famous Phrases (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_phrases@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Flora & Fauna (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_nature_translations@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Generative Quality (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark generative_quality@v1 primary metric quality_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Icelandic GEC (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_gec@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Icelandic WikiQA (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_wikiqa@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Meanings (Language Proficiency) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_idioms_meanings@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Song Lyrics (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_song_lyrics@v1 primary metric accuracy (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| Mideind Icelandic LLM leaderboard, Textile Terms (Knowledge) mideind-icelandic-leaderboard | Mideind Icelandic LLM leaderboard (HF space mideind/icelandic-llm-leaderboard), benchmark icelandic_textiles@v1 primary metric llm_judge_score (0-1), generation max_tokens 32000, one row per model x thinking setting, no tools; generated_at 2026-10-19T12:48:53Z; output_tokens is the total over the whole suite; read 2026-10-20 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Opus 4.8·high · Sonnet 5.5·high · Opus 4.8·nothink · Opus 5·nothink · Sonnet 5·high · Sonnet 5.5·nothink · Sonnet 5·nothink · Haiku 5.5·high · Haiku 5.5·nothink · Haiku 4.5·nothink · Haiku 4.5·solo |
| MILU (S55) anthropic-syscard | Sonnet 5.5 card p140 · 4 models at max, score-only ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Sonnet 5·max |
| Mine Odyssey: Minecraft spatial navigation (success rate, 180 tasks) arxiv-2610.11328 | 180 navigation tasks (125 outdoor/55 indoor), 500 decisions or 6 h; Opus 5 at 4 efforts plus Opus 5.5 max; Table 1 of arXiv 2610.11328; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Opus 5·max · Opus 5·high · Opus 5·medium · Opus 5·low |
| MineBench voxel arena minebench | minebench.ai live API · voxel builds, 15 prompts, Elo arena read 29 Sep, measured cost for 8 of 12 Claude models ✓ | xmodel | Opus 5.5·max · Opus 5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·xhigh · Opus 4.7·max · Sonnet 4.6·max |
| minilab: self-repo code review, Sonnet 5.5 (medium/high) vs Opus 5.5 (medium) note-minilab | note.com/minilab/n/n7ccd7f6a6421; author's own notectl CLI repo, 2 questions, 1 run per setting, Claude Code; cost = /usage session total (Q1+Q2) at list price, approximate (article uses 約); effort-from-config: "Sonnet 5.5のmedium、Sonnet 5.5のhigh、Opus 5.5。Opusのeffortは確認して、mediumでした"; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·medium · Sonnet 5.5·high · Opus 5.5·medium |
| MLE-Bench Lite medal average, Claude Code harness (Frontis-MA1 paper) arxiv-2607.28568 | MLE-Bench Lite 22 competitions, medal average %, harness Claude Code; Table 1 (OpenMLE rows = mean of 3 evaluation epochs); paper 2026-07-30; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Sonnet 4.6·default |
| MM-DeepResearch-Bench mmdeepresearch-bench.github.io | MM-DeepResearch-Bench, multimodal deep-research reports with web search · overall score (%) of ten judged metrics; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Haiku 4.5·solo |
| MMLU-ProX-Lite: strict accuracy averaged over 10 languages (588 parallel questions each), lm-evaluation-harness github-mmlu-prox-lite-evals | 3ae3ae/mmlu-prox-lite-evals README 'Strict accuracy (%) by language' (en zh ja ko pt de id ru ar sw); developer-default reasoning effort; 'not a leaderboard'; mean of the 10 per-language accuracies computed by us; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Sonnet 5.5·default |
| MMPostTrainBench: mean task score of the model post-trained by the research agent (Claude Code) arxiv-2610.05398 | multimodal post-training research by agent, 8 benchmarks, 24h limit; Table 3 of arXiv 2610.05398; read 2026-10-15 ✓ | xmodel | Opus 5·default · Opus 4.8·default |
| model-routing-eval: 3 fromager ticket tasks (r01t-r03t) github-model-routing-eval | andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['r01t_fromager_distinfo_ticket', 'r02t_fromager_age_filter_ticket', 'r03t_fromager_dep_chain_ticket']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓ | xmodel | Haiku 5.5·low · Sonnet 5.5·medium · Opus 5.5·medium |
| model-routing-eval: 3 harder coding tasks (t14-t16) github-model-routing-eval | andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['t14_dep_resolver', 't15_perf_regression', 't16_json_patch']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓ | xmodel | Haiku 5.5·low · Sonnet 5.5·medium |
| model-routing-eval: 5 fromager repo tasks (r01-r05) github-model-routing-eval | andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['r01_fromager_distinfo', 'r02_fromager_age_filter', 'r03_fromager_dep_chain', 'r04_fromager_age_fallback', 'r05_fromager_version_prebuilt']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓ | xmodel | Haiku 5.5·low · Sonnet 5.5·medium |
| model-routing-eval: 8 small coding tasks (t04-t12 subset) github-model-routing-eval | andre-motta/model-routing-eval session_runs.csv; Claude Code (backend claude), hidden tests, 1 run per task; tasks ['t04_lru_cache', 't05_cli_json_flag', 't06_log_parser', 't07_mutable_default', 't08_timezone_bug', 't09_thread_safety', 't11_async_migration', 't12_needle_bug']; score = pass% over tasks; cost = mean Claude Code cost per run; read 2026-10-15 ✓ | xmodel | Haiku 4.5·req-low · Sonnet 5·medium · Haiku 5.5·low |
| Modelica Agent Workflow Benchmark (Claude Code) arxiv-2608.23653 | Pufibara/Modelica Agent Workflow Benchmark (TU Dresden, arXiv 2608.23653 v1 24 Aug 2026) Table 4: 232 tasks (132 repair, 50 generation, 50 tuning), one fresh run per task, harness Claude Code; pass count = every gate passes; score = total passed / 232; read 2026-10-20 ✓ | xmodel | Sonnet 5·medium |
| Modelica Agent Workflow Benchmark (Pufibara) arxiv-2608.23653 | Pufibara/Modelica Agent Workflow Benchmark (TU Dresden, arXiv 2608.23653 v1 24 Aug 2026) Table 4: 232 tasks (132 repair, 50 generation, 50 tuning), one fresh run per task, harness Pufibara; pass count = every gate passes; score = total passed / 232; read 2026-10-20 ✓ | xmodel | Sonnet 5·medium |
| morinokenjin 6th comparison: lesson slides (photosynthesis; template) zenn-morinokenjin | claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin 6th comparison: presentation slides (new service) zenn-morinokenjin | claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin 6th comparison: report slides (library usage) zenn-morinokenjin | claude -p headless; 10-slide index.html; blind scoring by GPT-6 Luna (3 criteria x 10; 2 swapped passes averaged); read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender animation judged by Gemini 3.8 Flash (n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender mechanical watch judged by Gemini 3.8 Flash (n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, Blender still life judged by Gemini 3.8 Flash (n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; judge gemini-3.8-flash, 2 passes with swapped order averaged, out of 30; zenn.dev/morinokenjin/articles/opus-sonnet-55-blender; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, CSV report speed-up (9 hidden tables, Claude Code, n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1, score = byte-identical output vs original on 9 hidden tables; zenn.dev/morinokenjin/articles/opus-sonnet-55-keisan; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, mental arithmetic (8 questions, Claude Code, n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1; zenn.dev/morinokenjin/articles/opus-sonnet-55-keisan; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| morinokenjin: Opus 5.5 vs Sonnet 5.5, program-solved math (3 questions, Claude Code, n=1) zenn-morinokenjin | Claude Code -p, claude-opus-5-5 vs claude-sonnet-5-5, both effort medium (Claude Code standard), n=1, Python+numpy used; zenn.dev/morinokenjin/articles/opus-sonnet-55-keisan; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| moriryota62 Terraform coding on Bedrock Converse (3 tasks x 3 rounds) qiita-moriryota62 | Bedrock Converse API prompt only (no MCP/tools); mechanical score /100 (fmt validate tflint checkov coverage conventions); 3 rounds; cost = tokens x Bedrock US-Geo price; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Sonnet 5.5·default · Haiku 5.5·default |
| Morphology (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| MRS-to-text (Hajdik split) arxiv-2609.30071 | 3-exemplar prompt · Sonnet4.5/Opus5 default, BLEU, score-only ✓ | xmodel | Opus 5·default |
| mythos-5-1-travas: find 5 planted bugs in one repo (3 runs per model), claude -p effort high github-mythos-travas | ulissesflores/mythos-5-1-travas results.json (10 Oct 2026); Claude Max subscription; tools Read,Grep,Glob; score = mean planted bugs found (of 5) over 3 runs; cost = mean list-price cost per review; read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Opus 5.5·high |
| nantokanaru_log 7-task total note-nantokanaru_log | note.com 29 Sep · same setup, 7 tasks × 2 runs aggregated, cost only ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| nantokanaru_log browser game note-nantokanaru_log | note.com 29 Sep · same setup, n=2, no browser-test tool ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| nantokanaru_log web-tool task note-nantokanaru_log | note.com 29 Sep · Claude Code high, n=2, fresh context each run, cost = API-price conversion ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| NatureBench Lite Surpass-SOTA %, Claude Code reference agents (Frontis-MA1 paper) arxiv-2607.28568 | NatureBench Lite 10 tasks, All S (Surpass-SOTA g>0.1) %, harness Claude Code; All M (Match-SOTA) same runs not used; paper 2026-07-30; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| navikt/copilot golden-baselines, code-review suite batch1 (2026-09-30) navikt-copilot | GitHub Copilot CLI 1.0.90-5, code-review agent, 2 fixtures (Kotlin+TSX) with 4 planted-defect assertions each, repeated 10x (opus-5.5) or 5x (sonnet-5.5); cost = Copilot usage event total_nano_aiu/1e9 (AIU, not USD), tokens from assistant_usage_events; source: src/repos/navikt__copilot/gb/*.psv; read 2026-10-01 ✓ | xmodel | Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Sonnet 5.5·medium |
| NEO-generated reasoning benchmark (13 hard prompts, Qwen judge) github-dakshjain-opus47-gpt55-dsv4pro | dakshjain-1616 Claude-Opus-4.7-vs-GPT-5.5-vs-DeepSeek-V4-Pro-Reasoning-Benchmark (GitHub, results 2026-04-27): 13 hand-curated hard prompts, judge qwen/qwen3-235b-a22b-2507, average judge score /10, models via OpenRouter; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| NeuroSploit on GOAD (author harness): chain completion arxiv-2610.04243 | Game of Active Directory full lab, 9 models inside the author's NeuroSploit v4.2.0 harness, median of 3 independent runs, temperature 0.0, 8 h timeout; cost = total LLM cost per engagement (method not stated); 6 of 9 models at 100 % chain/coverage = ceiling; Oct 2026; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Opus 4.7·default |
| NeutronGym McStasBench seen tasks (of 16), agent loop arxiv-2610.03631 | 16 seen McStasBench tasks, one episode each, temperature 0; Table 1 of arXiv 2610.03631; read 2026-10-15 ✓ | xmodel | Sonnet 5·default · Haiku 4.5·solo |
| NeutronGym McStasBench seen tasks (of 16), one-shot arxiv-2610.03631 | 16 seen McStasBench tasks, one episode each, temperature 0; Table 1 of arXiv 2610.03631; read 2026-10-15 ✓ | xmodel | Sonnet 5·default · Haiku 4.5·solo |
| Next.js Agent Evals (Vercel), base run nextjs-evals | 31 Next.js agent evals, 4 runs each (earlyExit), Claude Code via AI Gateway, no AGENTS.md; https://nextjs.org/evals, github.com/vercel/next-evals-oss@38cb8ad, agent-results.json metadata.experiments[].baseSuccessRate/avgCostUsd; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·default · Sonnet 5·default |
| Next.js Agent Evals (Vercel), with AGENTS.md nextjs-evals | Same 31 evals/4 runs/Claude Code setup as nextjsevals-base, but with an AGENTS.md/CLAUDE.md file telling the agent the Next.js version has breaking changes; score = docsSuccessRate (board); cost = mean of per-eval costUsd in agent-results.json['results'][model] (computed-by-us, since the board's avgCostUsd field is the base-run figure, not the AGENTS.md-run figure — the per-eval array's own mean, 0.2456 for Opus 5.5, differs from the base 0.234, confirming the two are different runs); read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·default · Sonnet 5·default |
| NonoBench nonobench.com | NonoBench, 30 nonogram puzzles · puzzles solved per effort level (the board's effort-ladder chart); total cost of the 30 puzzles for each model's best configuration (leaderboard rows); Sonnet 4.5 reasoning off/on as nothink/default; read 2026-09-30 ✓ | xmodel | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max |
| note AI実験室 #1 arithmetic_parser (165 hidden pytest tests) note-sane_mink364 | Claude Code CLI (subscription); hidden pytest; API-equivalent cost from tokens x list price; Haiku 5.5 low/medium vs Opus 5.5 medium; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Opus 5.5·medium · Haiku 5.5·low |
| note AI実験室 #1 duration_parse (29 hidden pytest tests) note-sane_mink364 | Claude Code CLI (subscription); hidden pytest; API-equivalent cost from tokens x list price; Haiku 5.5 low/medium vs Opus 5.5 medium; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Opus 5.5·medium · Haiku 5.5·low |
| note AI実験室 #1 inventory_allocation (36 hidden pytest tests) note-sane_mink364 | Claude Code CLI (subscription); hidden pytest; API-equivalent cost from tokens x list price; Haiku 5.5 low/medium vs Opus 5.5 medium; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Opus 5.5·medium · Haiku 5.5·low |
| note AI実験室 #2 spec contradiction detection, easy (20 req x 3 specs x 3 runs) note-sane_mink364 | Claude Code CLI; 3 specs x 3 runs = 9 per condition; exact-set pass; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Haiku 5.5·max · Opus 4.8·high |
| note AI実験室 #2-#4 spec contradiction detection, hard (28 req x 3 specs x 5 runs) note-sane_mink364 | Claude Code CLI; 3 specs x 5 runs = 15 per condition; F1 on contradiction IDs (auto-scored; code fences stripped for all); cost API-equivalent from tokens x price; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·medium · Haiku 5.5·max · Sonnet 4.6·high · Opus 4.8·high |
| note AI実験室 #5 long-document lookup, 8 auxiliary items x 3 context sizes (10K/50K/90K) note-sane_mink364 | Claude Code CLI; 1 run per cell; score = auxiliary items correct summed over 3 sizes (final answer correct in all 9 runs); read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·medium · Haiku 5.5·high |
| nurbbench-haikulow github-nurb | config ✓ | xmodel | Haiku 4.5·solo |
| Nuxt agent evals (nuxt/nuxt-evals), Claude Code via AI Gateway nuxt/nuxt-evals | agent-results.json exportedAt 2026-09-22; 31 Nuxt agent evals, runs: 4, earlyExit: true; score is avgPassRate (fraction of the 4 runs passing, averaged over 31 evals), *100 for a percentage; cost is the gateway's avg cost USD per eval run; models ran on different dates (opus-5.5 2026-09-22, the rest 2026-08-27 to 09-05); read 2026-10-01 ✓ | xmodel | Fable 5·default · Fable 5.1·default · Opus 4.8·default · Opus 5·default · Opus 5.5·default · Sonnet 5·default · Sonnet 5.5·default |
| NYT Connections, one submission per puzzle, 20 puzzles (matsonj/eval-connections) matsonj/eval-connections | results.md (data behind matsonj.github.io/eval-connections); pts out of max 100 (20 puzzles x up to 5 pts, wins/traps/groups also recorded); cost is total USD for the 20-game run; models ran on different dates (see confound); single run each; read 2026-10-01 ✓ | xmodel | Opus 5.5·req-minimal · Fable 5·req-minimal · Fable 5.1·req-minimal · Opus 5·req-minimal · Sonnet 5·req-minimal · Haiku 4.5·req-minimal · Haiku 5.5·req-minimal · Sonnet 5.5·req-minimal |
| ODBB CRC strict concordance arxiv-2608.28592 | ODBB Oncology Decision Boundary Benchmark: 2,005 decision points (1,586 NCCN structured items + 419 CRC case items); deterministic scorer; all models via OpenRouter, temperature 0, max output 8,192 tokens, identical prompt; efforts not stated; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default |
| ODBB NCCN strict concordance arxiv-2608.28592 | ODBB Oncology Decision Boundary Benchmark: 2,005 decision points (1,586 NCCN structured items + 419 CRC case items); deterministic scorer; all models via OpenRouter, temperature 0, max output 8,192 tokens, identical prompt; efforts not stated; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default |
| OfficeQA (S55) anthropic-syscard | Sonnet 5.5 card p132 · 3 models at max, score-only ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Sonnet 5·max |
| OfficeQA Pro (S55) anthropic-syscard | Sonnet 5.5 card p132 · 133 q, 3 models at max, score-only ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Sonnet 5·max |
| OfficeQA Pro (table) openai-gpt55-launch | OfficeQA Pro; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| OhMyUnicorn / Dream Loop: ThreeJS planet one-shot, Opus 5.5 vs Sonnet 5.5 ohmyunicorn | one-shot ThreeJS planet build, no harness, Claude Code, list-price cost from session logs; src/tinyworld.html; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default |
| OpenAgents briefing-model factorial (control) github-openagents | Claude Code (coder-connect agent) · frozen historical reserve issue, control arm, Opus 5.5 vs Sonnet 5.5 × medium effort, n=4 reps each; score=pass-at-first/final-attempt%, cost=cumulative list-price USD; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| OpenAgents briefing-model factorial (treatment) github-openagents | Claude Code (coder-connect agent) · frozen historical reserve issue, treatment arm, Opus 5.5 vs Sonnet 5.5 × medium effort, n=4 reps each; score=pass-at-first/final-attempt%, cost=cumulative list-price USD; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| OpenAI MRCR v2 8-needle 128K-256K (table) openai-gpt55-launch | OpenAI MRCR v2 8-needle 128K-256K; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| OpenAI MRCR v2 8-needle 512K-1M (table) openai-gpt55-launch | OpenAI MRCR v2 8-needle 512K-1M; OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| OpenClaw Arena performance Elo (UniClaw), snapshot 18 Jul 2026 uniclaw-openclaw-arena | app.uniclaw.ai/arena API leaderboard + pareto (both generatedAt 2026-07-18T06:47Z, served and saved 20 Oct 2026): OpenClaw agents run real benchmark tasks in public battles judged by a model (self-judged battles excluded), Elo-like score (1000 + 400/ln10 x latent strength, bootstrap CI); cost = average eligible cost per run (US cents, converted to USD); methodology v1.3; the 20 Oct model-stats (mean judged score 0-10) has identical battle counts and costs = same battles, other statistic, not a group; read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| OpenGCUA paper, OSWorld-v2 offline, batched GUI+CLI actions arxiv-2610.22200 | same tasks, harness = batched GUI+CLI actions (earlier agent), 4 models, selected runs; mean score 0-100 (Table 3); no cost; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| OpenGCUA paper, OSWorld-v2 offline, full GCUA configuration (code execution + decision transfer) arxiv-2610.22200 | same tasks and models, authors' OpenGCUA harness; mean partial score 0-100 (Table 2); cost per task only where the text prints it (Qwen3.8-Flash, GPT-6 Astra: approximately, computed by the authors from token usage at published prices incl. cache discounts); arXiv 2610.22200 v1 2026-10-15; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| OpenGCUA paper, OSWorld-v2 offline, GUI-only baseline with batched actions arxiv-2610.22200 | OSWorld-v2 v2.1 82-task offline subset (excludes the 26 internet tasks; text 3.3.1), harness = GUI-only baseline with batched actions, 8 models, 'selected runs', mean partial score 0-100 (Table 2); no cost; arXiv 2610.22200 v1 2026-10-15; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| OpenGCUA paper, real-device macOS, 4 tasks, OpenGCUA harness arxiv-2610.22200 | four real-device macOS tasks (hotel etc.), OpenGCUA harness, run 1 per task ('selected run'), completion = tasks judged complete of 4 (as pct), cost = est. USD/task mean over the four tasks computed by the authors from token usage at published prices; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| OpenHands Index: Frontend openhands | OpenHands Index Frontend (SWE-bench multimodal family); one run per model; harness OpenHands software-agent-sdk (version per row, v0.38 to v1.24); cost = 'Macro-average cost per instance (USD)' (method not stated on the page); rows run on different dates (Date column) and SDK versions; no effort setting on the page; board read 2026-10-19; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default |
| OpenHands Index: Greenfield openhands | OpenHands Index Greenfield (Commit0 family); one run per model; harness OpenHands software-agent-sdk (version per row, v0.38 to v1.24); cost = 'Macro-average cost per instance (USD)' (method not stated on the page); rows run on different dates (Date column) and SDK versions; no effort setting on the page; board read 2026-10-19; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default |
| OpenHands Index: Information Gathering openhands | OpenHands Index Information Gathering (GAIA family); one run per model; harness OpenHands software-agent-sdk (version per row, v0.38 to v1.24); cost = 'Macro-average cost per instance (USD)' (method not stated on the page); rows run on different dates (Date column) and SDK versions; no effort setting on the page; board read 2026-10-19; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default |
| OpenHands Index: Issue Resolution openhands | OpenHands Index Issue Resolution (SWE-bench family, macro-average); one run per model; harness OpenHands software-agent-sdk (version per row, v0.38 to v1.24); cost = 'Macro-average cost per instance (USD)' (method not stated on the page); rows run on different dates (Date column) and SDK versions; no effort setting on the page; board read 2026-10-19; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default |
| OpenHands Index: Testing openhands | OpenHands Index Testing (SWT-bench family); one run per model; harness OpenHands software-agent-sdk (version per row, v0.38 to v1.24); cost = 'Macro-average cost per instance (USD)' (method not stated on the page); rows run on different dates (Date column) and SDK versions; no effort setting on the page; board read 2026-10-19; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 4.6·default |
| OpenMC neutron-shielding single-shot script generation (v2): execution gate, L1-L5 zenodo-neutron-shielding | zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) Table §5.1 'the gate' (scripts that ran, of 15 total, L1-L5); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| OpenMC neutron-shielding single-shot script generation (v2): physics-correct, L2-L4 zenodo-neutron-shielding | zenodo.org/records/23067522 (PDF, extracted via pypdf, numbers printed in the PDF table) §5.2 (physics-correct of 9, L2-L4, the rungs with a scorable physics criterion); effort-from-config:"Anthropic models ran at effort=medium in both versions"; N=3/model/rung, author's own words: 'signals, not rates'; read 2026-10-05 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium |
| OpenRouter GPQA-Diamond rolling-mean board (snapshot 2026-10-03) openrouter | reasoning-math/hard, OR-internal rolling-mean harness, auto-routing provider aggregate; ref openrouter.ai/benchmarks/gpqa-diamond; SNAPSHOT REPLACES prior orgpqa snapshot (2026-09-27); read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Fable 5.1·default · Opus 4.8·default · Opus 4.7·default · Opus 5·default · Haiku 5.5·default · Fable 5·default · Sonnet 5·default · Haiku 4.5·solo · Sonnet 4.6·default |
| OpenRouter tau2-bench-airline board (snapshot 2026-10-03) openrouter | agentic-tool/mid, tau2-bench airline tasks, OR-internal harness, auto-routing provider aggregate; ref openrouter.ai/benchmarks/tau2-bench-airline; SNAPSHOT REPLACES prior ortau2 snapshot (2026-09-27); read 2026-10-05 ✓ | xmodel | Fable 5·default · Fable 5.1·default · Opus 5·default · Sonnet 5.5·default · Opus 4.8·default · Opus 4.7·default · Haiku 5.5·default · Sonnet 5·default · Opus 5.5·default · Sonnet 4.6·default · Haiku 4.5·solo |
| OpenRUA: Claude Code driving a robot, LIBERO-10 (LIBERO-PRO) arxiv-2610.02459 | 200 trials per configuration across both perturbation cells, Claude Code at high effort; Table 11 of arXiv 2610.02459; read 2026-10-15 ✓ | xmodel | Sonnet 5·high · Opus 5·high · Fable 5.1·high |
| OpenWrt docs4ai rubric, alpha batch (6 scenarios), authoritative Opus-4.6-Thinking pass 2026-03-28 0807am luminairprime-openwrt | OpenWrt docs4ai rubric, alpha batch (6 scenarios), authoritative Opus-4.6-Thinking pass 2026-03-28 0807am; single-turn code answers graded pass/fail per scenario by an LLM against the author's golden key; other passes of the same outputs not used (one run one group); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| OpenWrt docs4ai rubric, beta batch (6 scenarios), pass 2026-03-30 luminairprime-openwrt | OpenWrt docs4ai rubric, beta batch (6 scenarios), pass 2026-03-30; single-turn code answers graded pass/fail per scenario by an LLM against the author's golden key; other passes of the same outputs not used (one run one group); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| OS-Bench cross-platform migration, ClaudeCode, Opus 4.8 vs Sonnet 5 arxiv-2610.19368 | arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 91 Linux tasks; medium thinking effort; read 2026-10-18 ✓ | xmodel | Opus 4.8·medium · Sonnet 5·medium |
| OS-Bench cross-platform migration, OpenHands, Opus 4.8 vs Sonnet 5 arxiv-2610.19368 | arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 91 Linux tasks; medium thinking effort; read 2026-10-18 ✓ | xmodel | Opus 4.8·medium · Sonnet 5·medium |
| OS-Bench cross-platform migration, SWE-agent, Opus 4.8 vs Sonnet 5 arxiv-2610.19368 | arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 91 Linux tasks; medium thinking effort; read 2026-10-18 ✓ | xmodel | Opus 4.8·medium · Sonnet 5·medium |
| OS-Bench cross-platform migration, Win-agent, Opus 4.8 vs Sonnet 5 arxiv-2610.19368 | arXiv 2610.19368 (13 Oct 2026); Table 2 overall both-platform success; 49 Windows tasks; medium thinking effort; read 2026-10-18 ✓ | xmodel | Opus 4.8·medium · Sonnet 5·medium |
| OSWorld 2.0 (0808 offline, 82 tasks) binary %, RSIAgent paper table arxiv-2609.15364 | OSWorld 2.0 0808 offline 82 tasks binary %; partial column and opus-4.8 20.6 identical to held cells omitted; paper 2026-09-15; read 2026-10-20 ✓ | xmodel | Opus 5·default |
| OSWorld 2.0 batch arxiv-2606.29537 | arXiv 2606.29537 · 108 workflows, 500 steps, batched tool calls, max ✓ | xmodel | Opus 4.8·max · Opus 4.7·max |
| OSWorld 2.0 code_mode=false (OpenAI GPT-5.6 post chart) openai-gpt56-launch | OSWorld 2.0, code_mode=false; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max |
| OSWorld 2.0 offline (OpenAI GPT-6 Astra post chart) openai-gpt6-astra-launch | OSWorld 2.0 offline; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = not recorded; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| OSWorld 2.0 single arxiv-2606.29537 | arXiv 2606.29537 · 108 workflows, 500 steps, single tool call, max ✓ | xmodel | Opus 4.8·max · Opus 4.7·max · Sonnet 4.6·max · Sonnet 4.6·medium |
| OSWorld 2.0 single 150 steps osworld-v2-board | osworld-v2.xlang.ai official JSON · 108 tasks, 150-step budget, Opus 4.7 max, Sonnet 4.6 medium/max ✓ | xmodel | Opus 4.7·max · Sonnet 4.6·medium · Sonnet 4.6·max |
| OSWorld 2.0 single 300 steps osworld-v2-board | osworld-v2.xlang.ai official JSON · 108 tasks, 300-step budget, Opus 4.7 max, Sonnet 4.6 medium/max ✓ | xmodel | Opus 4.7·max · Sonnet 4.6·medium · Sonnet 4.6·max |
| OSWorld 2.0, offline set (OpenAI GPT-6 Sol and Luna post) openai-gpt6-sol-luna-launch | OSWorld 2.0, offline set; OpenAI GPT-6 Sol and Luna post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-22; Wayback first capture; read 2026-10-19 ✓ | xmodel | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh · Opus 5·max |
| OSWorld-Pro arxiv-2609.24890 | arXiv 2609.24890 · OSWorld harness, 250 turns, 4 Claude models all at max, tokens × OpenRouter list incl. cache ✓ | xmodel | Opus 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max |
| PA100 rubric verification, verifier AAAJ, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier AAAJ; cost per full run; read 2026-10-15 ✓ | xmodel | Opus 5·default · Sonnet 5·default · Fable 5.1·default |
| PA100 rubric verification, verifier Archipelago, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Archipelago; cost per full run; read 2026-10-15 ✓ | xmodel | Opus 5·default · Sonnet 5·default · Fable 5.1·default |
| PA100 rubric verification, verifier Autorubric, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Autorubric; cost per full run; read 2026-10-15 ✓ | xmodel | Opus 5·default · Sonnet 5·default · Fable 5.1·default |
| PA100 rubric verification, verifier Gandalf, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Gandalf; cost per full run; read 2026-10-15 ✓ | xmodel | Opus 5·default · Sonnet 5·default · Fable 5.1·default |
| PA100 rubric verification, verifier Sauron, Claude backbones arxiv-2610.17724 | all-criteria F1 vs human labels (2,586 criteria), backbone = the Claude model, verifier Sauron; cost per full run; read 2026-10-15 ✓ | xmodel | Opus 5·default · Sonnet 5·default · Fable 5.1·default |
| Para-FR flight sim github-parafr | Claude Code medium · one prompt, ≈$5 each, Sonnet 5.5 vs Opus 5.5, cost only (approx) ✓ | xmodel | Sonnet 5.5·medium · Opus 5.5·medium |
| Parallel Search Capability · with search parallel.ai | Parallel Search Capability Leaderboard, Search Intelligence Score = mean of DSQA, HLE and WISER (100 questions each) with Parallel Search; cost per task incl. estimated search/extract usage; Claude 5.x at max effort per the page notes (Haiku 5.5 effort unstated); read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Haiku 5.5·default · Opus 5·max · Opus 5.5·max · Sonnet 5·max · Sonnet 5.5·max |
| Parallel Search Capability · without search parallel.ai | Same suites without the search tool (model alone); inference cost only; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Haiku 5.5·default · Opus 5·max · Opus 5.5·max · Sonnet 5·max · Sonnet 5.5·max |
| ParseBench document-parsing leaderboard (run-llama/ParseBench) run-llama/ParseBench | leaderboard.csv at commit afb36bd8 (2026-09-29); Overall score (Tables/Charts/ContentFaithfulness/SemanticFormatting/VisualGrounding average); cost is USD-cents per page from the repo's own pricing table (not independently verified against token counts, so flagged list-price-cost); n (documents) and run count not given in the csv; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Opus 5.5·default · Opus 5.5·low · Sonnet 5.5·default · Fable 5.1·default · Fable 5·default · Sonnet 5·default · Opus 4.8·default · Haiku 4.5·nothink · Haiku 5.5·default · Haiku 4.5·solo · Opus 4.7·default |
| Patch-performance corpus: resolution rate, total (339 cases) arxiv-2610.13563 | Corpus of 339 pre-filtered SWE-bench/SWE-bench-Live cases (instances with at least one F2P test), mini-SWE-agent, step limit 250, trajectories 17-18 Aug 2026; resolution rate = share of the corpus resolved (pre-filtered subset, not full benchmarks); no effort and no cost stated; read 2026-10-19 ✓ | xmodel | Opus 5·default |
| PBT-Bench Baseline mode (open-ended prompt): bug recall pbtbench-2605.15229 | PBT-Bench Table 1 BASE mode: 100 problems over 40 Python libraries with injected bugs, agents write Hypothesis tests (F->P per bug), OpenHands SDK 1.11.5 (200 iterations), OpenRouter, provider-default temperature; metric = problem-level average recall %; Claude Code CLI row for Opus 4.8 (different harness, 78.6% PBT) not taken; cost = OpenRouter-billed API spend of the whole arm (runs per arm differ; per-arm costs given only where printed in A.15); arXiv 2605.15229 (May 2026; post-review additions); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default · Opus 4.8·default |
| PBT-Bench PBT mode (Hypothesis-scaffolded prompt): bug recall pbtbench-2605.15229 | PBT-Bench Table 1 PBT mode: 100 problems over 40 Python libraries with injected bugs, agents write Hypothesis tests (F->P per bug), OpenHands SDK 1.11.5 (200 iterations), OpenRouter, provider-default temperature; metric = problem-level average recall %; Claude Code CLI row for Opus 4.8 (different harness, 78.6% PBT) not taken; cost = OpenRouter-billed API spend of the whole arm (runs per arm differ; per-arm costs given only where printed in A.15); arXiv 2605.15229 (May 2026; post-review additions); read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default · Opus 4.8·default |
| pd-control arxiv-2608.02670 | config ✓ | xmodel | Sonnet 5·high · Opus 4.8·high · Fable 5·high · Opus 5·high |
| pd-nisthigh arxiv-2608.02670 | config ✓ | xmodel | Sonnet 5·high · Opus 4.8·high · Fable 5·high · Opus 5·high |
| pd-nonroot arxiv-2608.02670 | config ✓ | xmodel | Sonnet 5·high · Opus 4.8·high · Fable 5·high · Opus 5·high |
| PeterTh llm-eval-local: LLM autoparallelization of 220 programs (Claude Code, medium) github-peterth-llm-eval-local | PeterTh/llm-eval-local analysis/tables/4d_all_models_score_vs_cost.csv; 220 programs per model-effort profile; slugs claude- | xmodel | Haiku 5.5·medium · Sonnet 5.5·medium · Haiku 4.5·solo · Sonnet 5·medium · Opus 5.5·medium · Fable 5·medium · Opus 5·medium |
| Phi-Bench (LLM infrastructure engineering, Full score), Claude Code, Opus 5 vs Sonnet 5 arxiv-2609.10226 | arXiv 2609.10226 v2 (15 Oct 2026); Claude Code; Table 2 Full score; one H20 GPU/task; read 2026-10-18 ✓ | xmodel | Opus 5·max · Sonnet 5·max |
| phi-Bench: file-system design and implementation, average pass rate arxiv-2608.00280 | Benchmarking LLMs on File System Design and Implementation (UIUC, arXiv 2608.00280, 4 Aug 2026) Figure 4 'Average' bars: pass rate % over six task types (BU/BI/PM/DE/PO/NF), 5 trials per task averaged; DIGITISED from the figure (src/img/phi_f4_zoom.png, pixel scale from the 60%/80% gridlines, +-1.5 points; script verify/phi_bars.py); Figure 10 labels give DeepSeek-V4-Flash 46 and Gemini-3.1-Pro 64 (different weighting); read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| phyblas Tibetan-script reading web page (Vue), blind grading qiita-phyblas | claude -p headless --effort medium, 3 models concurrently, 1 run each; blind grading by Opus 5.5 at high; 100 Wiktionary test words; cost = total_cost_usd (API pricing); read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Sonnet 5.5·medium · Haiku 5.5·medium |
| PhysAI-Bench held-out decision accuracy (500 instances, model-specific frozen configuration) physai-2609.23695 | PhysAI-Bench (Ferrag et al.; physical AI / autonomous-network decisions) Table IV: held-out set of 500 decision instances, 3 prespecified runs, per-model few-shot count and temperature chosen on a 35-instance development set (config differs per row); development-set overall accuracy over all configurations is a tuning table, not taken; arXiv 2609.23695 (Sep 2026); read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo · Fable 5·default |
| PinchBench pinchbench.com | PinchBench, OpenClaw agent tasks · average score over the model's submissions (%), average cost of a run (USD), best score in the flags; no effort stated (default); read 2026-09-30 ✓ | xmodel | Opus 4.8·fast-mode · Opus 4.8·default · Opus 4.7·default · Haiku 4.5·solo · Sonnet 4.6·default · Fable 5·default |
| PM-LLM-Benchmark v3, process-mining Q&A, judged by grok-4.7 (fit-alessandro-berti/pm-llm-benchmark3) fit-alessandro-berti/pm-llm-benchmark3 | leaderboard_grok-4.7.md at commit ab1371c (2026-09-30); 1-shot, 10-category score out of 60, LLM judge (grok-4.7, not a Claude judge); single run; no cost in this board; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Fable 5.1·high · Opus 5·high · Sonnet 5.5·default · Fable 5·high · Opus 4.8·high · Sonnet 5·default · Haiku 4.5·solo · Haiku 5.5·default |
| pokebench.tv: Pokemon Red speedrun to Brock, turns/time/cost pokebench | fresh-save Pokemon Red, screenshot+RAM state per turn, budget 1000 turns, Anthropic API list-price cost; https://pokebench.tv/runs.json, src/pokebench_runs.json; read 2026-10-05 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Fable 5.1·high · Haiku 5.5·high |
| ponytail caveman arm github-ponytail | promptfoo · 5 tasks ×30, caveman skill, cost only ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·nothink · Opus 4.8·nothink |
| ponytail skill arm github-ponytail | promptfoo · 5 tasks ×30, ponytail skill, 3 Claude models ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·nothink · Opus 4.8·nothink |
| PostTrainBench (GLM-5.2 card, evaluated by PostTrainBench) zai-hf-glm-5.2 | PostTrainBench, 1M context 128K output; board evaluation by posttrainbench.com; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| PostTrainBench (GLM-5.3 card) zai-hf-glm-5.3 | PostTrainBench, Claude Code 2.1.207, temperature=1.0 top_p=1.0, 128K output, 1M context, weighted average of 3 runs; only GLM-5.3 stated as evaluated; GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| PostTrainBench v1.2 (weighted avg, 6 benchmarks x 4 base models) posttrainbench | PostTrainBench leaderboard results v1.2 (posttrainbench.com/?version=v1.2, changelog 30 Sep / 6 Oct 2026): CLI agent post-trains Qwen3-1.7B, Qwen3-4B, SmolLM3-3B, Gemma-3-4B for 10 h on one H100; score = weighted mean of AIME25, Arena-Hard writing, GPQA Main, GSM8K, HealthBench, HumanEval (BFCL dropped), final evals averaged over up to 5 seeds, contamination judges majority of 3; mean over n runs (2-3); same-benchmark-as:posttrain (paper v1 snapshot); read 2026-10-20 ✓ | xmodel | Fable 5.1·max · Opus 5.5·max · Fable 5·max · Opus 5·default · Opus 4.8·max · Opus 4.8·high · Opus 4.7·xhigh |
| Precise Debugging Benchmark, PDB-Wild (multi-line / repository-level bugs), post-release models arxiv-2604.17338 | PDB-Wild post-release models (arXiv 2604.17338 v5 appendix B.6, tolerance eps=1): edit-level precision % (recall and unit % are the same runs); no cost; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| predev-tb20 github-predev | config ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Priced Guidance: median compression cost (bits, lower is better), Essence judge, 87 test papers arxiv-2610.04976 | generator models guided by a Fable guide, 87 recent deep-learning papers; Figure 2 text of arXiv 2610.04976; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Opus 5·high |
| prinzbench (private legal-research + search benchmark by a single author), leaderboard of 18 Jul 2026 github-prinz-ai-prinzbench | prinzbench (github.com/prinz-ai/prinzbench README 'Last updated on July 18, 2026'): 33 author-written questions (25 legal research + 8 needle search) asked 3 times, pass@1, graded by the author without blinding; score = full score x/99; access through chat apps or terminals at different dates (Jan-Jul 2026); '*' = early access before public release; the author cautions that Pro models use parallel test-time compute; read 2026-10-20 ✓ | xmodel | Fable 5·max · Opus 4.8·max · Opus 4.7·app-extended-thinking · Sonnet 4.6·app-extended-thinking |
| ProbabilityBench covert: mean KL vs list price arxiv-2610.17016 | covert world, 25 labelled memos per case, 3 seeds; neg-KL = minus mean KL divergence (nats); cost = list price USD per 1,000 predictions; both digitised from the vector Fig. 1 coordinates; read 2026-10-15 ✓ | xmodel | Haiku 5.5·nothink · Sonnet 5.5·medium |
| ProbabilityBench credit: mean KL vs list price arxiv-2610.17016 | credit world, 25 labelled memos per case, 3 seeds; neg-KL = minus mean KL divergence (nats); cost = list price USD per 1,000 predictions; both digitised from the vector Fig. 1 coordinates; read 2026-10-15 ✓ | xmodel | Haiku 5.5·nothink · Sonnet 5.5·medium |
| ProbabilityBench peer: mean KL vs list price arxiv-2610.17016 | peer world, 25 labelled memos per case, 3 seeds; neg-KL = minus mean KL divergence (nats); cost = list price USD per 1,000 predictions; both digitised from the vector Fig. 1 coordinates; read 2026-10-15 ✓ | xmodel | Haiku 5.5·nothink · Sonnet 5.5·medium |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product given, one-shot baseline github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product given, Wiggum harness github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product hidden, one-shot baseline github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low |
| Professor Wiggum FlowER mechanism prediction, 10 hard cases, product hidden, Wiggum harness github-professor-wiggum | scottmreed/professor-wiggum README harness-vs-baseline table; quality_v2 score (0-1000) on 10 hard FlowER-derived cases; thinking level low; models answered through the agent bridge (cost opaque); read 2026-10-15 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·low · Haiku 5.5·low |
| ProgramBench (Almost Solved) (GLM-5.3 card) zai-hf-glm-5.3 | ProgramBench, 'Almost Solved' metric as labelled in the card, no footnote; GLM-5.3 card 2026-10-14; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| ProgramBench (GLM-5.2 card) zai-hf-glm-5.2 | ProgramBench 200 instances, Claude-Code 2.1.156, max_turns=2000, 400K context; GLM-5.2 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| ProgramBench 166 (S55) anthropic-syscard | Sonnet 5.5 card p118 · mini-swe-agent, 3 models, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Opus 5.5·max |
| ProgramBench board (Meta/Stanford/Harvard), mini-SWE-agent programbench | ProgramBench: 200 tasks, rebuild a program from a compiled binary + docs, hidden behavioural tests, scaffold mini-SWE-agent; score = JSON 'score' (page 'Score / Cost' Pareto view; the page names 'Avg. pass rate' as 3rd sort key), cost = average API cost in USD per task; page 'Updated Sep. 28, 2026'; read 2026-10-20; Resolved % and Almost % columns are other summaries of the same runs (not rows); read 2026-10-20 ✓ | xmodel | Opus 5·xhigh · Opus 4.8·xhigh · Opus 4.7·xhigh · Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| ProgramBench paper (arXiv 2605.03546), mini-SWE-agent, % almost resolved arxiv-2605.03546 | ProgramBench 200 tasks (rebuild a program from its executable), mini-SWE-agent scaffold, 9 models; the primary metric '% Resolved' is 0.0 for every model, so the only varying score column, '% Almost' (>=95% of tests pass), is used; cost = average per task as printed ('$', method not stated); step limit 1000; arXiv v1 2026-05-05; separate snapshot from the programbench.com board; read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| ProgramDistill-300 arxiv-2609.18805 | arXiv 2609.18805 · 300 web-app repair tasks, R2E-Gym harness, Opus 5 vs Sonnet 5 at high, tokens × LiteLLM list ✓ | xmodel | Opus 5·high · Sonnet 5·high |
| ProofBench v1.0-era values (vals.ai, Epoch relay) · accuracy % epoch.ai (vals.ai ProofBench v1.0-era) | ProofBench v1.0-era values relayed by Epoch AI (proofbench_external.csv, rows annotated 'v1.0-era value; model absent from the v1.1 board ... Not comparable to v1.1 rows'); vals.ai ProofBench v1.0 methodology (the held group valsproof is the v1.1 re-grade of 2026-08-14); cost per test as relayed; read 2026-10-20 ✓ | xmodel | Opus 4.8·max · Opus 4.7·max · Sonnet 4.6·max |
| Protein library (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| Protein sequence (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| Protein trouble-shoot (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| Protein understanding (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| pxpipe pixel-reading: arithmetic probe (N=100) github-pxpipe | exact-match count on rendered-text arithmetic probes read back by the model, digitised: image reflow rendering; read 2026-10-05 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Sonnet 5.5·default · Opus 4.8·default |
| pxpipe pixel-reading: dense hex probe (N=15) github-pxpipe | exact-match count on rendered dense-hex-id probes (near-floor for every Claude model); read 2026-10-05 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Sonnet 5.5·default · Opus 4.8·default |
| pxpipe pixel-reading: gist probe (N=98) github-pxpipe | exact-match count on rendered-text gist-recall probes; read 2026-10-05 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Sonnet 5.5·default · Opus 4.8·default |
| pxpipe pixel-reading: state probe (N=18) github-pxpipe | exact-match count on rendered-text running-state probes; read 2026-10-05 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Sonnet 5.5·default · Opus 4.8·default |
| Qiita suwa_nobu cache-read qiita-suwanobu | Opus 5 vs Opus 5.5 · same task, warm-cache rounds 2-3, CLI-metered cost ✓ | xmodel | Opus 5·default · Opus 5.5·default |
| Qiita suwa_nobu cache-write qiita-suwanobu | Opus 5 vs Opus 5.5 · same task, cold-cache round, CLI-metered cost ✓ | xmodel | Opus 5·default · Opus 5.5·default |
| Qiita Takuya coding set (5 tasks ×3) qiita-takuya | Opus 5.5 high/medium, Opus 5, Fable 5.1; hidden tests 100 % (saturated) ✓ | xmodel | Opus 5.5·high · Opus 5.5·medium · Opus 5·high · Fable 5.1·high |
| Qiskit v0.43 version fidelity, Pass@1 quantum-api-drift-2607.04072 | quantum-api-drift Table 2 diagonal: prompted for and executed on Qiskit v0.43; 50 tasks x 3 samples, temp 0.8, 1024-token output cap, REST API, no tools; arXiv 2607.04072 (5 Jul 2026); Pass@3 same samples (rule 8); Codex CLI row excluded (different interface, no cap); read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default |
| Qiskit v1.3 version fidelity, Pass@1 quantum-api-drift-2607.04072 | quantum-api-drift Table 2 diagonal: prompted for and executed on Qiskit v1.3; 50 tasks x 3 samples, temp 0.8, 1024-token output cap, REST API, no tools; arXiv 2607.04072 (5 Jul 2026); Pass@3 same samples (rule 8); Codex CLI row excluded (different interface, no cap); read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default |
| Qiskit v2.0 version fidelity, Pass@1 quantum-api-drift-2607.04072 | quantum-api-drift Table 2 diagonal: prompted for and executed on Qiskit v2.0; 50 tasks x 3 samples, temp 0.8, 1024-token output cap, REST API, no tools; arXiv 2607.04072 (5 Jul 2026); Pass@3 same samples (rule 8); Codex CLI row excluded (different interface, no cap); read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default |
| QoderCLI benchmark, Deep Research (OpenAlex AI landscape), trust-insight index github-qoder-bench | QoderCLI 1.1.12, same prompt bytes and gates for four models, deep-research task on the OpenAlex AI landscape dataset, authors' 'trust-insight index' /100 (rubric weights chosen by the authors), one run per model; no USD cost (tokens + Qoder credit multiplier given); read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| QoderCLI benchmark, F2E front-end visual tasks (same 6 cases), median %-of-max github-qoder-bench | QoderCLI, 18 front-end visual tasks C01-C18; metric = median %-of-max on the SAME 6 cases (authors' 8-dimension rubric, Qwen as anchor); Fable 5 produced 15/18 (upstream error 113 on 3 cases); one run per case; no cost; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| QoderCLI benchmark, Lord of the Rings 92-second Three.js animation github-qoder-bench | QoderCLI, single-file Three.js narrative animation (92 s), score /100 = authors' relative human rubric among the four artifacts (frames at 4/20/35/50/75/90 s); one case, one run; no cost; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| quesma-baba quesma | config ✓ | xmodel | Opus 5·default · Fable 5·default · Sonnet 5·default · Opus 4.8·default |
| Qwen-AgentWorld card: AgentWorldBench Android qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column Android: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| Qwen-AgentWorld card: AgentWorldBench MCP qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column MCP: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| Qwen-AgentWorld card: AgentWorldBench OS qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column OS: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| Qwen-AgentWorld card: AgentWorldBench Overall qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column Overall: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| Qwen-AgentWorld card: AgentWorldBench Search qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column Search: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| Qwen-AgentWorld card: AgentWorldBench SWE qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column SWE: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| Qwen-AgentWorld card: AgentWorldBench Term. qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column Term.: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| Qwen-AgentWorld card: AgentWorldBench Web qwen-hf-qwen-agentworld | Qwen-AgentWorld-35B-A3B model card (README as fetched 2026-10-19; card created 2026-06-22), table 'AgentWorldBench (Open-Ended Evaluation)', column Web: five-dimensional rubric mean per domain normalised to 0-100 (Format; Factuality; Consistency; Realism; Quality, scored by an LLM judge whose identity the card does not state); task = predicting the next environment observation (language world model); no cost or tokens; MiniMax-M2.7 out of the eight providers; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| R2R-CE (mini-swe-agent) arxiv-2607.26148 | arXiv 2607.26148 Fig. 3 · minimal interface, vendor-default effort, score only ✓ | xmodel | Fable 5·default · Opus 5·default · Opus 4.8·default · Sonnet 5·default |
| rampnet github-rampnet | config ✓ | xmodel | Sonnet 5·low · Sonnet 5·high · Opus 5·low · Opus 5·high |
| RampNet Fable github-rampnet | same rig/prompt, effort=low, Anthropic 1P path · Fable 5 vs 5.1 ✓ | xmodel | Fable 5·low · Fable 5.1·low |
| Reader-highlight sentence ranking (215 dev pages), zero-shot arxiv-2610.17768 | average-precision difference vs the cached gemini-3.1-pro-preview reference, first sample, one scoring run; low reasoning effort; read 2026-10-15 ✓ | xmodel | Opus 5.5·low · Fable 5.1·low · Haiku 5.5·low |
| RealityBench acceptability (1,378 frames) arxiv-2610.14397 | zero-shot VLM acceptability F1 against the rule scorer; cost = whole set at developer list price, thinking at output rate; read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Opus 5.5·medium · Opus 5·high · Sonnet 5·high |
| Reddit 1x0poki (smith2008): Three Little Pigs SVG film, blind cross-judged reddit-smith2008 | 20-s film of 600 SVG frames, each model in Claude Code, one run; each model scored all three films blind (frame sheets); score /10; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Haiku 5.5·default |
| Reddit 1x132yy (Escobar747): cross-language agentic task in a real Rust repo reddit-Escobar747 | one agentic task (move logic across a language boundary following repo conventions); cost per task as self-reported (Sonnet: at list prices); qualitative score only; read 2026-10-15 ✓ | xmodel | Haiku 5.5·medium · Haiku 5.5·high · Sonnet 5.5·medium |
| Reddit 1x1jbry (AlexXx52): one read-only Unity investigation, subscription allowance reddit-AlexXx52 | same read-only investigation of a Unity 6 project on each consumer subscription (Claude Pro for Claude); 'cost' = % of the subscription's own usage allowance (NOT USD, not comparable across providers); score = author's subjective quality /10; 1 run; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·medium · Opus 5.5·medium · Opus 4.8·medium · Sonnet 4.6·medium · Sonnet 5.5·high · Opus 5.5·high · Opus 4.8·high · Sonnet 4.6·high |
| Reddit 1x1nsaz (emarkosov): 5 web tasks (3 research + 2 fact-check) cost reddit-emarkosov | 5 web tasks as subagents, high effort; cost for all 5 tasks as self-reported; read 2026-10-15 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high · Haiku 5.5·xhigh |
| Reddit 1x1nsaz (emarkosov): code search task, Haiku/Sonnet/Opus 5.5 as subagents reddit-emarkosov | re-run of a real task (find every file for a feature; 48 true files) on each model as subagent, high effort; score = files found of 48; cost as self-reported; read 2026-10-15 ✓ | xmodel | Opus 5.5·high · Sonnet 5.5·high · Haiku 5.5·high · Haiku 5.5·xhigh |
| Reddit: mine-cart ride in Godot (Opus 5.5 vs Sonnet 5.5) reddit-1wx2inx | one-line prompt, two fresh Godot projects, both in Claude Code over MCP (Flockbay harness); cost = author's own figure 'at API rates', n=1 each, qualitative pass/fail only (no numeric score); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default |
| RedlineBench legal-redlining benchmark (HF Datasets Leaderboard) hf-redlinebench | legal-contract redlining, agent=claude-code, 3-LLM judge panel majority vote, turn-weighted weighted pass rate 0-100; read 2026-10-05 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Fable 5·default · Opus 4.8·default |
| ReguSim: financial manipulation detection, overall F1 arxiv-2608.19974 | ReguSim (HKUST, arXiv 2608.19974 v2 12 Oct 2026) Table 5: 900 targets, LLMs receive target-marked windows with neutral account identifiers; overall F1 (%) over 45 cells; supplemental campaign of 29-30 Sep 2026 through one hosted gateway (zctotoken.com); Flash 99.3% valid outputs; read 2026-10-20 ✓ | xmodel | Fable 5.1·high |
| ReLE 专业能力 (professional ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/专业能力.md (专业能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Opus 4.8·default · Opus 4.8·nothink · Sonnet 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 医疗与心理健康 (medical/mental health) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/医疗与心理健康.md (医疗与心理健康); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Opus 4.8·default · Opus 4.8·nothink · Sonnet 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 总分 (total) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/total.md (alldata); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Opus 5·default · Opus 4.8·default · Opus 4.8·nothink · Sonnet 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 推理与数学计算 (reasoning/math) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/推理与数学计算.md (推理与数学计算); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Opus 4.8·default · Sonnet 5·default · Opus 4.8·nothink · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 教育 (education) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/教育.md (教育); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Opus 5.5·default · Opus 4.8·nothink · Sonnet 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 法律与行政公务 (law/admin) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/法律与行政公务.md (法律与行政公务); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5·default · Opus 5.5·default · Opus 4.8·default · Sonnet 5·default · Opus 4.8·nothink · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 语言与指令遵从 (language/IF) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/语言与指令遵从.md (语言与指令遵从); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Opus 5·default · Opus 4.8·default · Opus 4.8·nothink · Sonnet 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 通用能力 (general ability) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/通用能力.md (通用能力); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Opus 5·default · Opus 4.8·default · Opus 4.8·nothink · Sonnet 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| ReLE 金融 (finance) chinese-llm-benchmark | jeinlee1991/chinese-llm-benchmark leaderboard/金融.md (金融); 171-leaderboard Chinese-language board; score=accuracy%, cost=CNY per 1,000 calls (花费/千次(元)); no effort level stated by the source (plain '-thinking' suffix vs plain model name is the only variant given); read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Opus 5·default · Opus 4.8·default · Sonnet 5·default · Opus 4.8·nothink · Sonnet 5.5·default · Haiku 5.5·default |
| renchris review s55 + o55 rejudge github-renchris-infra | claude -p no-tools, CC 2.1.284 · same 9-brief corpus as renchris-o55 (7 scored / 36 anchored defects), Sonnet 5.5 low→max (new) judged in the same pass as a re-judging of the held Opus 5.5 medium/high/xhigh outputs; 3 blind judges, strict majority; CLI list cost for the Sonnet 5.5 cells only ✓; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5·max · Fable 5.1·high |
| renchris synthesis-worker probe (Sonnet 5.5 vs Opus 5.5 xhigh) github-renchris-infra | 6 paired synthesis briefs, 130 key items, 1 cell per arm/brief; judges 2x Opus 5.5 + 1x Opus 5 (claude-judge); mean score /10 over all 18 judge rows; cost/tokens summed over the 6 cells; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Opus 5.5·xhigh |
| renkon40 · 50-mail triage note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, 50-mail triage (49 scored); Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| renkon40 · sycophancy test, 20 questions note-renkon40 | note.com 30 Sep · Claude Code 2.1.284, sycophancy test, 20 questions; Sonnet 5.5 vs Opus 5.5, n=1 per setting, blind AI judges (own model excluded), cost = API list-price conversion ✓ | xmodel | Sonnet 5.5·high · Opus 5.5·high |
| ResearchClawBench, ResearchHarness arms (matched tasks) researchclawbench | ResearchClawBench leaderboard.json (InternScience/ResearchClawBench-Home commit e475e00, 13 Oct 2026): 40 auto-research tasks, standalone LLM in the ResearchHarness baseline agent, multimodal LLM judge on a rubric, one run per (arm, task); score = mean 0-100 over the 15 tasks completed by all 21 ResearchHarness arms, cost = sum of cost_usd over the same tasks (per suite); read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Opus 4.8·default |
| Retort brazil-soccer-mcp (c): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (clojure): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (cpp): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (csharp): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (elixir): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (erlang): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (go): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (java): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (objc): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (python): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (rust): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (swift): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort brazil-soccer-mcp (typescript): Opus 5.5 low (exp-79, n=1) vs Haiku 5.5 low (exp-86, n=3) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); cost per run = CLI total (cost-only); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| retort exp-15 brazil BDD github-retort | Claude Code · brazil-soccer-mcp BDD, Sonnet 5 vs Opus 4.8, 2 languages ✓ | xmodel | Opus 4.8·default · Sonnet 5·default |
| retort exp-15 brazil TDD github-retort | Claude Code · brazil-soccer-mcp TDD, Sonnet 5 vs Sonnet 4.6, 3 languages ✓ | xmodel | Sonnet 4.6·default · Sonnet 5·default |
| retort exp-15 rest BDD github-retort | Claude Code · rest-api-crud BDD prompt, Sonnet 5 vs Opus 4.8, 4 languages n=1 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default |
| retort exp-15 rest TDD github-retort | Claude Code · rest-api-crud TDD prompt, Sonnet 5 vs Sonnet 4.6, 5 languages n=1 ✓ | xmodel | Sonnet 4.6·default · Sonnet 5·default |
| retort exp-46/48 bookshop github-retort | Claude Code · bookshop task, Opus 5 vs Fable 5 default, 8 matched languages n=1 ✓ | xmodel | Opus 5·default · Fable 5·default · Opus 5.5·low |
| retort exp-5 brazil github-retort | Claude Code · brazil-soccer-mcp, Opus 4.7 vs 4.8, 6 languages ✓ | xmodel | Opus 4.7·default · Opus 4.8·default |
| retort exp-6 github-retort | Claude Code · rest-api-crud, Opus 4.7 vs 4.8 default, 6 languages ×3 ✓ | xmodel | Opus 4.7·default · Opus 4.8·default |
| retort exp-6 beads github-retort | Claude Code + beads · rest-api-crud, Opus 4.7 vs 4.8, 6 languages ×3 ✓ | xmodel | Opus 4.7·default · Opus 4.8·default |
| retort exp-8 github-retort | Claude Code · rest-api-crud Elixir/Erlang, Opus 4.7 vs 4.8 ×3 ✓ | xmodel | Opus 4.7·default · Opus 4.8·default |
| Retort rest-api-crud (c): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (clojure): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (cpp): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (csharp): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (elixir): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (erlang): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (java): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (objc): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (rust): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (swift): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| Retort rest-api-crud (typescript): Opus 5.5 low (exp-79) vs Haiku 5.5 low (exp-86) github-retort | retort playpen, Claude Code, isolated (--strict-mcp-config); n=3 runs per cell; cost per run = CLI total (cost-only: requirement_coverage saturated); CLI 2.1.284 vs 2.1.302; read 2026-10-18 ✓ | xmodel | Opus 5.5·low · Haiku 5.5·low |
| retort49-fast github-retort | config ✓ | xmodel | Opus 4.8·default |
| retroboard arxiv-2607.02436 | config ✓ | xmodel | Opus 4.7·high · Opus 4.7·xhigh · Sonnet 4.6·high |
| Roblox LLM Leaderboard (TorpedoSoftware) composite hf-roblox-llm-leaderboard | RobloxQA v2.0 (3000 freeform LLM-judged + 3000 multiple-choice) + Luau Leetcode (208 single-shot problems); composite = 0.5 knowledge(0.7 ff+0.3 mc)+0.5 coding(difficulty-weighted, scores.js); board's own harness, one run per model; cost of whole suite; snapshot 2026-09-30; read 2026-10-20 ✓ | xmodel | Fable 5·medium · Fable 5.1·medium · Haiku 4.5·solo · Opus 4.8·medium · Opus 5·medium · Opus 5.5·medium · Sonnet 5·medium · Sonnet 5.5·medium |
| roboco-io EXP-001 RealWorld backend in cpp: Sonnet 5.5 vs Opus 5.5 github-roboco-language-benchmark | Claude Code 2.1.302 Ralph loop, 3 runs per cell, cost = API-price equivalent per run (pass rate saturated at 100%); read 2026-10-18 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| roboco-io EXP-001 RealWorld backend in go: Sonnet 5.5 vs Opus 5.5 github-roboco-language-benchmark | Claude Code 2.1.302 Ralph loop, 3 runs per cell, cost = API-price equivalent per run (pass rate saturated at 100%); read 2026-10-18 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| roboco-io EXP-001 RealWorld backend in python: Sonnet 5.5 vs Opus 5.5 github-roboco-language-benchmark | Claude Code 2.1.302 Ralph loop, 3 runs per cell, cost = API-price equivalent per run (pass rate saturated at 100%); read 2026-10-18 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| roboco-io EXP-001 RealWorld backend in rust: Sonnet 5.5 vs Opus 5.5 github-roboco-language-benchmark | Claude Code 2.1.302 Ralph loop, 3 runs per cell, cost = API-price equivalent per run (pass rate saturated at 100%); read 2026-10-18 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| roboco-io EXP-001 RealWorld backend in typescript: Sonnet 5.5 vs Opus 5.5 github-roboco-language-benchmark | Claude Code 2.1.302 Ralph loop, 3 runs per cell, cost = API-price equivalent per run (pass rate saturated at 100%); read 2026-10-18 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| Roboflow Playground Vision Evals, OCR (low tier) roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; task column: OCR; only cells that differ from the held vlm-exam cells (github-vlmexam, group vlmexam-ocr) or models the held board never ran are kept; no cost per task; read 2026-10-20 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·low · Fable 5.1·low · Fable 5·low · Opus 5·low · Haiku 5.5·low · Opus 4.8·low · Sonnet 5·low |
| Roboflow Playground Vision Evals, overall (mean of tasks) with cost per sample roboflow-playground-evals | Roboflow Playground Vision Evals (playground.roboflow.com/evals), 62-63 VLMs, 5 real-world tasks scored against ground truth by Roboflow, scores pooled at the low effort tier (page: 'scores and efficiency pooled across all five tasks at low effort'); page updated 2026-10-13, pricing 2026-10-19; read 2026-10-20 via r.jina.ai; Overall = page's mean over the tasks scored (4/5 where OCR is missing); cost = 'Avg cost / sample' in USD from price x tokens at the pricing date 2026-10-19 ('~' = estimated from self-host/OpenRouter pricing); read 2026-10-20 ✓ | xmodel | Opus 5.5·low · Sonnet 5.5·low · Fable 5.1·low · Fable 5·low · Opus 5·low · Haiku 5.5·low · Opus 4.8·low · Sonnet 5·low |
| RoboQuest (10 tasks, 50 episodes per cell), Inspect Robots harness arxiv-2610.10388 | 200-decision budget per episode, success rate over 50 episodes x 10 tasks; cost per episode at list prices; read 2026-10-15 ✓ | xmodel | Opus 5.5·medium · Fable 5.1·medium |
| RoTA-Bench (interactive 3D projection alignment, 100 tasks) arxiv-2610.17665 | RoTA-Bench (arXiv 2610.17665, 13 Oct 2026): 100 tasks, interactive 3D projection alignment, 48 decisions max, same prompt/harness for every model; score = success % (IoU >= 0.9; Final IoU and steps are other columns); cost = Table 20 total USD over the 100 tasks estimated by the authors from recorded tokens and published list prices; Temperature 0, reasoning effort medium where supported; read 2026-10-20 ✓ | xmodel | Sonnet 5·medium · Opus 5·medium |
| rtk-base arxiv-2607.12161 | config ✓ | xmodel | Sonnet 5·low · Sonnet 5·high · Opus 4.8·low · Opus 4.8·high · Haiku 4.5·req-low · Haiku 4.5·req-medium · Haiku 4.5·req-high · Haiku 4.5·req-xhigh · Haiku 4.5·req-max |
| rtx4090-local-vs-cloud: 8 manufacturing-domain tasks, programmatic grading, cloud Claude rows via Claude Code github-rtx4090-local-vs-cloud | Howard159357/rtx4090-local-vs-cloud-coding-benchmark README (15 Oct 2026); 3 runs per setup, Claude Code CLI with built-in shell/web/plugins disabled; subscription (no API cost); score = mean of 3 runs as printed; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·medium · Haiku 5.5·medium · Haiku 5.5·xhigh |
| RuBench 1.0 arxiv-2607.06411 | arXiv 2607.06411 · 25 Russian repo tasks × 3 runs, Claude Code 2.1.201, all at xhigh, CLI-reported cost ✓ | xmodel | Opus 4.8·xhigh · Sonnet 5·xhigh · Haiku 4.5·req-xhigh |
| Rubric transfer: expert score, ordinary rubric arxiv-2610.10809 | Rubric-transfer study on HealthBench-style typical split (n=105, 15 per theme over 7 themes): responses from a panel of response models are scored by judge Gemini 3.8 Flash (reasoning effort medium, majority of 3) against the expert rubric ('ordinary') or a foreign rubric; expert score 0..1 with 95% paired bootstrap CI; sampling temperature = each provider's default; ordinary expert rubric; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·default |
| RuneBench github-runebench | Harbor+Modal · 16 skills, 30-min budget, XP metric; mispriced Opus 5 cells omitted ✓ | xmodel | Opus 5·low · Opus 5·xhigh · Fable 5·xhigh · Fable 5.1·xhigh · Opus 5.5·medium · Opus 5.5·xhigh · Fable 5·default · Fable 5.1·default · Opus 4.7·default · Opus 4.7·xhigh · Opus 4.8·default · Opus 4.8·max · Sonnet 4.6·default · Sonnet 5·default · Sonnet 5·xhigh · Haiku 4.5·solo · Opus 5·default · Opus 5·medium · Sonnet 5.5·default · Sonnet 5.5·xhigh · Haiku 5.5·default · Haiku 5.5·high |
| RuntimeWire Editorial Craft (12 writing tasks, judge gpt-5.4) runtimewire.com | RuntimeWire own 12-task Editorial Craft battery, one task at a time, no tools, no retries on content; 2 deterministic tasks + 10 open-ended graded 0-10 blind by gpt-5.4, normalised 0-1 (x100); cost = publisher's estimate from published per-token pricing; article/board 2026-08-20; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Fable 5·default |
| sakana-rhi47 arxiv-2607.15524 | config ✓ | xmodel | Opus 4.7·high · Opus 4.7·max · Opus 4.7·xhigh |
| sakana-rhi48 arxiv-2607.15524 | config ✓ | xmodel | Opus 4.8·high · Opus 4.8·max · Opus 4.8·xhigh |
| Scale EnigmaEval (Wayback 2026-10-16, current 5-model board) scale-seal | https://labs.scale.com/leaderboard/enigma_eval; Wayback capture 20261016182047 (saved gpt-5.1-chat-latest/src/scale/dec_enigma_eval.html); score = board score % with 95% CI; puzzle hunts; current board (models listed with a -high / -xhigh suffix); read 2026-10-20 ✓ | xmodel | Fable 5·high · Opus 4.8·xhigh |
| Scale Humanity's Last Exam text-only (Wayback 2026-10-07) scale-seal | https://labs.scale.com/leaderboard/humanitys_last_exam_text_only; Wayback capture 20261007135520 (saved gpt-5.1-chat-latest/src/scale/dec_humanitys_last_exam_text_only.html); score = board score % with 95% CI; HLE text-only, board page of 2026-10-07 (HLE-Rolling note of 17 Sep 2026); read 2026-10-20 ✓ | xmodel | Fable 5.1·xhigh |
| SCBench scbench.ai | SCBench, agentic coding through successive checkpoints in Claude Code · strict solve rate in %, cost per checkpoint (USD); effort from the model label (High); the Claude Code version differs per row; read 2026-09-30 ✓ | xmodel | Opus 4.7·high · Sonnet 4.6·high |
| SCDBench smart-contract decompilation, with one zero-shot compilation-repair pass (dagger rows) arxiv-2605.29059 | SCDBench (arXiv 2605.29059, 27 May 2026): 600 real Solidity contracts (200 easy/medium/hard), with one zero-shot compilation-repair pass (dagger rows); score = semantic consistency (% of 14,553 public functions passing replayable semantic checkpoints; Table 4; the earlier stages format/compile/ABI are other cumulative columns); cost = Table 9 average API cost per contract at OpenRouter prices at the time; read 2026-10-20 ✓ | xmodel | Opus 4.7·high |
| SCDBench smart-contract decompilation, zero-shot decompilation arxiv-2605.29059 | SCDBench (arXiv 2605.29059, 27 May 2026): 600 real Solidity contracts (200 easy/medium/hard), zero-shot decompilation; score = semantic consistency (% of 14,553 public functions passing replayable semantic checkpoints; Table 4; the earlier stages format/compile/ABI are other cumulative columns); cost = Table 9 average API cost per contract at OpenRouter prices at the time; read 2026-10-20 ✓ | xmodel | Opus 4.7·high |
| scgdpval anthropic-syscard | config ✓ | xmodel | Sonnet 5·max · Sonnet 4.6·max · Opus 4.8·max |
| schealthbench anthropic-syscard | config ✓ | xmodel | Sonnet 5·max · Sonnet 4.6·max |
| schle-nt anthropic-syscard | config ✓ | xmodel | Sonnet 5·max · Sonnet 4.6·max · Opus 4.8·max |
| schletools anthropic-syscard | config ✓ | xmodel | Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max |
| scorch-bench: artillery duel game vs 7 bots, Claude Code subagents (reasoning track), seed 17 github-scorch-bench | smerwin/scorch-bench bench/RESULTS-claude.md (2026-10-09); 8 matches per model on one seed; one fresh Claude Code subagent per round via relay provider; score = win rate %; read 2026-10-15 ✓ | xmodel | Haiku 5.5·default · Fable 5.1·default · Sonnet 5.5·default · Opus 5.5·default |
| Search agent with gold passages kept, answer EM on HOTPOTQA (vanilla retrieval tool) arxiv-2610.05348 | 300 HotpotQA questions, BM25 over wiki-18, vanilla tool; Table 1/4 of arXiv 2610.05348; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Sonnet 5.5·default · Opus 5.5·default · Sonnet 4.6·default |
| Search agent with gold passages kept, answer EM on NQ (vanilla retrieval tool) arxiv-2610.05348 | 257 NQ questions, BM25 over wiki-18, vanilla tool; Table 1/4 of arXiv 2610.05348; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Sonnet 5.5·default · Opus 5.5·default · Sonnet 4.6·default |
| SecLens mattersec-labs.github.io | SecLens, security vulnerability detection graded for five stakeholder roles · leaderboard score, cost per task (USD); no effort stated (default); read 2026-09-30 ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| Semantic Layers for Data Analytics (+semantic doc) arxiv-2604.25149 | paired-protocol ✓ | xmodel | Opus 4.7·medium · Sonnet 4.6·medium |
| Semantic Layers for Data Analytics (schema-only) arxiv-2604.25149 | paired-protocol ✓ | xmodel | Opus 4.7·medium · Sonnet 4.6·medium |
| Senko Flysim senko | Senko Rašić · one-shot flight-sim game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓ | xmodel | Opus 5.5·xhigh · Opus 5·xhigh · Sonnet 5.5·xhigh |
| Senko RTS senko | Senko Rašić · one-shot RTS game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓ | xmodel | Opus 5.5·xhigh · Opus 5·xhigh · Sonnet 5.5·xhigh |
| Senko Voxel senko | Senko Rašić · one-shot Minecraft-like game in Claude Code, xhigh, CLI-reported cost (rounded $, quality not scored) ✓ | xmodel | Opus 5.5·xhigh · Opus 5·xhigh · Sonnet 5.5·xhigh |
| SenseBench (WSD) · prompt p001 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p001 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Fable 5·xhigh · Opus 5·max · Opus 4.8·xhigh · Opus 4.8·high · Opus 4.8·default · Opus 4.8·low · Opus 4.8·medium · Opus 4.7·medium · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·xhigh · Sonnet 5·high · Sonnet 4.6·low · Haiku 4.5·req-low · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh · Opus 5.5·low · Opus 5.5·xhigh · Sonnet 5.5·low · Sonnet 5.5·xhigh |
| SenseBench (WSD) · prompt p002 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p002 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Opus 4.8·high · Sonnet 4.6·low · Haiku 4.5·req-low · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh |
| SenseBench (WSD) · prompt p003 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p003 · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-09-30 ✓ | xmodel | Opus 4.7·medium · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh |
| SenseBench (WSD) · prompt p004 sense-bench.com | SenseBench lexEN v1, word-sense disambiguation, prompt set p004 (new prompt variant) · accuracy in % (± in the flags), cost in USD per million items as the board prices it; effort from the model name; read 2026-10-15 ✓ | xmodel | Haiku 5.5·high · Haiku 5.5·low · Haiku 5.5·max · Haiku 5.5·medium · Haiku 5.5·nothink · Haiku 5.5·xhigh · Opus 4.8·xhigh |
| ServeLearnBench Avg. Hidden (Blind) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Blind; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Opus 5·high |
| ServeLearnBench Avg. Hidden (CH) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness CH; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Opus 5·high |
| ServeLearnBench Avg. Hidden (Mem0) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Mem0; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Opus 5·high |
| ServeLearnBench Avg. Hidden (Oracle) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness Oracle; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Opus 5·high |
| ServeLearnBench Avg. Hidden (RAG) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness RAG; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Opus 5·high |
| ServeLearnBench Avg. Hidden (SkillOpt) arxiv-2610.07792 | ServeLearnBench (Infini-AI Lab, arXiv 2610.07792, 6 Oct 2026) Table 2: harness SkillOpt; Avg. H = mean Hidden-case reward over 9 settings (Retail/Banking/Pitch x L1-L3); identical task streams; cost = total USD of the harness lanes at official/Fireworks prices of 2026-09-06; unified high-effort regime; read 2026-10-20 ✓ | xmodel | Opus 5·high |
| serverworks-1m serverworks | config ✓ | xmodel | Sonnet 4.6·default |
| serverworks-200k serverworks | config ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies): MCC on primary studies sesr-eval-2610.10633 | SESR-Eval screening of SE systematic reviews (SESR-Eval RQ1.2 SESR-Eval-Mini 5% sample (1,726 primary studies)): LLM title/abstract screening via OpenRouter + Pydantic AI, MCC pooled over primary studies (Accuracy/Precision/Recall/F1/lost evidence and the secondary-study table are other summaries of the same runs: rule 8); cost = USD per secondary study (5% sample) from the secondary-study table; Claude 3.7 Sonnet is an old model whose cost is a previously recorded estimate; arXiv 2610.10633 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·low |
| Simbian Cyber Defense Benchmark simbian | median-cost-per-investigation ✓ | xmodel | Sonnet 4.6·default · Opus 4.8·default · Opus 4.7·default |
| SimpleBench multiple-choice (AVG@5) with cost per run simplebench | SimpleBench >200 private multiple-choice questions (spatio-temporal, social intelligence, trick questions); score = Score (AVG@5) as printed; cost = modelMeta.cpr 'pre-computed cost to run the whole benchmark once (measured tokens x OpenRouter price)' USD; tout = measured output tokens (unit unstated); live board, rows added between 2025-10 and 2026-10-01 (dateAdded printed); read 2026-10-19 ✓ | xmodel | Opus 5.5·default · Fable 5.1·default · Fable 5·default · Opus 5·default · Sonnet 5.5·default · Opus 4.8·default · Opus 4.7·default · Sonnet 5·default |
| SimpleBench open-ended leaderboard simplebench | simple-bench.com static/js/leaderboard-data.js, block openEndedData (3 models; open-ended answers variant of SimpleBench; n/attempts not stated in the payload); read 2026-10-20; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| Simulation-output evidence (arXiv 2610.26836): Agent-Raw100k (first 100k characters inline), full pool of 200 items arxiv-2610.26836 | keyed-value questions about simulation outputs (arXiv 2610.26836, 19 Oct 2026); Agent-Raw100k condition, headline run, accuracy on the full pool (200 items; Table 7); temperature 0; Claude without extended thinking, GPT and Gemini at API defaults with no reasoning-effort setting; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·nothink |
| Simulation-output evidence (arXiv 2610.26836): Agent-TOC (retrieval agent with table of contents), full pool of 200 items arxiv-2610.26836 | keyed-value questions about simulation outputs (arXiv 2610.26836, 19 Oct 2026); Agent-TOC condition, headline run, accuracy on the full pool (200 items; Table 7; the 158 verified and 134 strict subsets are the same runs); temperature 0; Claude without extended thinking, GPT and Gemini at API defaults with no reasoning-effort setting; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·nothink |
| Simulation-output evidence (arXiv 2610.26836): Chain-of-thought prompting without tools, 158 verified items arxiv-2610.26836 | keyed-value questions about simulation outputs (arXiv 2610.26836, 19 Oct 2026); CoT (no tools) headline run, accuracy on the 158 verified items (Table 1); temperature 0; Claude without extended thinking, GPT and Gemini at API defaults with no reasoning-effort setting; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·nothink |
| Simulation-output evidence (arXiv 2610.26836): Direct prompting without tools, 158 verified items arxiv-2610.26836 | keyed-value questions about simulation outputs (arXiv 2610.26836, 19 Oct 2026); Direct (no tools) headline run, accuracy on the 158 verified items (Table 1); temperature 0; Claude without extended thinking, GPT and Gemini at API defaults with no reasoning-effort setting; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·nothink |
| Single-cell (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| SkillForgeBench: skill-conditioned agent tasks, strict completion score with skill arxiv-2610.14918 | 842 tasks, OpenHands harness, LLM-judge rubrics; Table 2 of arXiv 2610.14918; read 2026-10-15 ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default · Haiku 4.5·solo |
| SkillsBench arxiv-2602.12670 | Claude Code 2.1.19 · temp 0, thinking NS ✓ | xmodel | Opus 4.8·max · Opus 4.7·max · Sonnet 4.6·max · Opus 4.7·high |
| SkillsBench (no skills) arxiv-2602.12670 | arXiv 2602.12670v4 T2+T15 · same runs' no-Skills condition ✓ | xmodel | Opus 4.8·max · Opus 4.7·max · Sonnet 4.6·max · Opus 4.7·high |
| SkillsBench v1.1, Claude Code, skill condition Anthropic Skill-Creator arxiv-2608.23417 | SkillAlchemy (Beihang et al., arXiv 2608.23417, 24 Aug 2026) Table 1: SkillsBench v1.1, 87 tasks, avg@5 task pass rate, harness Claude Code, skill condition 'Anthropic Skill-Creator'; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SkillsBench v1.1, Claude Code, skill condition Human-Curated Skill arxiv-2608.23417 | SkillAlchemy (Beihang et al., arXiv 2608.23417, 24 Aug 2026) Table 1: SkillsBench v1.1, 87 tasks, avg@5 task pass rate, harness Claude Code, skill condition 'Human-Curated Skill'; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SkillsBench v1.1, Claude Code, skill condition MUSE-Autoskill arxiv-2608.23417 | SkillAlchemy (Beihang et al., arXiv 2608.23417, 24 Aug 2026) Table 1: SkillsBench v1.1, 87 tasks, avg@5 task pass rate, harness Claude Code, skill condition 'MUSE-Autoskill'; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SkillsBench v1.1, Claude Code, skill condition No Skill arxiv-2608.23417 | SkillAlchemy (Beihang et al., arXiv 2608.23417, 24 Aug 2026) Table 1: SkillsBench v1.1, 87 tasks, avg@5 task pass rate, harness Claude Code, skill condition 'No Skill'; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SkillsBench v1.1, Claude Code, skill condition OpenAI Skill-Creator arxiv-2608.23417 | SkillAlchemy (Beihang et al., arXiv 2608.23417, 24 Aug 2026) Table 1: SkillsBench v1.1, 87 tasks, avg@5 task pass rate, harness Claude Code, skill condition 'OpenAI Skill-Creator'; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SkillsBench v1.1, Claude Code, skill condition OpenSkill arxiv-2608.23417 | SkillAlchemy (Beihang et al., arXiv 2608.23417, 24 Aug 2026) Table 1: SkillsBench v1.1, 87 tasks, avg@5 task pass rate, harness Claude Code, skill condition 'OpenSkill'; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SkillsBench v1.1, Claude Code, skill condition SkillAlchemy arxiv-2608.23417 | SkillAlchemy (Beihang et al., arXiv 2608.23417, 24 Aug 2026) Table 1: SkillsBench v1.1, 87 tasks, avg@5 task pass rate, harness Claude Code, skill condition 'SkillAlchemy'; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| Snorkel Terminal-Bench+ snorkel | Snorkel blog 23 Sep · 24 tasks pass@1, score only, effort unstated ✓ | xmodel | Opus 5.5·default · Opus 5·default · Fable 5.1·default |
| snow-cc snowflake | config ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| snow-coco snowflake | config ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| SOC T1a concentration estimation (MAE g C/kg, lower better) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Sonnet 5·nothink |
| SOC T1b concentration estimation with extra predictors (MAE g C/kg, lower better) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Sonnet 5·nothink |
| SOC T2 stock calculation accuracy (%) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Sonnet 5·nothink |
| SOC T3 pairwise paddock choice accuracy (%) soc-llm-2610.18201 | Australian SOC-LLM benchmark Table 4: 20 responses/item, no tools/retrieval, 1,024-token limit; responses 29 Sep-5 Oct 2026; cost = provider-rate estimate (list prices frozen 2026-09-29), T1 cost covers T1a and T1b (attached to T1a only); arXiv 2610.18201; read 2026-10-20 ✓ | xmodel | Sonnet 5·nothink |
| Spatial transcriptomics (S55) anthropic-syscard | Sonnet 5.5 card p142 · 4 models at max, bio safeguards off, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Fable 5.1·max · Opus 5.5·max |
| SpatialBench-Long pass rate, Claude Code spatialbenchlong-2605.28065 | SpatialBench-Long 24 spatial-biology evaluations x 3 replicates = 72 runs per pair, deterministic final-answer grading, harness Claude Code; metric pass runs/72 %; cost = USD per run from harness metadata (Pi only); arXiv 2605.28065 Table 2 (27 May 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| SpatialBench-Long pass rate, Pi terminal coding harness spatialbenchlong-2605.28065 | SpatialBench-Long 24 spatial-biology evaluations x 3 replicates = 72 runs per pair, deterministic final-answer grading, harness Pi terminal coding harness; metric pass runs/72 %; cost = USD per run from harness metadata (Pi only); arXiv 2605.28065 Table 2 (27 May 2026); read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default |
| SpeedrunBench ONLINE Pokemon Red: frames to the first badge (lower is better) speedrunbench-2610.08076 | SpeedrunBench Sec. 4.3: ONLINE Pokemon Red play in OpenCode, frames to Badge 1 (human world record ~40,435); only three vision-capable proprietary models reach it (Kimi K3 stalls: no value); all other figures (offline, per-game) are charts only and not extracted; arXiv 2610.08076 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 5·default |
| SpireBench: Slay the Spire Ascension 20 Ironclad, mean final floor reached arxiv-2610.12596 | 10 runs per model on the same held-out seeds, themed reskin with note-taking; Table 1 and Table 26 of arXiv 2610.12596; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Fable 5.1·low · Opus 5·max · Opus 5·low · Sonnet 5·max · Sonnet 5·low · Haiku 4.5·solo |
| SpireClimb (Epoch AI) epoch.ai | epoch.ai/benchmarks/spireclimb: win rate averaged over Slay the Spire 1 (A20 Heart, 60 runs) and Slay the Spire 2 (A10, 75 runs); computer-use harness (Inspect ReAct), each model at maximum reasoning setting; mean_score; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·max · Fable 5.1·max · Opus 5.5·max |
| spullara/models grid-path single prompt (pass/fail, n=1) github-spullara-models | one prompt solve_grid (max-sum path, every 3rd move diagonal), 4x4 test grid, one run per model, max_tokens 32768, no reasoning/effort param sent; pass = syntax valid + path valid + returned sum = recomputed sum (optimality not tested); eval date per row; commit 59c2b35a; gridpath = gridsum (same runs, rule 8); read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Opus 4.8·default · Opus 5.5·default · Sonnet 4.6·default · Sonnet 5·default · Sonnet 5.5·default · Haiku 5.5·default |
| SRE-Bench (table) openai-gpt6-astra-launch | SRE-Bench; OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Opus 5·default |
| SRE-Bench public-facing (score /6, cost $/instance) sre-bench-2608.11469 | SRE-Bench Table 4: 262 instances x 6 tasks, one run each, mini-SWE-agent, 500 steps/6h, cyber safeguards on, averages over graded runs only; arXiv 2608.11469 (11 Aug 2026); Table 7 packed subset is the same runs (rule 8); read 2026-10-20 ✓ | xmodel | Fable 5.1·default · Opus 5·default |
| sss_ailabo: 3 tasks (game/LP/bugfix), total time/cost, n=1 note-sss_ailabo | note.com/sss_ailabo/n/n218e2eddafa9; Claude Code or Codex, 1 run per model per task, 3 tasks summed; effort not named, article says "今回の標準の設定" (this time's standard settings) -> default; read 2026-10-01 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default · Sonnet 5·default |
| STAGE-Claw arxiv-2606.10394 | OpenClaw · reasoning DISABLED, temp 0 ✓ | xmodel | Opus 4.7·nothink · Sonnet 4.6·nothink |
| Stet 25-PR stet.sh | Claude Code · 25 replayed PRs, Opus 5 vs Opus 4.8 both at medium, relative cost ✓ | xmodel | Opus 4.8·medium · Opus 5·medium |
| stet-pr24 stet.sh | config ✓ | xmodel | Sonnet 5·low · Sonnet 5·max · Opus 4.8·low · Opus 4.8·max · Sonnet 5·medium · Opus 4.8·medium · Sonnet 5·high · Opus 4.8·high · Sonnet 5·xhigh · Opus 4.8·xhigh |
| StoreBench: full-year composite arxiv-2610.10942 | StoreBench full-year task (Table 2): 365-window episode, Claude Code with maximum extended thinking (32,000 tokens per turn) and context compaction, 3 seeds x 3 attempts; cost = full-year rollout 'about' per App. H; read 2026-10-20 ✓ | xmodel | Opus 4.8·think-32k · Fable 5·think-32k |
| StoreBench: short-horizon composite arxiv-2610.10942 | StoreBench (AfterQuery, arXiv 2610.10942, 7 Oct 2026) Table 1: 11 dense tasks x 3 seeds x 3 attempts, goose harness + 29-tool MCP, matched reasoning effort; composite = 0.7 business + 0.2 reliability + 0.1 continuity; cost = per short-horizon rollout 'at first-party prices' (App. H); read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| Supabase evals (with skills) supabase.com | Supabase evals in Claude Code, with skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Sonnet 5·high |
| Supabase evals (without skills) supabase.com | Supabase evals in Claude Code, without skills, 23 evals × 3 runs · pass rate in % (computed by us from the published runs), tokens per run; reasoningEffort high from the presets; no cost; read 2026-10-01 ✓ | xmodel | Opus 5.5·high · Sonnet 5·high |
| SuperCLUE 2026年5月 代码生成 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 代码生成, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Opus 4.8·high · Opus 4.7·high |
| SuperCLUE 2026年5月 幻觉控制 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 幻觉控制, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Opus 4.8·high · Opus 4.7·high |
| SuperCLUE 2026年5月 数学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 数学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Opus 4.8·high · Opus 4.7·high |
| SuperCLUE 2026年5月 智能体(任务规划) superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 智能体(任务规划), closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Opus 4.8·high · Opus 4.7·high |
| SuperCLUE 2026年5月 科学推理 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 科学推理, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Opus 4.8·high · Opus 4.7·high |
| SuperCLUE 2026年5月 精确指令遵循 superclue-general | SuperCLUE general leaderboard (CLUE), edition 2026年5月, column 精确指令遵循, closed-book private questions via API, n/attempts/judge not stated on the sheet; sheet 总排行榜 only; read 2026-10-19 ✓ | xmodel | Opus 4.8·high · Opus 4.7·high |
| superpowers github-superpowers | config ✓ | xmodel | Opus 5·default · Sonnet 5·default · Haiku 4.5·solo |
| superpowers PR2258 github-superpowers | superpowers-evals campaign 5 · Opus 5 vs Opus 4.8 xhigh, interrupted, base+head pooled ✓ | xmodel | Opus 5·xhigh · Opus 4.8·xhigh |
| Surface Evolver bench yhenon-surface-evolver | yhenon/surface-evolver-llm-eval aggregates.json · 16 runs/model, OpenRouter recorded cost, Opus 4.8 none/high, Sonnet 5 medium, Fable 5 high ✓ | xmodel | Opus 4.8·nothink · Opus 4.8·high · Sonnet 5·medium · Fable 5·high |
| SurfSense capability screen, case board-pack (verdict of the screening run) surfsense-sweep | SurfSense sweep 2026-10-07/08 via OpenRouter, case=board-pack; score = verdict of the retained run (100 pass / 0 fail); 'below'-level models are not run on later cases (conditional selection); cost = recorded USD of that run; effort not in manifest; models 'assumed' not run are excluded; read 2026-10-20 ✓ | xmodel | Haiku 5.5·default |
| SurfSense capability screen, case pdf-brief (verdict of the screening run) surfsense-sweep | SurfSense sweep 2026-10-07/08 via OpenRouter, case=pdf-brief; score = verdict of the retained run (100 pass / 0 fail); 'below'-level models are not run on later cases (conditional selection); cost = recorded USD of that run; effort not in manifest; models 'assumed' not run are excluded; read 2026-10-20 ✓ | xmodel | Haiku 5.5·default |
| SurfSense capability screen, case smoke (verdict of the screening run) surfsense-sweep | SurfSense sweep 2026-10-07/08 via OpenRouter, case=smoke; score = verdict of the retained run (100 pass / 0 fail); 'below'-level models are not run on later cases (conditional selection); cost = recorded USD of that run; effort not in manifest; models 'assumed' not run are excluded; read 2026-10-20 ✓ | xmodel | Haiku 5.5·default |
| Surge AI · Antidote: Everyday Edition surgehq.ai | Surge AI Antidote: Everyday Edition leaderboard (surgehq.ai/benchmarks/antidote); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Fable 5·high · Fable 5.1·high · Opus 5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 4.6·high · Haiku 4.5·solo |
| Surge AI · Chartography surgehq.ai | Surge AI Chartography leaderboard (surgehq.ai/benchmarks/chartography); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Sonnet 5·max · Opus 4.7·max · Opus 4.8·max |
| Surge AI · ComplexConstraints surgehq.ai | Surge AI ComplexConstraints leaderboard (surgehq.ai/benchmarks/complex-constraints); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max |
| Surge AI · DAYJOB: Finance surgehq.ai | Surge AI DAYJOB: Finance leaderboard (surgehq.ai/benchmarks/dayjob-finance); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max |
| Surge AI · DAYJOB: Healthcare surgehq.ai | Surge AI DAYJOB: Healthcare leaderboard (surgehq.ai/benchmarks/dayjob-healthcare); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max |
| Surge AI · EnterpriseBench: CoreCraft surgehq.ai | Surge AI EnterpriseBench: CoreCraft leaderboard (surgehq.ai/benchmarks/enterprisebench-corecraft); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Fable 5.1·max · Opus 5.5·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·max · Sonnet 4.6·high |
| Surge AI · GDP.pdf surgehq.ai | Surge AI GDP.pdf leaderboard (surgehq.ai/benchmarks/gdp-pdf); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5·max · Fable 5.1·max · Opus 4.8·max · Opus 5·max · Opus 4.7·max · Sonnet 4.6·max |
| Surge AI · GDP.xlsx surgehq.ai | Surge AI GDP.xlsx leaderboard (surgehq.ai/benchmarks/gdp-xlsx); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Sonnet 5·max |
| Surge AI · HANDBOOK.md Agents surgehq.ai | Surge AI HANDBOOK.md Agents leaderboard (surgehq.ai/benchmarks/handbook); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max |
| Surge AI · Hemingway-bench surgehq.ai | Surge AI Hemingway-bench leaderboard (surgehq.ai/benchmarks/hemingway-bench); Elo from human preference comparisons (95% CI in flags); effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Fable 5·high · Fable 5.1·high · Opus 5·high · Opus 4.7·high · Opus 4.8·high · Sonnet 4.6·high |
| Surge AI · Riemann-bench surgehq.ai | Surge AI Riemann-bench leaderboard (surgehq.ai/benchmarks/riemann-bench); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max |
| Surge AI · sudo L7 surgehq.ai | Surge AI sudo L7 leaderboard (surgehq.ai/benchmarks/sudo-l7); score in %; effort from the model label; vendor-run; no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max |
| SWE Atlas Fig. 14: cost per task vs Pass@1 on QnA + Test Writing scale-swe-atlas | SWE Atlas Appendix G Figure 14 (arXiv 2605.08366): per-trial token usage from the scaffold logs x each provider's published per-million-token API rates; 642 trials per model (372 Q&A + 270 Test Writing); native vendor scaffold; both axes DIGITISED from vector markers (axes calibrated on tick marks, residual 2e-5 pt); price date not given; read 2026-10-19 ✓ | xmodel | Sonnet 4.6·high · Opus 4.7·xhigh |
| SWE Atlas Pass@1 (QnA + Test Writing + Refactoring pooled), mini scale-swe-atlas | SWE Atlas (Scale AI, arXiv 2605.08366) Table 1, Pass@1 (QnA + Test Writing + Refactoring pooled), mean of 3 trials, mini-SWE-agent (common scaffold); Harbor framework, Modal sandboxes; rubric judge anthropic/claude-opus-4-5-20251101; open models via Fireworks AI; no cost in Table 1; paper May 2026; read 2026-10-19 ✓ | xmodel | Opus 4.7·xhigh |
| SWE Bench Pro (xAI Grok 4.5 post) xai-grok-4-5-launch | SWE Bench Pro resolve rate %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; output tokens per task for Grok 4.5 and Opus 4.8 from the caption; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max · Opus 4.7·max |
| SWE Marathon (xAI Grok 4.5 post) xai-grok-4-5-launch | SWE Marathon resolution rate pass@1 %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; read 2026-10-19 ✓ | xmodel | Opus 4.8·max · Fable 5·max · Opus 4.7·max |
| SWE-bench Multilingual (S55) anthropic-syscard | Sonnet 5.5 card table 8.1.A · 3 models at max, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Opus 5.5·max |
| SWE-bench Multimodal (S55) anthropic-syscard | Sonnet 5.5 card table 8.1.A · 3 models at max, score-only ✓ | xmodel | Sonnet 5.5·max · Sonnet 5·max · Opus 5.5·max |
| SWE-bench Pro (cards) anthropic-syscard | Sonnet 5 card and Sonnet 5.5 card table 8.1.A · 5 Claude models at max, score only ✓ | xmodel | Sonnet 5·max · Sonnet 4.6·max · Opus 4.8·max · Sonnet 5.5·max · Opus 5.5·max |
| SWE-Bench Pro (Public) (table) openai-gpt55-launch | SWE-Bench Pro (Public); OpenAI GPT-5.5 launch post tables (the repeated summary table, first table, is skipped as a duplicate); 'Evals of GPT were run with reasoning effort set to xhigh and were conducted in a research environment'; published 2026-04-23; read 2026-10-19 ✓ | xmodel | Opus 4.7·default |
| SWE-Bench Pro (table) openai-gpt56-launch | SWE-Bench Pro; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default |
| SWE-Bench Pro V2 (Scale) scale-seal | Scale Labs board via Wayback 23 Sep 2026 · full set, Claude Code, one effort per model, score only, saturated ✓ | xmodel | Opus 5·xhigh · Fable 5.1·high · Sonnet 5·xhigh |
| SWE-Bench Pro V2 HARD (Scale) scale-seal | Scale Labs board via Wayback 23 Sep 2026 · HARD subset, Claude Code, score only ✓ | xmodel | Opus 5·xhigh · Fable 5.1·high · Sonnet 5·xhigh · Haiku 4.5·req-xhigh |
| SWE-bench-Live Go (AMI Agent) swe-bench-live.github.io | SWE-bench-Live, Go set (138 tasks), AMI Agent scaffold · resolved in %; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| SWE-Duel self-scaling arena (Bradley-Terry strength), mini-swe-agent harness arxiv-2610.12569 | SWE-Duel (arXiv 2610.12569, 8 Oct 2026): tournament of coding agents creating and solving bugs against each other, main tournament Bradley-Terry log-strength (Table 2, one joint fit for all competitors); mini-swe-agent harness; models via OpenRouter, temperature 0.2; reasoning effort not stated; tournament-total cost per competitor is not a per-task cost (different match counts) so no cost rows; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SWE-Duel self-scaling arena (Bradley-Terry strength), vendor-native harness (Codex CLI / Claude Code) arxiv-2610.12569 | SWE-Duel (arXiv 2610.12569, 8 Oct 2026): tournament of coding agents creating and solving bugs against each other, main tournament Bradley-Terry log-strength (Table 2, one joint fit for all competitors); vendor-native harness (Codex CLI / Claude Code); models via OpenRouter, temperature 0.2; reasoning effort not stated; tournament-total cost per competitor is not a per-task cost (different match counts) so no cost rows; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| SWE-Marathon (GLM-5.2 card, evaluated by Abundant AI) zai-hf-glm-5.2 | SWE-Marathon, 1M context 128K output; evaluation by Abundant AI; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| SWE-Marathon v1.1 (GLM-5.3 card) zai-hf-glm-5.3 | SWE-Marathon v1.1, Claude Code 2.1.207, temperature=1.0 top_p=0.95 128K output, 1M context; footnote: only GLM-5.3 stated as evaluated by Z.ai; GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·default · Fable 5·default |
| swe-rebench 07/26 swe-rebench | ReAct minimal · Opus5/Fable5/Sonnet5 all at high ✓ | xmodel | Opus 5·high · Fable 5·high · Sonnet 5·high |
| SWE-sweep board (Meta/Harvard/UW/Stanford), mini-SWE-agent swesweep | SWE-sweep: 100 repositories / 4.1k bugs, agent finds and fixes bugs with no hint, hidden PR tests + original suite, scaffold mini-SWE-agent, score = fraction of all bugs resolved, cost = USD per repository (page 'USD / repo', averaged over evaluated repositories, includes non-deprecated retries); page 'Updated September 24, 2026'; read 2026-10-20; read 2026-10-20 ✓ | xmodel | Opus 5·xhigh |
| SWEPatchTest joint validation rate (JVR), Fable 5.1 vs Opus 5 arxiv-2610.18004 | arXiv 2610.18004 Table 2; native coding agents; JVR %; read 2026-10-18 ✓ | xmodel | Fable 5.1·default · Opus 5·default |
| SyclKittens causal GQA kernel, cold-iterative, condition 'examples' (peak@3 vs PyTorch XPU SDPA) arxiv-2610.04277 | 3 cold-iterative trials per cell on one Intel Max GPU, Claude Code 2.1.214; Table 4 of arXiv 2610.04277; read 2026-10-15 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Haiku 4.5·solo |
| SyclKittens causal GQA kernel, cold-iterative, condition 'raw' (peak@3 vs PyTorch XPU SDPA) arxiv-2610.04277 | 3 cold-iterative trials per cell on one Intel Max GPU, Claude Code 2.1.214; Table 4 of arXiv 2610.04277; read 2026-10-15 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Haiku 4.5·solo |
| SyclKittens causal GQA kernel, cold-iterative, condition 'syclkittens' (peak@3 vs PyTorch XPU SDPA) arxiv-2610.04277 | 3 cold-iterative trials per cell on one Intel Max GPU, Claude Code 2.1.214; Table 4 of arXiv 2610.04277; read 2026-10-15 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Haiku 4.5·solo |
| SyclKittens single-GPU BF16 GEMM, cold-iterative, condition 'examples' (peak@3 vs oneDNN) arxiv-2610.04277 | 3 cold-iterative trials per cell, Claude Code 2.1.214; Table 4/10 of arXiv 2610.04277; read 2026-10-15 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Haiku 4.5·solo |
| SyclKittens single-GPU BF16 GEMM, cold-iterative, condition 'raw' (peak@3 vs oneDNN) arxiv-2610.04277 | 3 cold-iterative trials per cell, Claude Code 2.1.214; Table 4/10 of arXiv 2610.04277; read 2026-10-15 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Haiku 4.5·solo |
| SyclKittens single-GPU BF16 GEMM, cold-iterative, condition 'syclkittens' (peak@3 vs oneDNN) arxiv-2610.04277 | 3 cold-iterative trials per cell, Claude Code 2.1.214; Table 4/10 of arXiv 2610.04277; read 2026-10-15 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default · Haiku 4.5·solo |
| SynthBench Parity Score (SPS) vs human survey distributions, mean of 4 datasets, althing prompt github-synthbench | DataViking-Tech/SynthBench leaderboard-results (13-14 Oct 2026); 100 questions x 30 samples per dataset (globalopinionqa, gss, opinionsqa, subpop); via OpenRouter; persona-conditioned (althing/) prompt; SPS 0..1 (1 = human-like); mean over datasets of the mean over available runs, computed by us; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·default · Sonnet 4.6·default · Sonnet 5.5·default |
| SynthBench Parity Score (SPS) vs human survey distributions, mean of 4 datasets, plain prompt github-synthbench | DataViking-Tech/SynthBench leaderboard-results (13-14 Oct 2026); 100 questions x 30 samples per dataset (globalopinionqa, gss, opinionsqa, subpop); via OpenRouter; plain prompt; SPS 0..1 (1 = human-like); mean over datasets of the mean over available runs, computed by us; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·nothink · Sonnet 4.6·default · Sonnet 5.5·default |
| Synthorai (ja mirror): 4-question tool-reading loop, Sonnet 5.5 (sweep) and Opus 5.5 default synthorai | native Messages API via gateway, 4 tasks x3 reps per row (n=12), cost computed from list price; qiita.com/Synthorai/items/98e776a66f483f3ceeaf (mirror of synthorai.io/ja/blog/claude-sonnet-5-5-cost/); read 2026-10-05 ✓ | xmodel | Sonnet 5.5·default · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·max · Opus 5.5·default |
| T2J-Bench PyTorch->JAX repo migration, Claude-Code, Opus 5 vs Sonnet 5 arxiv-2605.29054 | arXiv 2605.29054 (updated 14 Oct 2026); 63 tasks; reward 0-1 mean of 3 seeds; H100; 3-h wall budget; harness Claude-Code; read 2026-10-18 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| T2J-Bench PyTorch->JAX repo migration, OpenCode, Opus 5 vs Sonnet 5 arxiv-2605.29054 | arXiv 2605.29054 (updated 14 Oct 2026); 63 tasks; reward 0-1 mean of 3 seeds; H100; 3-h wall budget; harness OpenCode; read 2026-10-18 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| T2J-Bench PyTorch->JAX repo migration, OpenHands, Opus 5 vs Sonnet 5 arxiv-2605.29054 | arXiv 2605.29054 (updated 14 Oct 2026); 63 tasks; reward 0-1 mean of 3 seeds; H100; 3-h wall budget; harness OpenHands; read 2026-10-18 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| Takuya__ (qiita, item 2a513efa) 12 sqlite-utils bug-fix/feature tasks (incl 4 real upstream issues), x2 runs, full test suite qiita-takuya55 | Claude Code 2.1.293, Read/Edit/Write/Grep/Glob/Bash, pass = all hidden+existing tests (~1000) green; read 2026-10-08 ✓ | xmodel | Haiku 5.5·medium · Haiku 5.5·high · Sonnet 5.5·medium · Haiku 4.5·solo |
| Takuya__ (qiita, item 2a513efa) Banking77 intent classification + "narrow candidates first" prompt addition qiita-takuya55 | same task as qiita-takuya55-banking77 but with an added system-prompt instruction to narrow candidates first: different configuration; read 2026-10-08 ✓ | xmodel | Haiku 5.5·medium · Sonnet 5.5·medium |
| Takuya__ (qiita, item 2a513efa) Banking77 intent classification, 154 msgs/77 labels, no tools, single API call qiita-takuya55 | no-tools system-prompt label list, single API call per message, cost computed from measured tokens x price table per 1,000 msgs; read 2026-10-08 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max · Haiku 4.5·solo · Sonnet 5.5·medium |
| Takuya__ (qiita, item 2a513efa) code-reading Q&A, sqlite-utils 3.38, 9 questions x2 runs qiita-takuya55 | Claude Code 2.1.293 claude -p stream-json, Read/Grep/Glob/Bash only, cost = measured tokens x official price table; read 2026-10-08 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Sonnet 5.5·medium · Haiku 4.5·solo |
| Takuya__ (qiita, item 2a513efa) Sonnet-lead orchestration, 9 Q x3 runs, pure single-model baselines only qiita-takuya55 | Claude Code 2.1.293, Explore subagent delegation; only the two PURE single-model runs are rows (mixed Sonnet-lead+Haiku-Explore run excluded: not a model x effort couple); read 2026-10-08 ✓ | xmodel | Sonnet 5.5·medium · Haiku 5.5·default |
| TashonBraganca/messy-docs-bench: field-extraction accuracy, Opus 5.5 vs Sonnet 5 github-messydocsbench | 137 messy documents (receipts/invoices/scans/W-2/statements/contracts), field-level extraction accuracy; github.com/TashonBraganca/messy-docs-bench, src/messydocs_summary.json; read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5·default |
| Task-assignment accuracy, clone teams (Haiku 4.5 vs Sonnet 5.5) arxiv-2610.19051 | 300 episodes, one sample per model; Table 2 of arXiv 2610.19051; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Sonnet 5.5·default |
| TasteVal compute multiplier vs expert baseline (Researcher models, 8 ML-research tasks) arxiv-2610.06824 | 8 uncontaminated AI R&D tasks, 6 seeds/model/task, 40 H100-hours; Researcher model + fixed Coder (Opus 4.8); Table 2 of arXiv 2610.06824; score only (cost in round 2); read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Opus 5·medium · Opus 5·low · Opus 4.8·max · Opus 4.7·max |
| TasteVal performance multiplier vs expert baseline arxiv-2610.06824 | 8 uncontaminated AI R&D tasks, 6 seeds/model/task, 40 H100-hours; Researcher model + fixed Coder (Opus 4.8); Table 2 of arXiv 2610.06824; score only (cost in round 2); read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Opus 5·medium · Opus 5·low · Opus 4.8·max · Opus 4.7·max |
| TasteVal total cost per run, 6-task subset (cost only) arxiv-2610.06824 | total API spend of all agents + GPU at $2.5/h per run, standard scaffold, 6 tasks; read from the log axis of Fig. 6 (right); score rows live in the TasteVal score groups; read 2026-10-15 ✓ | xmodel | Opus 5·low · Opus 5·medium · Opus 5·max · Opus 5.5·max |
| TaxCalcBench TY25, no web search (column-tax/tax-calc-bench) column-tax-tax-calc-bench | 50 TY25 federal+state tax-return cases, 1 run each; strict-correct-return % is the score (lenient% also in the evidence but not used here per instructions); cost = mean litellm-estimate USD/return (n priced noted when <50); source: taxcalc_ty25_claude_by_effort.txt, re-verified against raw evaluation_result_*.md for opus-5.5/ultrathink (40.00%/50.00%/$1.135, exact match); read 2026-10-01 ✓ | xmodel | Fable 5·xhigh · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·max · Fable 5.1·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·max · Opus 4.8·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·max · Opus 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·max · Opus 5.5·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·max · Sonnet 5·xhigh · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5.5·xhigh · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max |
| TaxCalcBench TY25, with web search (column-tax/tax-calc-bench) column-tax-tax-calc-bench | Same as taxcalc-ty25-notool but with the web-search tool enabled; read 2026-10-01 ✓ | xmodel | Fable 5·xhigh · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·max · Fable 5.1·xhigh · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·max · Opus 4.8·xhigh · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·max · Opus 5·xhigh · Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·max · Opus 5.5·xhigh · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·max · Sonnet 5.5·xhigh · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·max · Haiku 5.5·xhigh · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·max |
| TB-Science 0.1 (S55) anthropic-syscard | Sonnet 5.5 card p114 · Claude Code --bare, 5 models at max, score-only ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max |
| tb21cc tbench | config ✓ | xmodel | Fable 5·xhigh · Opus 4.8·high · Sonnet 5·high · Opus 4.7·max |
| tb21t2 tbench | config ✓ | xmodel | Fable 5·high · Opus 4.7·max |
| TCSAlgBench 10-round discussion acceptance % (calibration runs) arxiv-2609.35606 | TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); separate runs of Table 9 (acceptance %, RMS-CE reported separately); read 2026-10-19 ✓ | xmodel | Opus 4.8·xhigh · Opus 4.8·high · Opus 4.8·max |
| TCSAlgBench 10-round discussion, 5-run coverage arxiv-2609.35606 | TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); 10-round discussion, coverage over 5 runs; read 2026-10-19 ✓ | xmodel | Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·high · Fable 5·xhigh |
| TCSAlgBench direct inference, 10-run coverage arxiv-2609.35606 | TCSAlgBench: 398 research-level TCS proof-discovery challenges (STOC/COLT 2026 papers); verifier = three GPT-5.5 high-effort voters, accept if >=2 PASS; all models through Amazon Bedrock, 128K output cap; efforts high/xhigh/max as labelled; accepted challenges (count of 398); direct inference, coverage = challenges accepted in at least one of 10 runs; read 2026-10-19 ✓ | xmodel | Opus 4.8·max · Opus 4.8·xhigh · Opus 4.8·high · Fable 5·xhigh · Fable 5·high |
| Terminal Bench 2.1 (xAI Grok 4.5 post) xai-grok-4-5-launch | Terminal Bench 2.1 score %, harness not stated; xAI Grok 4.5 post 16 Jul 2026; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 4.8·max · Opus 4.7·max |
| Terminal-Bench 2.0 pass@1 (89 tasks), Terminus-2 full-history baseline arxiv-2609.35540 | Terminal-Bench 2.0, 89 tasks, pass@1, Terminus-2 agent, temp 0.6, 8192 tok/turn, 50 turns, mode Baseline; cost estimated mean per task; paper 2026-09-28; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| Terminal-Bench 2.0 pass@1 (89 tasks), Terminus-2 with Continuous Context Management (paper method) arxiv-2609.35540 | Terminal-Bench 2.0, 89 tasks, pass@1, Terminus-2 agent, temp 0.6, 8192 tok/turn, 50 turns, mode CCM; cost estimated mean per task; paper 2026-09-28; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| Terminal-Bench 2.1 (GLM-5.3 card) zai-hf-glm-5.3 | Terminal-Bench 2.1, Claude Code 2.1.207, temperature=1.0 top_p=1 max_new_tokens=65536, 6h timeout; Z.ai GLM-5.3 card 2026-10-14; step 0.1; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| Terminal-Bench 2.1 (OpenAI GPT-5.6 post chart) openai-gpt56-launch | Terminal-Bench 2.1 accuracy, code_mode=true; OpenAI GPT-5.6 launch post chart, publication snapshot Wayback 20260709172124; scores = chart points (pass rates etc.); costs = simulated API cost at regular API pricing, scope (per task or per run) not stated; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| Terminal-Bench 2.1 (table) openai-gpt56-launch | Terminal-Bench 2.1; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default |
| Terminal-Bench 2.1 Terminus-2 (GLM-5.2 card) zai-hf-glm-5.2 | Terminal-Bench 2.1, Terminus-2 parser=json timeout=4h temp 1.0 max_new_tokens=48k 256K context; GLM-5.2 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Terminal-Bench 2.1, Terminus-2 same-harness block (T1 paper Table 1) arxiv-2609.11042 | arXiv 2609.11042 v1 (10 Sep 2026) Table 1 upper block 'evaluated under the same harness as ours (Harbor/Terminus-2)': Terminal-Bench 2.1 resolved %; rows from the authors' evaluation workbook (provenance of each model's run not stated); LHTB column of the same table = held group lhtb1-mean, not re-made; own model (Qwen3.5-122B + RL) and Muse/Hy3/MiniMax are outside the eight providers; read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default |
| Terminal-Bench 3.0 (GLM-5.3 card) zai-hf-glm-5.3 | Terminal-Bench-3, Claude Code 2.1.207, avg@3, 600 turns, 10h timeout, Tool Search disabled, official verifier; GLM-5.3 card 2026-10-14; read 2026-10-19 ✓ | xmodel | Opus 4.8·max · Fable 5·max |
| Terminal-Bench 4.0 tbench | tbench.ai primary payload · 66 tasks × 330 trials, Claude Code, all at max; cost basis undocumented ✓ | xmodel | Fable 5·max · Fable 5.1·high · Fable 5.1·low · Fable 5.1·max · Fable 5.1·medium · Fable 5.1·xhigh · Opus 4.8·max · Opus 5·high · Opus 5·low · Opus 5·max · Opus 5·medium · Opus 5·xhigh · Sonnet 5·max · Opus 5.5·max · Sonnet 5.5·max |
| Terminal-Bench 4.0 (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | Terminal-Bench 4.0; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5·max · Opus 5·max |
| Terminal-Bench 4.0 (table) openai-gpt6-astra-launch | Terminal-Bench 4.0; OpenAI GPT-6 Astra launch post tables, publication snapshot Wayback 20260903193913; 'Evaluation scores are the maximum at any effort. GPT evaluations were run in our research environment or via our API'; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 5·default |
| Terminal-Bench 4.0 (table) openai-gpt6-astra-launch-0929 | Terminal-Bench 4.0; OpenAI GPT-6 Astra post, LIVE copy (Wayback 20261015182235) carrying the Sep 22/29 update banners; later snapshot of the same tables as the Sep-3 groups (cells changed vs Sep 3 in these columns); 'Evaluation scores are the maximum at any effort'; read 2026-10-19 ✓ | xmodel | Fable 5.1·default · Opus 5·default |
| Terminal-Bench 4.0 (xAI Grok 4.7 post) xai-grok-4-7-launch | Terminal-Bench 4.0 'multi-hour terminal work' %, harness not stated; xAI Grok 4.7 post 21 Sep 2026; read 2026-10-19 ✓ | xmodel | Fable 5.1·max |
| Terminal-Bench Science 0.1 (OpenAI GPT-6 Astra post chart, non-code mode) openai-gpt6-astra-launch | Terminal-Bench Science 0.1; OpenAI GPT-6 Astra launch post chart (publication snapshot), per reasoning effort where the point has an effort label; cost = API cost estimate, method not stated; mode = noncode; published 2026-09-03; read 2026-10-19 ✓ | xmodel | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Fable 5·max · Opus 5·max |
| Terminal-Bench Science 0.1 (OpenAI GPT-6.1 Sol post) openai-gpt61-sol-launch | Terminal-Bench Science 0.1; OpenAI GPT-6.1 Sol post chart, per reasoning effort; cost = 'Cost per task' axis (method not stated); published 2026-09-29; Wayback first capture; read 2026-10-19 ✓ | xmodel | Opus 5.5·max |
| Terminal-Bench v3.0 (xAI Grok 4.6 post) xai-grok-4-6-launch | Terminal-Bench v3.0 %, harness not stated; xAI Grok 4.6 post 12 Aug 2026; read 2026-10-19 ✓ | xmodel | Fable 5·max |
| Terraform code review (26 seeded bugs): Haiku 4.5 / Sonnet 5.5 / Haiku 5.5 on Bedrock github-claude-haiku-terraform-review-benchmark | Bedrock us-east-1 inference profiles, 3 tasks x buggy+clean x 3 runs; score /100; cost per run = sum of 6 reviews at author prices (list+10%); read 2026-10-18 ✓ | xmodel | Haiku 4.5·solo · Sonnet 5.5·default · Haiku 5.5·default |
| Test Cabinet · claude 2.1.203 · shatter v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.203; case shatter v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.204 · shatter v1.0.0 · warhead testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.204; case shatter v1.0.0; variant warhead; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.205 · fathom v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.205; case fathom v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Sonnet 4.6·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.206 · floe-bear v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.206; case floe-bear v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.208 · fathom v1.1.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.208; case fathom v1.1.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.211 · fathom v1.1.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.211; case fathom v1.1.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| Test Cabinet · claude 2.1.211 · valence v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.211; case valence v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.233 · deepcore v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.233; case deepcore v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Opus 5·default |
| Test Cabinet · claude 2.1.233 · fathom v2.0.0 · kindle testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.233; case fathom v2.0.0; variant kindle; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Opus 5·default |
| Test Cabinet · claude 2.1.234 · arc-foundry v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case arc-foundry v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Opus 5·default |
| Test Cabinet · claude 2.1.234 · deepcore v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case deepcore v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Opus 5·default |
| Test Cabinet · claude 2.1.234 · pong v2.1.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.234; case pong v2.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Opus 5·default |
| Test Cabinet · claude 2.1.235 · coil v1.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case coil v1.0.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 4.8·default · Opus 5·default |
| Test Cabinet · claude 2.1.235 · pong v2.1.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case pong v2.1.0; variant base; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Sonnet 4.6·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.235 · pong v2.1.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.235; case pong v2.1.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Opus 4.8·default · Opus 5·default |
| Test Cabinet · claude 2.1.237 · band-of-bots v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.237; case band-of-bots v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.237 · pong v2.1.0 · gyre testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.237; case pong v2.1.0; variant gyre; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Opus 5·default · Sonnet 4.6·default |
| Test Cabinet · claude 2.1.238 · band-of-bots v1.0.0 · default testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.238; case band-of-bots v1.0.0; variant default; engine none; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5·default · Sonnet 5·default |
| Test Cabinet · claude 2.1.302 · pong v3.0.0 · base testcabinet | snapshot.testcabinet.ai 2026-10-20T0238Z (runs.json); harness claude 2.1.302; case pong v3.0.0; variant base; engine simple-2d; score = validator checklist earned/total x100, mean over completed runs; cost = mean cost.comparable (list-price, as the held opencode groups tc-*); efforts not recorded; read 2026-10-20 ✓ | xmodel | Fable 5.1·default · Opus 5.5·default · Sonnet 5.5·default |
| text-to-MRS (Hajdik split) arxiv-2609.30071 | 3-exemplar prompt · Sonnet4.5/Opus5 default, EDM-F1, score-only ✓ | xmodel | Opus 5·default |
| tfcxzaqwerds: 7 coding/build tasks x3 runs (42 total), mean cost+tokens, both 100% pass note-tfcxzaqwerds | note.com/tfcxzaqwerds/n/nab4b132711a2; Claude Code 2.1.284 headless (claude -p), --setting-sources "" --strict-mcp-config (no CLAUDE.md/MCP), sandboxed Bash; cost = Claude Code costUSD (list-price API-equivalent); effort-from-config:"Claude Code 2.1.284、effort は両モデルとも high（API の既定値）"; 7 tasks = 5 impl/bugfix (semver,calc,cron,billing,ja_normalize) + 2 build-and-report tasks (budget webapp, sales-CSV report), 3 runs each = 21 runs/model, 42 total; both models scored 21/21 perfect on hidden tests; read 2026-10-01 ✓ | xmodel | Sonnet 5·high · Sonnet 5.5·high |
| tfcxzaqwerds: 9 hard reasoning questions x3 runs (no tools), score note-tfcxzaqwerds | note.com/tfcxzaqwerds/n/nab4b132711a2; same Claude Code headless setup, effort high, no tools, maze/24-digit-multiplication/logic-puzzles, 9 questions x3 runs = 27 trials/model, 15-min time cap per question; read 2026-10-01 ✓ | xmodel | Sonnet 5·high · Sonnet 5.5·high |
| The Endless Exam (2609.24555) — 69 open-ended maths constructions, Claude Code tools off arxiv-2609.24555 | Table 2, 69 instances, overall score = 100 x mean relative quality vs reference; verified against saved evidence in /work/.salvo14/s18/a1/src/, 2026-10-01; read 2026-10-01 ✓ | xmodel | Opus 5.5·medium · Opus 5.5·high · Fable 5.1·medium · Fable 5.1·high |
| Thinkingbox-bench pass@1 (507 tasks x20 trials, 5 stateful business domains) arxiv-2608.19741 | Thinkingbox-bench 507 tasks, N=20 trials, own MCP sandbox, simulated user GPT-5.4-mini, Azure-hosted API (local vLLM for Qwen), published 2026-08-19; cost = campaign total USD over 507x20 trials; read 2026-10-20 ✓ | xmodel | Opus 5·medium · Sonnet 4.6·medium |
| ThreadKeeper: summarise a 128k-token fake chat (facts, threads, inventions), composite v2 github-threadkeeper | neko-legends/threadkeeper results/leaderboard.csv; 128k corpus seed 12801; Claude Code 2.1.289; judge venice/gemini-3-8-flash (neutral family) for every arm; composite = 100*(0.45 facts + 0.30 threads + 0.25 invention_score); subscription, no cost; read 2026-10-15 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Opus 5.5·max |
| Tier-2 engineering solve items (of 25), Claude Code CLI no tools arxiv-2610.06668 | 25 multi-regime engineering solve problems, text only, Claude Code CLI replacing the system prompt, effort high where exposed; Supp. Table 3 of arXiv 2610.06668; read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Opus 5.5·high · Sonnet 5.5·high · Opus 5·high · Sonnet 5·high · Opus 4.8·high · Haiku 4.5·solo |
| tilth MCP arm github-tilth | tilth README @cff4dda · MCP arm, 3 models; author retracted 19 Sep (harness defects) ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| tilth-baseline github-tilth | config ✓ | xmodel | Sonnet 4.6·default · Haiku 4.5·solo |
| Timeline-Bench (video editing), all models in OpenCode arxiv-2609.35143 | Timeline-Bench (arXiv 2609.35143, 28 Sep 2026): 56 video-editing tasks, 1 run per task, harness OpenCode 1.18.31 for all ten models; score = resolution rate (Res. %, headline of Table 1; Human win-or-tie % and Tests are other summaries of the same runs); cost = mean model cost per run (Table 7, recorded by OpenCode); reasoning at the highest effort the provider offers (Table 9); read 2026-10-20 ✓ | xmodel | Fable 5.1·max · Opus 5·max |
| Timeline-Bench (video editing), vendor-native harnesses (Codex CLI / Claude Code) arxiv-2609.35143 | Timeline-Bench: same 56 tasks, 1 run per task; harness = developer's own (GPT in Codex CLI 0.155.1, Claude in Claude Code 2.1.278); score = Res. %; cost = Table 7 mean model cost per run (Claude Code recorded; Codex CLI estimated by the authors from tokens at base-rate prices); curated-guidance and computer-use Astra variants not rows (different setups); read 2026-10-20 ✓ | xmodel | Opus 5·max · Fable 5.1·max |
| Tinybird SQL benchmark tinybird.co | Tinybird LLM SQL benchmark, 50 natural-language questions answered in SQL over 200M GH Archive rows, via OpenRouter · composite score (efficiency, exactness); no effort set (default); no cost; read 2026-10-01 ✓ | xmodel | Opus 4.7·default · Fable 5·default · Fable 5.1·default · Opus 5.5·default · Sonnet 5.5·default · Opus 4.8·default · Opus 5·default · Sonnet 5·default · Sonnet 4.6·default · Haiku 4.5·solo · Haiku 5.5·default |
| TLA-Verus tlaverus_algo 40 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 12: tlaverus_algo 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| TLA-Verus tlaverus_llm 40 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 13: tlaverus_llm 40 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| TLA-Verus verusbench 150 tasks: p@3 tla-verus-2610.14923 | TLA-Verus Table 11: verusbench 150 tasks, Verus proof generation with verifier feedback (<=10 attempts), 3 samples per task, temperature 1.0, all via OpenRouter, completion limit 32,768 (verusbench) / 128,000 (tlaverus); p@3 = tasks solved by >=1 of 3 samples (p@1, turns, tokens = same runs); cost = provider billed figure summed over the whole run; arXiv 2610.14923 (Oct 2026); read 2026-10-20 ✓ | xmodel | Opus 5·default · Sonnet 5·default |
| Token Value Inequality Table 2: average accuracy on AIME25/MATH-500/AMC/OlympiadBench tokenvalue-2609.33970 | Token Value Inequality (Zeng et al.) Table 2 baseline rows: 4 in-domain math benchmarks (AIME 30 problems, MATH, AMC, OlympiadBench), avg accuracy %, avg #tokens per response; evaluation cap stated as 8,192 tokens for efficient/own models but not for API baselines (several baselines show #Tk ~2,039-2,048 on AIME, Grok 9,361: cap unclear); GLM-358B-32B = GLM-4.7 and DeepSeek-671B-37B = DeepSeek-V3.2 per Table S1; arXiv 2609.33970; read 2026-10-20 ✓ | xmodel | Sonnet 4.6·default |
| token-marxing heavy autoresearch task (toy CPU LM training, best val_bpb) token-marxing | yiheinchai/token-marxing, commit 3c895b32c89a7e9c0edb1c87e1603f8a50254251, labour=heavy, 'solo' arms only (cheap/cheap_h55/solo: one model does the whole run), effort=high (quoted from each run's summary.json) for all three arms; read 2026-10-08 ✓ | xmodel | Haiku 4.5·req-high · Haiku 5.5·high · Opus 5.5·high |
| Tool-Decathlon (GLM-5.2 card) zai-hf-glm-5.2 | Tool-Decathlon official evaluation service, max_token 128K; GLM-5.2 card; read 2026-10-19 ✓ | xmodel | Opus 4.8·max |
| Toolathlon (table) openai-gpt56-launch | Toolathlon; OpenAI GPT-5.6 launch post main tables, snapshot Wayback 20260709172124; effort not stated next to the tables; published 2026-07-09; read 2026-10-19 ✓ | xmodel | Fable 5·default · Opus 4.8·default |
| Toolathlon previous leaderboard (pre-Verified snapshot, 51 models) toolathlon | Toolathlon archived 'Previous Toolathlon leaderboard' (snapshot before Toolathlon-Verified, 51 models; task definitions differ so not comparable with the current table), Pass@1 %; badge = evaluated by the Toolathlon team, no badge = copied from a vendor post; fetched 2026-10-20; cells identical to held cells dropped; read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Sonnet 4.6·default · Haiku 4.5·solo |
| Toolathlon Verified pass@1 (GLM-5.3 card) zai-hf-glm-5.3 | Toolathlon Verified via official evaluation service, pass@1 averaged over 3 runs; GLM-5.3 card; read 2026-10-19 ✓ | xmodel | Fable 5·default |
| Toolathlon-Verified anthropic-syscard | Sonnet 5.5 card table 8.14.5.A · 7 Claude models at max, Pass@1, score-only ✓ | xmodel | Sonnet 5.5·max · Opus 5.5·max · Fable 5.1·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max · Fable 5·max |
| Toolathlon-Verified leaderboard (current, 108 tasks) toolathlon | Toolathlon-Verified (released 30 Jun 2026), 108 tasks, 32 MCP servers / 604 tools + 7 local toolkits, Pass@1 % (+-CI) over 3 runs; rows carry the 'Evaluated by us' badge unless noted; Agent 'Default'; page fetched 2026-10-20 (newest row 2026-08-30); cells identical to held cells (vendor cards/blogs copying the board) dropped; read 2026-10-20 ✓ | xmodel | Sonnet 5·max |
| Towards AI internal writing-Elo board, Sep 22, 2026 reddit-1wr03jc-img1 | 10 script tasks x 5 scripts, scored blind out of 100 by three AI judges from three different labs; top-16 leaderboard bar chart, footer 'Board of Sep 22, 2026'; values are printed bar labels, read verbatim, not pixel-digitised; read 2026-10-05 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max |
| Trap-task failure rate (lower better), 12 hard + 4 advanced tasks x5 runs github-coleam00-kimi-k3-reliability | Pi SDK + OpenRouter harness, same for every model, graded by reading transcripts; README table failure rate (lower is better); read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| TrialPD-Bench, Eligibility criteria extraction F1 arxiv-2610.18507 | TrialPD-Bench (arXiv 2610.18507, submitted 13 Oct 2026): coding agents under Harbor digitise clinical-trial protocol PDFs (task: Eligibility criteria extraction F1); each model in its own vendor harness (Codex / Claude Code / Kimi Code / DeepSeek Harness / pi); score = first-listed final-answer metric of the task block of Table 2 (others, evidence recall, are further columns of the same runs); cost = Table 5 average cost per run (USD); effort not stated in the paper; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default |
| TrialPD-Bench, Indication extraction F1 arxiv-2610.18507 | TrialPD-Bench (arXiv 2610.18507, submitted 13 Oct 2026): coding agents under Harbor digitise clinical-trial protocol PDFs (task: Indication extraction F1); each model in its own vendor harness (Codex / Claude Code / Kimi Code / DeepSeek Harness / pi); score = first-listed final-answer metric of the task block of Table 2 (others, evidence recall, are further columns of the same runs); cost = Table 5 average cost per run (USD); effort not stated in the paper; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default |
| TrialPD-Bench, Objectives & endpoints: objective F1 arxiv-2610.18507 | TrialPD-Bench (arXiv 2610.18507, submitted 13 Oct 2026): coding agents under Harbor digitise clinical-trial protocol PDFs (task: Objectives & endpoints: objective F1); each model in its own vendor harness (Codex / Claude Code / Kimi Code / DeepSeek Harness / pi); score = first-listed final-answer metric of the task block of Table 2 (others, evidence recall, are further columns of the same runs); cost = Table 5 average cost per run (USD); effort not stated in the paper; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default |
| TrialPD-Bench, SoA tables: Activity-Visit schedule F1 arxiv-2610.18507 | TrialPD-Bench (arXiv 2610.18507, submitted 13 Oct 2026): coding agents under Harbor digitise clinical-trial protocol PDFs (task: SoA tables: Activity-Visit schedule F1); each model in its own vendor harness (Codex / Claude Code / Kimi Code / DeepSeek Harness / pi); score = first-listed final-answer metric of the task block of Table 2 (others, evidence recall, are further columns of the same runs); cost = Table 5 average cost per run (USD); effort not stated in the paper; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default |
| TrialPD-Bench, Structured timeline: epoch accuracy arxiv-2610.18507 | TrialPD-Bench (arXiv 2610.18507, submitted 13 Oct 2026): coding agents under Harbor digitise clinical-trial protocol PDFs (task: Structured timeline: epoch accuracy); each model in its own vendor harness (Codex / Claude Code / Kimi Code / DeepSeek Harness / pi); score = first-listed final-answer metric of the task block of Table 2 (others, evidence recall, are further columns of the same runs); cost = Table 5 average cost per run (USD); effort not stated in the paper; read 2026-10-20 ✓ | xmodel | Opus 5·default · Opus 4.8·default · Sonnet 5·default |
| TrueFoundry truefoundry | AI Gateway · single-turn no tools, effort NS ✓ | xmodel | Opus 4.8·nothink · Opus 4.7·nothink |
| tuabench-cc arxiv-2606.28480 | config ✓ | xmodel | Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Opus 4.8·max · Opus 4.7·nothink |
| tuabench-msa arxiv-2606.28480 | config ✓ | xmodel | Opus 4.8·max |
| tuabench-oh arxiv-2606.28480 | config ✓ | xmodel | Opus 4.8·max |
| tuabench-t2 arxiv-2606.28480 | config ✓ | xmodel | Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·xhigh · Opus 4.7·max · Sonnet 4.6·max · Opus 4.8·max · Opus 4.7·nothink · Haiku 4.5·nothink |
| tw-support-agent-evals: 40 Traditional-Chinese support cases, Claude Code CLI github-tw-support-evals | hsuanmingli/tw-support-agent-evals reports/sample-report.md (2026-10-13); backend claude -p (subscription); pass = handoff rule + string rules + LLM judge claude-haiku-5-5; score = pass%; read 2026-10-15 ✓ | xmodel | Sonnet 5.5·default · Haiku 5.5·default |
| tycho-arc3 arxiv-2607.28287 | config ✓ | xmodel | Opus 4.8·xhigh · Opus 5·xhigh |
| TypeScript API-migration task, base documents + pointed instruction, Claude Code 2.1.286 / Harbor 0.22.0 zenodo-validate-benchmark | zenodo.org/records/23075022 (PDF, extracted via pypdf, numbers printed in the PDF table) Table 3, documents=base / instruction=pointed row; 3 attempts/cell; no effort stated -> default (Haiku 4.5 -> solo); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Haiku 4.5·solo |
| TypeScript API-migration task, hard documents + neutral instruction, Claude Code 2.1.286 / Harbor 0.22.0 zenodo-validate-benchmark | zenodo.org/records/23075022 (PDF, extracted via pypdf, numbers printed in the PDF table) Table 3, documents=hard / instruction=neutral row; 3 attempts/cell; no effort stated -> default (Haiku 4.5 -> solo); read 2026-10-05 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default · Haiku 4.5·solo |
| UGI Leaderboard — UGI composite (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, composite UGI score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high |
| UGI Leaderboard — World Model factual recall (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, recipe/geoguessr/weight/music percent-error battery; test dates vary by model; read 2026-10-05 ✓ | xmodel | Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high |
| UGI Leaderboard — Writing quality (all Claude models × efforts) ugi-leaderboard | DontPlanToEnd/UGI-Leaderboard, fixed prompt set, LLM-judged Writing score; test dates vary by model (CSV Test Date column); read 2026-10-05 ✓ | xmodel | Sonnet 4.6·low · Sonnet 4.6·medium · Sonnet 4.6·high · Sonnet 4.6·max · Opus 4.7·low · Opus 4.7·medium · Opus 4.7·high · Opus 4.7·max · Opus 4.8·low · Opus 4.8·medium · Opus 4.8·high · Opus 4.8·xhigh · Opus 4.8·max · Fable 5·low · Fable 5·medium · Fable 5·high · Fable 5·xhigh · Fable 5·max · Sonnet 5·low · Sonnet 5·medium · Sonnet 5·high · Sonnet 5·xhigh · Sonnet 5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high |
| UI-Tau Bench Lite, genuiharness harness, Avg@3 state-EM arxiv-2610.11123 | 300 tasks, three sessions, temperature 0.7, genuiharness harness; Table 2 of arXiv 2610.11123; read 2026-10-15 ✓ | xmodel | Sonnet 4.6·default · Opus 5·default |
| UI-Tau Bench Lite, miniswe harness, Avg@3 state-EM arxiv-2610.11123 | 300 tasks, three sessions, temperature 0.7, miniswe harness; Table 2 of arXiv 2610.11123; read 2026-10-15 ✓ | xmodel | Sonnet 4.6·default · Opus 5·default |
| UI-Tau Bench Lite, pi harness, Avg@3 state-EM arxiv-2610.11123 | 300 tasks, three sessions, temperature 0.7, pi harness; Table 2 of arXiv 2610.11123; read 2026-10-15 ✓ | xmodel | Sonnet 4.6·default · Opus 5·default |
| UI-Tau Bench Lite, smolagents harness, Avg@3 state-EM arxiv-2610.11123 | 300 tasks, three sessions, temperature 0.7, smolagents harness; Table 2 of arXiv 2610.11123; read 2026-10-15 ✓ | xmodel | Sonnet 4.6·default · Opus 5·default |
| UK AISI advanced cyber tasks, Expert tier (pass rate, 50M token budget) uk-aisi | UK AISI advanced cyber suite, Expert-level tasks (21 Expert + 27 Practitioner in Figure 1), 50M token budget, basic ReAct agent with Bash+Python in Kali container; average pass rate (+-1 SEM); blog 2026-04-30; read 2026-10-20 ✓ | xmodel | Opus 4.7·default |
| Vals AI Finance Agent v2 vals.ai | vals.ai/benchmarks/vals_index page payload, tasks.finance_agent block, compute_effort=max for all rows present; read 2026-10-08; read 2026-10-08 ✓ | xmodel | Haiku 5.5·max · Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Fable 5·max · Opus 5·max · Sonnet 5·max · Opus 4.8·max |
| Vals AIME vals.ai | Vals AI · AIME, compute_effort null (effort unverified), board 2026-04-16 ✓ | xmodel | Haiku 4.5·solo · Opus 4.7·default · Sonnet 4.6·default |
| Vals BioMystery vals.ai | Vals AI · BioMysteryBench (Terminus), Sonnet 5.5, Opus 5.5 and Opus 5 at max — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Sonnet 5.5·max · Haiku 5.5·max |
| Vals CaseLaw v2 vals.ai | Vals AI · CaseLaw v2, compute_effort null (effort unverified), board 2026-05-04 ✓ | xmodel | Haiku 4.5·solo · Opus 4.7·default · Sonnet 4.6·default |
| Vals Code Migration vals.ai | Vals AI · Code Migration, 10 Claude entries, compute_effort max (Haiku thinking null) — snapshot 27 Sep, re-priced ✓ | xmodel | Fable 5·max · Fable 5.1·max · Haiku 4.5·solo · Opus 4.7·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max · Sonnet 4.6·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max |
| Vals CorpFin v2 vals.ai | Vals AI · CorpFin v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Haiku 4.5·solo · Opus 4.7·max · Opus 5·default |
| Vals CUA-bench vals.ai | Vals AI · CUA-bench via Claude Code, Opus 5.5, Opus 5 and Fable 5.1 at max — snapshot 27 Sep, re-priced (board 22 Sep) ✓ | xmodel | Fable 5.1·max · Opus 5·max · Opus 5.5·max |
| Vals CyberBench vals.ai | Vals AI · OSS-Fuzz PoC+patch, mini-swe-agent, 5 Claude models at max — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5·max · Fable 5.1·max · Sonnet 5·max · Sonnet 5.5·max · Opus 5.5·max · Haiku 5.5·max |
| Vals Finance Agent v2 vals.ai | Vals AI · Finance Agent v2, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·high · Sonnet 5·max · Sonnet 4.6·max · Haiku 4.5·solo · Sonnet 5.5·max |
| Vals GPQA vals.ai | Vals AI · GPQA, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo · Opus 5·default |
| Vals HLAB vals.ai | Vals AI · Harvey's Legal Agent Benchmark, all at max, several at floor — snapshot 27 Sep, re-priced ✓ | xmodel | Fable 5·max · Fable 5.1·max · Haiku 4.5·solo · Opus 4.7·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max · Sonnet 4.6·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max |
| Vals Index vals.ai | Vals AI · 24-bench composite v1.2 (archived), 7 entries, Opus 4.7 high; overlaps EMB+VibeCode — snapshot 27 Sep, re-priced (archive re-read 29 Sep) ✓ | xmodel | Fable 5·max · Opus 5·max · Opus 4.8·max · Sonnet 5·max · Opus 4.7·high · Sonnet 4.6·max · Haiku 4.5·solo |
| Vals Index (29 Sep) vals.ai | vals.ai Vals Index as recomputed on 29 Sep 2026 (older models dropped), max effort, accuracy and cost per test ✓ | xmodel | Fable 5.1·max · Opus 5·max · Fable 5·max · Sonnet 5·max · Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Opus 4.8·max |
| Vals Index 09/26 vals.ai | Vals AI · composite v2 snapshot of 27 Sep: the four older entries (Opus 4.8, Opus 4.7 high, Sonnet 4.6, Haiku 4.5) that the 29 Sep recomputation dropped; the current models are in valsindex3 ✓ | xmodel | Opus 4.8·max · Haiku 4.5·solo · Opus 4.7·high · Sonnet 4.6·max |
| Vals IOI vals.ai | Vals AI · IOI, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max |
| Vals Legal Research vals.ai | Vals AI · Legal Research, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 4.6·max · Haiku 4.5·solo · Sonnet 5.5·max · Haiku 5.5·max |
| Vals LegalBench vals.ai | Vals AI · LegalBench, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo |
| Vals LiveCodeBench vals.ai | Vals AI · LiveCodeBench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo |
| Vals MedCode vals.ai | Vals AI · ICD-10-CM coding, 2,755 records, all at max ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Haiku 4.5·solo · Sonnet 5.5·max · Haiku 5.5·max |
| Vals MedQA vals.ai | Vals AI · MedQA, Sonnet 4.6 vs Haiku 4.5 thinking, effort null, board 2026-04-16 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default |
| Vals MedScribe vals.ai | Vals AI · MedScribe, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Haiku 4.5·solo · Sonnet 5.5·max · Haiku 5.5·max |
| Vals MMLU-Pro vals.ai | Vals AI · MMLU-Pro, Claude models at their stated compute_effort, measured cost/test, near-ceiling → cost only ✓ | xmodel | Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo |
| Vals MMMU vals.ai | Vals AI · MMMU, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo |
| Vals MortgageTax vals.ai | Vals AI · MortgageTax, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo · Opus 5·default |
| Vals Multimodal Index vals.ai | Vals AI · Multimodal Index, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·high · Haiku 4.5·solo · Opus 5·default |
| Vals MysteryMechanism vals.ai | Vals AI · MysteryMechanism, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5.5·max · Haiku 5.5·max |
| Vals ProgramBench vals.ai | Vals AI · ProgramBench, max, mostly floor scores, 3 entries without cost — snapshot 27 Sep, re-priced ✓ | xmodel | Fable 5·max · Fable 5.1·max · Haiku 4.5·solo · Opus 4.7·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max · Sonnet 4.6·max · Sonnet 5·max · Haiku 5.5·max · Sonnet 5.5·max |
| Vals ProofBench vals.ai | Vals AI · ProofBench v1.1, all at max (Opus 5 now max in payload), 4 saturated — snapshot 28 Sep, re-priced ✓ | xmodel | Fable 5·max · Fable 5.1·max · Opus 5·max · Opus 5.5·max · Sonnet 5·max · Sonnet 5.5·max · Haiku 5.5·max |
| Vals Public Benefits v1 vals.ai | Vals AI · Public Benefits Bench v1 (board 2026-06-09), max + Haiku default ✓ | xmodel | Fable 5·max · Haiku 4.5·solo · Opus 4.8·max · Sonnet 4.6·max |
| Vals Public Benefits v1.1 vals.ai | Vals AI · Public Benefits v1.1, Claude models at their stated compute_effort, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Haiku 4.5·solo · Sonnet 5.5·max · Haiku 5.5·max |
| Vals RSI Index vals.ai | Vals AI · RSI Index v1.1 (autonomous LLM R&D, 4 campaigns), Claude models at max, cost per suite = sum of 4 — snapshot 27 Sep, re-priced (board 25 Sep) ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max |
| Vals SAGE vals.ai | Vals AI · grading student math work against a rubric, all at max ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Fable 5·max · Opus 4.8·max · Opus 4.7·max · Sonnet 5·max · Sonnet 4.6·max · Haiku 4.5·solo · Sonnet 5.5·max · Haiku 5.5·max |
| Vals Science Index (biology data investigation, mechanism discovery, research workflows) vals.ai | vals.ai Science Index, equal-weighted BioMysteryBench / MysteryMechanism / Terminal-Bench-Science; cost per test; read 2026-10-18 ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Haiku 5.5·max · Opus 5·max |
| Vals SkillsBench vals.ai | Vals AI · SkillsBench with skills, OpenHands, 5 Claude models at max — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5·max · Fable 5.1·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max |
| Vals SRE-Bench vals.ai | Vals AI · binary reverse engineering (despite the name), all at max — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5.5·max · Haiku 5.5·max · Sonnet 5·default |
| Vals SWE-bench vals.ai | Vals AI · SWE-bench, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo · Opus 5·default |
| Vals SWE-bench Claude Code vals.ai | Vals AI · SWE-bench, Opus 4.8 in Claude Code harness (single row, pairs with valsswebench) ✓ | xmodel | Opus 4.8·default |
| Vals Tax Agent vals.ai | Vals AI · Tax Agent, 10 Claude entries at their stated compute_effort — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Opus 5·max · Fable 5.1·max · Sonnet 5·max · Sonnet 5.5·max · Fable 5·max · Opus 4.8·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo · Haiku 5.5·max |
| Vals TaxEval v2 vals.ai | Vals AI · TaxEval v2, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Fable 5.1·max · Fable 5·max · Opus 4.8·max · Sonnet 5·max · Sonnet 4.6·max · Opus 4.7·max · Haiku 4.5·solo · Opus 5·default |
| Vals TB 2.1 vals.ai | Vals AI · Terminal-Bench 2.1, Claude models at their stated compute_effort (Sonnet 5.5, Opus 5.5 high) — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5·high · Sonnet 5·max · Opus 4.8·max · Opus 4.7·high · Sonnet 4.6·max · Opus 5.5·high · Fable 5.1·high · Fable 5·max · Haiku 4.5·solo · Sonnet 5.5·high |
| Vals TB 4.0 vals.ai | Vals AI · Terminal-Bench 4.0, all Claude at max, measured cost/test — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Haiku 5.5·max · Opus 4.8·max · Sonnet 5·max |
| Vals TB-Science vals.ai | Vals AI · Terminal-Bench-Science, all at max — snapshot 27 Sep, re-priced ✓ | xmodel | Opus 5.5·max · Sonnet 5.5·max · Fable 5.1·max · Opus 5·max · Fable 5·max · Haiku 5.5·max · Sonnet 5·max · Opus 4.8·max |
| Vals Terminal-Bench 2.0 vals.ai | Vals AI · Terminal-Bench 2.0 (older board, 2026-06-04), stated compute_effort ✓ | xmodel | Haiku 4.5·solo · Opus 4.7·high · Opus 4.8·max · Sonnet 4.6·max |
| Vals Time Horizon KSP vals.ai | Vals AI · Time Horizon Index: KSP, 3 Claude at max, board 2026-09-14 ✓ | xmodel | Fable 5.1·max · Opus 4.8·max · Opus 5·max · Opus 5.5·max |
| Vals Vibe Code Bench 1-100 vals.ai | Vals AI · Vibe Code Bench 1-100, Claude models at their stated compute_effort, measured cost/test ✓ | xmodel | Opus 5.5·xhigh · Opus 5·max · Fable 5.1·max · Sonnet 5·max · Haiku 5.5·max |
| Vals VibeCode Claude Code vals.ai | Vals AI · Vibe Code Bench, Opus 4.8 and Sonnet 4.6 in Claude Code harness ✓ | xmodel | Opus 4.8·default · Sonnet 4.6·default |
| vals.ai Robo-bench (robot arm + humanoid control in a physics sim, most tasks held out) vals.ai | claude-code harness; overall score; cost = vals 'Est. API cost' per test (USD, estimate); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max |
| vals.ai SWE-bench Multimodal (JS front-end issues with screenshots) vals.ai | Vals-run agent; resolve rate overall, cost per test (USD); effort max per vals model pages; page updated 2026-10-13; read 2026-10-15 ✓ | xmodel | Opus 5.5·max · Fable 5.1·max · Sonnet 5.5·max · Haiku 5.5·max |
| valstb21cc vals.ai | config ✓ | xmodel | Opus 4.8·default |
| valswsfinance-exa vals.ai | config ✓ | xmodel | Fable 5·max |
| valswslegal-exa vals.ai | config ✓ | xmodel | Fable 5·max |
| Vectara HHEM-2.3 summarisation faithfulness (own 7700-article set) vectara-hhem | Vectara hallucination leaderboard README 'Last updated on October 16, 2026'; each LLM summarises every document with the stated prompt (summary <=20% of length); judge HHEM-2.3; answer rate and avg length given; no tools; score = Factual Consistency Rate (100 - hallucination rate); live snapshot; read 2026-10-19 ✓ | xmodel | Haiku 4.5·solo · Sonnet 4.6·default · Opus 4.7·default |
| Vectorise doc-search, judge-graded (Vidali gist) vidali-gist | citation recall by cross-model judge, 22 Sep 2026 evening run ✓ | xmodel | Opus 5·default · Sonnet 5·default · Haiku 4.5·solo |
| Vending-Bench 2 (Andon Labs) andonlabs-vb2 | andonlabs.com/evals/vending-bench-2: mean final money balance after a simulated year (start $500), mean of runs; 3 rows read from the page, the rest relayed by Epoch's mirror; no effort stated by the page; no cost; read 2026-10-15 ✓ | xmodel | Opus 5·default · Sonnet 5.5·default · Opus 4.7·default · Opus 5.5·default · Sonnet 4.6·default · Sonnet 5·default · Opus 4.8·high · Fable 5·high · Fable 5.1·default · Fable 5·low · Fable 5·max · Fable 5·nothink · Fable 5·medium · Haiku 5.5·default · Opus 4.8·max · Haiku 4.5·solo |
| vercel/eve agent evals (baseline), case set 336fa6e2 github-vercel-eve | vercel/eve 7 cases x 3 runs, Claude Code, baseline; cost = list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Opus 5.5·high · Sonnet 5·default |
| vercel/eve agent evals (guided), case set 336fa6e2 github-vercel-eve | vercel/eve 7 cases x 3 runs, Claude Code, guided; cost = list price; read 2026-10-15 ✓ | xmodel | Fable 5.1·high · Opus 5.5·high · Sonnet 5·default |
| vercel/eve CI benchmark, eve-code harness (swe-lean@v3) github-vercel-eve | CI bot 'eve benchmark' PR comments, swe-lean@v3 (7 tasks x 5 attempts = 35 trials), harness=eve-code; weak: harness/eve-code commit and day differ per PR, effort not stated (default); read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default |
| vercel/eve CI benchmark, opencode harness (swe-lean@v3) github-vercel-eve | CI bot 'eve benchmark' PR comments, swe-lean@v3 (7 tasks x 5 attempts = 35 trials), harness=opencode; weak: harness/eve-code commit and day differ per PR, effort not stated (default); read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default |
| vercel/eve CI benchmark, pi harness (swe-lean@v3) github-vercel-eve | CI bot 'eve benchmark' PR comments, swe-lean@v3 (7 tasks x 5 attempts = 35 trials), harness=pi; weak: harness/eve-code commit and day differ per PR, effort not stated (default); read 2026-10-01 ✓ | xmodel | Opus 5.5·default · Sonnet 5.5·default |
| VeriContest-Lean (theorems proved of 10; cost per proved theorem) arxiv-2610.03994 | 10 theorems x 5 Claude models x 5 effort settings, one attempt each, Claude Code, 10 iterations/3 h; Table 4-5 of arXiv 2610.03994; read 2026-10-15 ✓ | xmodel | Fable 5.1·low · Fable 5.1·medium · Fable 5.1·high · Fable 5.1·xhigh · Fable 5.1·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Haiku 4.5·solo |
| VEX-Bench arxiv-2609.08040 | Claude Code 2.1.150 · high, Opus 4.6 vs Sonnet 4.6, cost+tokens per case ✓ | xmodel | Sonnet 4.6·high |
| vibe-openscad iter-png-1 github-vibeopenscad | vibe-openscad · 7 tasks, 1 PNG-feedback round, 5 Claude models ✓ | xmodel | Fable 5·default · Haiku 4.5·solo · Opus 4.7·default · Opus 4.8·default · Sonnet 4.6·default |
| vibe-openscad iter-png-2 github-vibeopenscad | vibe-openscad · 7 tasks, 2 PNG rounds ✓ | xmodel | Fable 5·default · Haiku 4.5·solo · Opus 4.7·default · Opus 4.8·default · Sonnet 4.6·default |
| vibe-openscad iter-png-3 github-vibeopenscad | vibe-openscad · 7 tasks, 3 PNG rounds ✓ | xmodel | Fable 5·default · Haiku 4.5·solo · Opus 4.7·default · Opus 4.8·default · Sonnet 4.6·default |
| vibe-openscad PDF page github-vibeopenscad | vibe-openscad · 2 tier-4 datasheet tasks, 8 Claude models default ✓ | xmodel | Fable 5·default · Fable 5.1·default · Haiku 4.5·solo · Opus 4.7·default · Opus 4.8·default · Opus 5·default · Sonnet 4.6·default · Sonnet 5·default |
| Video2World (222 instances), coding agents arxiv-2610.04432 | Video2World (arXiv 2610.04432, 3 Oct 2026): 222 instances (189 clips, 39 task families), agent builds a simulation package from video; score = V2WScore (0-100, first column of main table; build rate, functionality success, progress, geometry errors are other columns); seven models in OpenCode (via OpenRouter), GPT-6 Astra in Codex CLI, Fable-5.1 in Claude Code; cost = Table A17 mean API cost per episode (USD); read 2026-10-20 ✓ | xmodel | Fable 5.1·max · Opus 5·default |
| VideoGAIA pass@1 overall, ReAct agent loop arxiv-2608.14718 | VideoGAIA, ReAct agent loop (web search, page visit, thinking-with-videos; 20 coarse frames, <=40 steps); paper 2026-08-12; read 2026-10-20 ✓ | xmodel | Opus 4.7·default · Opus 4.8·default |
| vlm-exam counting github-vlmexam | roboflow/vlm-exam · 74 samples, LLM-judge accuracy ✓ | xmodel | Fable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high |
| vlm-exam detection github-vlmexam | roboflow/vlm-exam · 250 images, mAP@50 ✓ | xmodel | Fable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high |
| vlm-exam extraction github-vlmexam | roboflow/vlm-exam · 97 samples, LLM-judge accuracy; same models as OCR ✓ | xmodel | Fable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high |
| vlm-exam identification github-vlmexam | roboflow/vlm-exam · 32 samples, near ceiling ✓ | xmodel | Fable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high |
| vlm-exam OCR github-vlmexam | roboflow/vlm-exam · 37 samples, similarity; Fable 5.1 & Opus 5.5 low/high ×3 runs, others low ×1 ✓ | xmodel | Fable 5·low · Fable 5.1·low · Fable 5.1·high · Sonnet 5·low · Opus 4.8·low · Opus 5·low · Opus 5.5·low · Opus 5.5·high · Sonnet 5.5·low · Sonnet 5.5·high · Haiku 5.5·low · Haiku 5.5·high |
| vlnr2r arxiv-2607.26148 | config ✓ | xmodel | Fable 5·max · Opus 5·max · Fable 5·default · Opus 5·default · Opus 4.8·default · Opus 4.8·max · Sonnet 5·default · Sonnet 5·max |
| vulcanbench3 github-vulcanbench | config ✓ | xmodel | Fable 5·default · Haiku 4.5·solo |
| VWE-Bench (VibeWorlding, arXiv 2608.15265) overall Pass@1 arxiv-2608.15265 | VWE-Bench test split (254 queries: 46 construction + 208 refinement), frontier models run as agents in the authors' sandbox with geometric + MLLM verifier; Overall = mean reward over all query types, Pass@1 %; no cost; arXiv v1 2026-08-15; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| WC2026 pick accuracy (all 104 matches) arxiv-2607.17765 | WC2026-Agents: four consumer assistants (web tools on) forecast all 104 FIFA World Cup 2026 matches (11 June - 19 July 2026): 1X2 probabilities + virtual $100 bet, prompt byte-identical; benchmark is contamination-free; not API runs; Table 2 task T1 over all 104 matches; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| WeasyPrint 8-file feature, hidden tests, 3 runs each (u/Sufficient-Storage87) reddit-sufficient-storage87 | real WeasyPrint feature touching 8 files, hidden tests re-scored in fresh container, cost repriced from transcript tokens at API list price; 3 runs/setup; read 2026-10-18 ✓ | xmodel | Sonnet 5.5·default · Opus 5.5·default |
| weather-card benchmark (2nd1st) 2026-10-18 full dataset: Anthropic API, official transport, 4 min + 4 q cards per config; fidelity match share (weak score) + tokens github-weather-card | weathercard.secondfirst.ai wcb-full-dataset-2026-10-18 configs/*--api--official*; metered API, cost_usd null in dataset; score = share of scalar card fields (condition,date,hourly,max,min,name) whose fidelity-extractor-v1 state is 'match' over 8 slots x 6 fields (computed by us from meta.json); read 2026-10-18 ✓ | xmodel | Haiku 5.5·low · Haiku 5.5·medium · Haiku 5.5·high · Haiku 5.5·xhigh · Haiku 5.5·max · Sonnet 5.5·low · Sonnet 5.5·medium · Sonnet 5.5·high · Sonnet 5.5·xhigh · Sonnet 5.5·max · Opus 5.5·low · Opus 5.5·medium · Opus 5.5·high · Opus 5.5·xhigh · Opus 5.5·max |
| WebCoder paper, Vision2Web L2-frontend (arXiv 2610.23104) arxiv-2610.23104 | Vision2Web L2-frontend 66 titles, Avg of visual and functional score; undelivered tasks count as 0; 14 models 'all evaluated under the same agent framework, tools, prompts, container environment, time budget, and judges' (App. A); 2 h per task; Table 1; no cost; arXiv v1 2026-10-16; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| WebCoder paper, Webdev-Bench (arXiv 2610.23104) arxiv-2610.23104 | Webdev-Bench 40 composite tasks (188 requirements, 1125 hidden checklist items), Avg of RS/VS/FS/IS; undelivered tasks count as 0; 14 models 'all evaluated under the same agent framework, tools, prompts, container environment, time budget, and judges' (App. A); 2 h per task; Table 1; no cost; arXiv v1 2026-10-16; read 2026-10-20 ✓ | xmodel | Opus 4.8·default |
| WeirdML v3 (agentic, token-axis area score) epoch-archive | 11 ML tasks x 15 configurations; score = token-axis area 0-1 with 95% CI; cost = mean API $/run; agent claude_code; read 2026-10-01 ✓ | xmodel | Opus 5.5·xhigh · Fable 5.1·xhigh · Sonnet 5.5·xhigh · Opus 5·xhigh · Fable 5·xhigh |
| WhiteKUMALabo F5/O4.8 T1 whitekumalabo | Claude Code 2.1.170 · bugfix, effort high, n=3 median, saturated pass ✓ | xmodel | Fable 5·high · Opus 4.8·high |
| WhiteKUMALabo F5/O4.8 T2 whitekumalabo | Claude Code 2.1.170 · intro-text, blind 15pt judge ✓ | xmodel | Fable 5·high · Opus 4.8·high |
| WhiteKUMALabo F5/O4.8 T3 whitekumalabo | Claude Code 2.1.170 · task decomposition, blind 15pt judge ✓ | xmodel | Fable 5·high · Opus 4.8·high |
| WhiteKUMALabo F5/O4.8 T4 whitekumalabo | Claude Code 2.1.170 · 1-sentence summary, cost/speed only ✓ | xmodel | Fable 5·high · Opus 4.8·high |
| WorkBench arxiv-2606.13715 | ReAct natif · temp 0, like-for-like, thinking NS ✓ | xmodel | Fable 5·default · Opus 4.8·nothink · Sonnet 4.6·nothink · Haiku 4.5·nothink |
| Worklore MCP tool-count benchmark, 100 tools (tool search off) worklore-toolcount | 18 file tasks x 2 reps = 36 runs per cell, Claude Code 2.1.285 claude -p, --tools '' ENABLE_TOOL_SEARCH=false, N=100 tools; read 2026-10-18 ✓ | xmodel | Opus 5.5·default · Haiku 5.5·default |
| Worklore MCP tool-count benchmark, 5 tools (tool search off) worklore-toolcount | 18 file tasks x 2 reps = 36 runs per cell, Claude Code 2.1.285 claude -p, --tools '' ENABLE_TOOL_SEARCH=false, N=5 tools; read 2026-10-18 ✓ | xmodel | Opus 5.5·default · Haiku 5.5·default |
| Worklore MCP tool-count benchmark, 50 tools (tool search off) worklore-toolcount | 18 file tasks x 2 reps = 36 runs per cell, Claude Code 2.1.285 claude -p, --tools '' ENABLE_TOOL_SEARCH=false, N=50 tools; read 2026-10-18 ✓ | xmodel | Opus 5.5·default · Haiku 5.5·default |
| WorldBench (Three.js voxel world, total /280) worldbench-2610.10622 | WorldBench Table 6 total of 5 tests (WC001-005); one world per model, one completion, temp 1.0, repair loop <=5 rounds; agentic judge; arXiv 2610.10622 (7 Oct 2026); per-test columns are the same runs (rule 8); read 2026-10-20 ✓ | xmodel | Fable 5.1·default |
| WorldBuild Bench worldbuild-bench | own harness · 3 game briefs, Fable 5 vs Opus 5 both at high, real API ledger ✓ | xmodel | Fable 5·high · Opus 5·high · Opus 4.8·high · Sonnet 5·high |
| XRepoTest: mean test pass rate over 5 languages arxiv-2608.25939 | XRepoTest (arXiv 2608.25939, Aug 2026) standard setting (static context) Table A.4; test pass rate TPR (%) per language Rust/Julia/Go/Ruby/PHP; the table prints no overall column: score = mean of the 5 per-language TPR computed by the verifier; read 2026-10-20 ✓ | xmodel | Haiku 4.5·solo |
| yama3133 · 400k-character log qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, 400k-character log, needle arithmetic; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Opus 5.5·medium · Opus 5·default · Sonnet 5.5·medium · Sonnet 5·default · Haiku 4.5·default |
| yama3133 · 8 closed-answer problems × 3 qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, 8 closed-answer problems × 3; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Opus 5.5·medium · Opus 5·default · Sonnet 5.5·medium · Sonnet 5·default · Haiku 4.5·default |
| yama3133 · agentic bug fix qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, agentic bug fix, 5 planted bugs, 21 tests; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Opus 5.5·medium · Opus 5·default · Sonnet 5.5·medium · Sonnet 5·default · Haiku 4.5·default |
| yama3133 · Japanese 300–400-character writing constraint qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, Japanese 300–400-character writing constraint; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Opus 5.5·medium · Opus 5·default · Sonnet 5.5·medium · Sonnet 5·default · Haiku 4.5·default |
| yama3133 · random arithmetic at high effort qiita-yama3133 | qiita 29 Sep · Claude Code -p minimal setup, random arithmetic at high effort, cost only; Sonnet 5.5 and Opus 5.5 at medium (the others at their Claude Code default), list-price cost from Claude Code ✓ | xmodel | Sonnet 5.5·high · Sonnet 5·high |
| YC-Bench collinear-ai.github.io | Collinear YC-Bench, a simulated startup run for a year from $200K · final net worth, mean of 3 seeds (k$), bankruptcies count toward the mean; no effort stated (default); no cost; read 2026-09-30 ✓ | xmodel | Sonnet 4.6·default · Opus 4.7·default · Opus 4.8·default · Fable 5·default · Sonnet 5·default · Opus 5.5·default · Sonnet 5.5·default · Opus 5·default |
| Z.ai Code Bench v1.0 at max effort (GLM-5.3-Flash docs, text only) zai-docs-glm-5.3-flash | Z.ai Code Bench v1.0 (in-house), Claude Code 2.1.207; two values quoted in the docs prose (chart with the effort ladder not available); docs.z.ai GLM-5.3-Flash page via r.jina.ai 2026-10-19; read 2026-10-19 ✓ | xmodel | Opus 4.8·default |
| zenn-inoh-clean zenn-inoh | config ✓ | xmodel | Opus 5·high · Opus 4.8·high |
| zenn-inoh-legacy zenn-inoh | config ✓ | xmodel | Opus 5·high · Opus 4.8·high |
| zenn-inoh-sweep zenn-inoh | config ✓ | xmodel | Opus 5·low · Opus 5·medium · Opus 5·high · Opus 5·xhigh |
| τ^τ-Bench arxiv-2609.04611 | arXiv 2609.04611 · 53 agent-construction tasks in Claude Code, Opus 5 vs Sonnet 5 both at max, build spend at list ✓ | xmodel | Opus 5·max · Sonnet 5·max |
| 純鈴AIエージェンシー (YouTube) Blender LLM bench: 12 tasks x 3 runs youtube-junrei-ai | Blender python script graded automatically; 12 tasks levels 1-5 x 3 runs; output cap 128K (after raising from 32K); Haiku via Kilo route; read 2026-10-15 ✓ | xmodel | Haiku 4.5·solo · Haiku 5.5·default |
How the numbers are built — the method
Every value in this report is computed from the raw measurements by one procedure — nothing is hand-entered. The full methodology is in METHODOLOGY.md (from collection to the fit) and DISPLAY-METHODOLOGY.md (from the fit to the tier picks); the validation scripts are in model/validation.
In short : scores from different benchmarks do not share a scale — a benchmark near its ceiling compresses differences, a hard one inflates them — so the report never averages raw scores or score ratios. A latent-quality model estimates, for every benchmark, its own zero and its own gain, and for every (model, effort) couple one quality and one cost. Each couple stays atomic: no model×effort separability is assumed.
The procedure
- Same task, same configuration. Measurements are compared only within a group — one source, one task, one harness, one setting, one scale; a group that mixes them is split. The effort level is read from the source's configuration, never assumed; runs without a known effort stay out of the fit.
- Each metric on its natural scale. A bounded score (a percentage, a score out of N) goes through a logit, which removes floor and ceiling compression; money and points go through a log; an Elo or an unknown composite is left as is; cost goes through a log. The rule reads the metric's label, never the benchmark's identity. On the logit scale a score near 0 % or 100 % is noisier, and weighs accordingly. Every value carries its reading precision: the rounding of a printed number, and for a value read off a chart, its resolution and the chart's own error (measured on runs drawn on two charts, about 2 % on a digitised cost).
- One model for all groups. Every group gets its own offset, gain and noise, so the unit and zero of a metric never matter. A couple's latent value is shared by all groups; on top of it, a publisher's systematic effect on a couple and on a model (shared by the model's effort levels) and a couple's deviation on a task type are estimated, so a publisher with many benchmarks, or a couple tested mostly on one kind of task, cannot tilt the result; the sizes of these effects are estimated per model maker, and a maker's own charts about its own models carry an estimated home effect, left out of every value. The fit takes the other makers' couples measured in the same groups (OpenAI, Google, xAI, DeepSeek, Qwen, GLM, Kimi): they link more benchmarks; this page shows the Claude couples. Robust (Student-t) noise whose tails are estimated; early-access runs are down-weighted by an estimated factor. No couple serves as a reference in the fit: the latent values sum to zero. Fitted in Stan, published only if the standard convergence checks pass.
- Weights follow from the data. A group of k couples informs the spacing of couples in proportion to its discrimination² × (k − 2); a saturated or noisy benchmark weighs little; a publisher's weight saturates at its own systematic effect. No weight is set per source.
- Nothing counts twice. One execution printed twice — a system card reprinting an earlier card's series, the same run drawn on two charts, one run scored with two metrics — is detected and counted once: its score and its cost must both agree, within their reading precisions and a rounding made the wrong way. An index is left out for a couple whose index components are already in the data.
- Values and intervals. Quality = a couple's latent quality θ, shown as the score it predicts averaged over the benchmark panel; cost = its cost on a task of typical size, shown as a multiple of the cheapest couple. No couple serves as a reference, in the fit or on the page. Each couple carries its own 16–84 % interval, so two couples compare through their two intervals. A couple measured by a single publisher is not shown.
- From the fit to the picks. Every decision is taken on the latent scale (log cost, θ). The Pareto frontier keeps the couples nothing beats on both axes; they are the candidates of every pick. A price trend — what a given quality costs today, fitted on the latest model of each family — says how many times cheaper or dearer each couple is than the going rate. The four tiers spread their targets across the frontier and pick, around each target, the couple that brings the most quality for its cost, each tier paying for quality from a quarter of the market price (grunt work) to four times it (cutting-edge thinking); the crown goes to the couple furthest below the trend.
Key values
- Set by hand, never tuned to the data: weakly informative priors (half-Student-t(3, 0, 2.5) on every spread, Gamma(2, 0.1) on the noise tails); a logit kept half a step inside 0 and 100 %; a printed value's precision at its rounding (last digit / √12); an effect seen in a single measurement left out; a couple shown only when two publishers measured it; a reprint recognised by values that agree within their precisions, on two values at least and on all they share but one.
- Uncertainty. Every interval is 16–84 % (one standard deviation): the couple's own uncertainty.
- Published only when the fit is sound: R̂ ≤ 1.01 and at least 400 effective draws on every parameter, no divergent transition. The Monte Carlo error of each value is checked against its last displayed digit; in this fit four figures can move by one unit in that digit (the cost of Fable 5.1, Fable 5 and Opus 5.5 at max, the quality of Opus 4.8 solo), far inside their intervals.
- Display choices: the quality axis is compressed below the weakest tier target; the frontier, the price trend and the four tiers use constants listed with their reasons in DISPLAY-METHODOLOGY.md.
Techniques and software
- Bayesian hierarchical model sampled by Hamiltonian Monte Carlo with the No-U-Turn sampler (Hoffman & Gelman 2014), written in Stan and run with nutpie on both axes (CmdStan when a chain must restart from the previous fit); built and checked stage by stage following the Bayesian workflow.
- Convergence checked with the rank-normalised R̂ and effective sample sizes of Vehtari et al. 2021, computed with ArviZ.
- Intervals as quasi-standard errors (Firth & de Menezes 2004), so that any two couples compare through their own intervals.
- Scales: the empirical logit of bounded scores, with the binomial shape of its noise.
Checks. Scores held out of the fit and predicted: median error 1.96 points, against 4.78 for averaging score ratios; the 16–84 % predictive interval covers 66 % of them. On synthetic data with known qualities, pairs of couples in the wrong order: 0.1–0.8 % (0.4–2.2 % for score ratios). Without the largest publisher, the model vendor itself, costs move by 2.4 % and qualities by 0.9 % at the median. Details and scripts: METHODOLOGY.md.