Cutoff Data
The standing benchmark
A fixed set of practical tasks, re-run against every tracked model on a schedule, scored the same way each time — so you can see capability move rather than guess from launch-day claims.
Last reading 8/7/2026, 5:21:22 PM UTC
| # | Model | Score | Median time | Avg cost / task | Runs |
|---|---|---|---|---|---|
| 1 | GPT-5 Miniopenai/gpt-5-mini | 10 / 10 | 8.11s | — | 13 |
| 2 | Gemini 3.6 Flashgoogle/gemini-3.6-flash | 10 / 10 | 6.21s | — | 13 |
| 3 | GPT-5.6 Terraopenai/gpt-5.6-terra | 9.17 / 10 | 1.40s | — | 13 |
| 4 | GPT-5.6 Lunaopenai/gpt-5.6-luna | 8.78 / 10 | 1.11s | — | 13 |
| 5 | Gemini 2.5 Flashgoogle/gemini-2.5-flash | 8.78 / 10 | 919ms | $0.00049 | 13 |
| 6 | Gemini 3.1 Flash Litegoogle/gemini-3.1-flash-lite | 8.21 / 10 | 1.34s | $0.00026 | 13 |
The tasks
- ReasoningDate arithmetic across a month boundary with a weekday constraint
- Long contextRetrieve one buried identifier from a wall of similar text
- Constrained generationWrite one sentence under three simultaneous hard constraints
- Instruction followingObey an exact output format under conflicting pressure
- PrecisionCount letter occurrences across a fixed passage
- ReasoningMulti-step unit-cost arithmetic with a single exact answer
- ExtractionPull a funding round out of messy prose into strict JSON
- Data handlingAggregate, rank and serialise a table without arithmetic slips
How this is measured
Each task has a fixed prompt and a deterministic grader — an exact answer, a required structure, or a checklist of facts that must appear. No model sees the grader, and the prompts do not change between runs, so scores are comparable across dates.
We record the billed cost and wall-clock time of every run alongside the score, because the useful question is rarely “which model is best” but “which model is good enough for this job at this price”.
This is a small, practical suite, not an academic evaluation. It is designed to catch regressions and silent model changes over time. Anything we get wrong, tell us on the corrections page and we will republish.