DeepSWE
Coding agents on original, long-horizon software engineering tasks. The score is pass@1.
- Unit
- pass@1
- Results
- 69 results26 Tokscale models matched, 26 in the source
- Last update
- Sep 3, 2026Newest model release
- Retrieved
- Oct 21, 2026Source data through Oct 15, 2026
Conditions. One harness (mini-swe-agent) for every model, at each reasoning effort, averaged over several runs.
Attribution. DeepSWE results by Datacurve, collected by Epoch AI, ‘Capabilities & benchmarking’, epoch.ai/benchmarks (CC BY 4.0)
Results
One row per model: its best reported result. The rank is the position in this list, by score; the source's own ranking can differ.
Tokscale usage counts accounts that used the model in public Tokscale submissions, Oct 14 – Nov 12, 2026. It is shown next to the score and does not explain it.
| Rank | Model | Score (pass@1) | Conditions | Cost and tokens | Date | Tokscale usage (30 days) | Source link |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 AstraListed as gpt-6-astra (xhigh) | 74.1%Confidence interval 71.2–77.0% | XHigh effort · mini-swe-agent | $4.43 per task | Sep 3, 2026Model released | 562active users | Source for GPT-6 Astra (opens in a new tab) |
| 2 | Gemini 3.8 FlashListed as gemini-3-8-flash (high) | 73.8%Confidence interval 72.4–75.2% | High effort · mini-swe-agent | $2.36 per task | Sep 2, 2026Model released | 66active users | Source for Gemini 3.8 Flash (opens in a new tab) |
| 3 | Claude Opus 5Listed as claude-opus-5 (max) | 73.6%Confidence interval 69.8–77.5% | Max effort · mini-swe-agent | $11.84 per task | Jul 24, 2026Model released | 163active users | Source for Claude Opus 5 (opens in a new tab) |
| 4 | GPT-5.6 SolListed as gpt-5-6-sol (max) | 72.7%Confidence interval 69.8–75.5% | Max effort · mini-swe-agent | $8.39 per task | Jul 9, 2026Model released | 217active users | Source for GPT-5.6 Sol (opens in a new tab) |
| 5 | Claude Fable 5Listed as claude-fable-5 (xhigh) | 69.9%Confidence interval 66.7–73.2% | XHigh effort · mini-swe-agent | $13.41 per task | Jun 9, 2026Model released | 41active users | Source for Claude Fable 5 (opens in a new tab) |
| 6 | GPT-5.6 TerraListed as gpt-5-6-terra (max) | 69.6%Confidence interval 67.1–72.2% | Max effort · mini-swe-agent | $4.95 per task | Jul 9, 2026Model released | 119active users | Source for GPT-5.6 Terra (opens in a new tab) |
| 7 | GLM-5.3Listed as glm-5-3 (max) | 69.0%Confidence interval 65.9–72.0% | Max effort · mini-swe-agent | $3.99 per task | Aug 14, 2026Model released | 138active users | Source for GLM-5.3 (opens in a new tab) |
| 8 | Kimi K3Listed as kimi-k3 (max) | 68.5%Confidence interval 64.0–73.1% | Max effort · mini-swe-agent | $4.65 per task | Jul 16, 2026Model released | 56active users | Source for Kimi K3 (opens in a new tab) |
| 9 | Grok 4.6Listed as grok-4-6 (medium) | 67.5%Confidence interval 65.2–69.8% | Medium effort · mini-swe-agent | $3.45 per task | Aug 12, 2026Model released | 41active users | Source for Grok 4.6 (opens in a new tab) |
| 10 | GPT-5.6 LunaListed as gpt-5-6-luna (max) | 67.2%Confidence interval 63.2–71.2% | Max effort · mini-swe-agent | $3.03 per task | Jul 9, 2026Model released | 158active users | Source for GPT-5.6 Luna (opens in a new tab) |
| 11 | GPT-5.5Listed as gpt-5-5 (xhigh) | 67.0%Confidence interval 60.6–73.5% | XHigh effort · mini-swe-agent | $7.23 per task | Apr 23, 2026Model released | 30active users | Source for GPT-5.5 (opens in a new tab) |
| 12 | Gemini 3.7 FlashListed as gemini-3-7-flash (medium) | 65.5%Confidence interval 62.4–68.6% | Medium effort · mini-swe-agent | $2.03 per task | Aug 13, 2026Model released | 4active users | Source for Gemini 3.7 Flash (opens in a new tab) |
| 13 | GLM-5.3 FlashListed as glm-5-3-flash (max) | 63.4%Confidence interval 59.0–67.8% | Max effort · mini-swe-agent | $0.48 per task | Aug 20, 2026Model released | 170active users | Source for GLM-5.3 Flash (opens in a new tab) |
| 14 | Claude Opus 4.8Listed as claude-opus-4-8 (max) | 59.0%Confidence interval 57.2–60.7% | Max effort · mini-swe-agent | $13.22 per task | May 28, 2026Model released | 222active users | Source for Claude Opus 4.8 (opens in a new tab) |
| 15 | Qwen3.8 MaxListed as qwen3-8-max (xhigh) | 57.5%Confidence interval 54.8–60.1% | XHigh effort · mini-swe-agent | $3.73 per task | Aug 2, 2026Model released | 16active users | Source for Qwen3.8 Max (opens in a new tab) |
| 16 | Muse Spark 1.2Listed as muse-spark-1-2 (xhigh) | 54.9%Confidence interval 52.7–57.0% | XHigh effort · mini-swe-agent | $3.70 per task | Aug 5, 2026Model released | No Tokscale usage in the last 30 days | Source for Muse Spark 1.2 (opens in a new tab) |
| 17 | Claude Sonnet 5Listed as claude-sonnet-5 (max) | 53.8%Confidence interval 49.6–58.1% | Max effort · mini-swe-agent | $26.40 per task | Jun 30, 2026Model released | 138active users | Source for Claude Sonnet 5 (opens in a new tab) |
| 18 | Grok 4.5Listed as grok-4-5 (high) | 53.8%Confidence interval 51.5–56.0% | High effort · mini-swe-agent | $2.42 per task | Jul 8, 2026Model released | 9active users | Source for Grok 4.5 (opens in a new tab) |
| 19 | Muse Spark 1.1Listed as muse-spark-1-1 (xhigh) | 53.3%Confidence interval 50.3–56.4% | XHigh effort · mini-swe-agent | $2.36 per task | Jul 9, 2026Model released | No Tokscale usage in the last 30 days | Source for Muse Spark 1.1 (opens in a new tab) |
| 20 | GPT-5.4Listed as gpt-5-4 (xhigh) | 51.8%Confidence interval 50.3–53.3% | XHigh effort · mini-swe-agent | $5.65 per task | Mar 5, 2026Model released | 1active users | Source for GPT-5.4 (opens in a new tab) |
| 21 | Gemini 3.6 FlashListed as gemini-3-6-flash (high) | 46.7%Confidence interval 43.0–50.4% | High effort · mini-swe-agent | $4.42 per task | Jul 21, 2026Model released | 3active users | Source for Gemini 3.6 Flash (opens in a new tab) |
| 22 | GLM-5.2Listed as glm-5-2 (max) | 43.8%Confidence interval 42.1–45.5% | Max effort · mini-swe-agent | $3.92 per task | Jun 16, 2026Model released | 13active users | Source for GLM-5.2 (opens in a new tab) |
| 23 | Gemini 3.5 FlashListed as gemini-3-5-flash (medium) | 37.4%Confidence interval 35.6–39.2% | Medium effort · mini-swe-agent | $7.34 per task | May 19, 2026Model released | 3active users | Source for Gemini 3.5 Flash (opens in a new tab) |
| 24 | Kimi K2.7 CodeListed as kimi-k2-7-code | 30.5%Confidence interval 30.0–31.0% | mini-swe-agent | $2.82 per task | Jun 12, 2026Model released | 17active users | Source for Kimi K2.7 Code (opens in a new tab) |
| 25 | Claude Sonnet 4.6Listed as claude-sonnet-4-6 (high) | 29.9%Confidence interval 25.8–34.0% | High effort · mini-swe-agent | $5.52 per task | Feb 17, 2026Model released | 55active users | Source for Claude Sonnet 4.6 (opens in a new tab) |
| 26 | Gemini 3.1 Pro PreviewListed as gemini-3-1-pro-preview (high) | 11.7%Confidence interval 10.2–13.2% | High effort · mini-swe-agent | $2.14 per task | Feb 19, 2026Model released | 5active users | Source for Gemini 3.1 Pro Preview (opens in a new tab) |
Showing 26 of 26 models
Score by release date
Each dot is one model's best result on DeepSWE, placed at the model's release date. The dashed line is the best score among models released up to that date.
- Used on Tokscale in the last 30 days
- No Tokscale usage in the last 30 days
- Best score so far
GPT-6 Astra: 74.1% · released Sep 3, 2026 · XHigh effort · mini-swe-agent · 562 active users on Tokscale in the last 30 days
26 models, released Feb 17, 2026 to Sep 3, 2026. Highest: GPT-6 Astra, 74.1%, released Sep 3, 2026. Use the left and right arrow keys to read each model in release order.0%20%40%60%80%
Show data
| Model | Released | Score | Tokscale users, 30 days |
|---|---|---|---|
| GPT-6 Astra | Sep 3, 2026 | 74.1%XHigh effort · mini-swe-agent | 562 |
| Gemini 3.8 Flash | Sep 2, 2026 | 73.8%High effort · mini-swe-agent | 66 |
| GLM-5.3 Flash | Aug 20, 2026 | 63.4%Max effort · mini-swe-agent | 170 |
| GLM-5.3 | Aug 14, 2026 | 69.0%Max effort · mini-swe-agent | 138 |
| Gemini 3.7 Flash | Aug 13, 2026 | 65.5%Medium effort · mini-swe-agent | 4 |
| Grok 4.6 | Aug 12, 2026 | 67.5%Medium effort · mini-swe-agent | 41 |
| Muse Spark 1.2 | Aug 5, 2026 | 54.9%XHigh effort · mini-swe-agent | No Tokscale usage in the last 30 days |
| Qwen3.8 Max | Aug 2, 2026 | 57.5%XHigh effort · mini-swe-agent | 16 |
| Claude Opus 5 | Jul 24, 2026 | 73.6%Max effort · mini-swe-agent | 163 |
| Gemini 3.6 Flash | Jul 21, 2026 | 46.7%High effort · mini-swe-agent | 3 |
| Kimi K3 | Jul 16, 2026 | 68.5%Max effort · mini-swe-agent | 56 |
| Muse Spark 1.1 | Jul 9, 2026 | 53.3%XHigh effort · mini-swe-agent | No Tokscale usage in the last 30 days |
| GPT-5.6 Luna | Jul 9, 2026 | 67.2%Max effort · mini-swe-agent | 158 |
| GPT-5.6 Terra | Jul 9, 2026 | 69.6%Max effort · mini-swe-agent | 119 |
| GPT-5.6 Sol | Jul 9, 2026 | 72.7%Max effort · mini-swe-agent | 217 |
| Grok 4.5 | Jul 8, 2026 | 53.8%High effort · mini-swe-agent | 9 |
| Claude Sonnet 5 | Jun 30, 2026 | 53.8%Max effort · mini-swe-agent | 138 |
| GLM-5.2 | Jun 16, 2026 | 43.8%Max effort · mini-swe-agent | 13 |
| Kimi K2.7 Code | Jun 12, 2026 | 30.5%mini-swe-agent | 17 |
| Claude Fable 5 | Jun 9, 2026 | 69.9%XHigh effort · mini-swe-agent | 41 |
| Claude Opus 4.8 | May 28, 2026 | 59.0%Max effort · mini-swe-agent | 222 |
| Gemini 3.5 Flash | May 19, 2026 | 37.4%Medium effort · mini-swe-agent | 3 |
| GPT-5.5 | Apr 23, 2026 | 67.0%XHigh effort · mini-swe-agent | 30 |
| GPT-5.4 | Mar 5, 2026 | 51.8%XHigh effort · mini-swe-agent | 1 |
| Gemini 3.1 Pro Preview | Feb 19, 2026 | 11.7%High effort · mini-swe-agent | 5 |
| Claude Sonnet 4.6 | Feb 17, 2026 | 29.9%High effort · mini-swe-agent | 55 |
Method and matching
How results are run
- One harness (mini-swe-agent) for every model, at each reasoning effort, averaged over several runs.
- Results are Epoch AI's copy of the DeepSWE leaderboard and can lag the live page.
Matching to Tokscale models
- 69 of 69 results are matched to 26 Tokscale models: 68 by exact id, 1 by normalized name.
- No match is marked as reviewed by hand yet.
- Reasoning effort, thinking budget and agent are conditions of a result. They never change which model a result belongs to.
License and attribution
- CC BY 4.0 (opens in a new tab), retrieved Oct 21, 2026.
- DeepSWE results by Datacurve, collected by Epoch AI, ‘Capabilities & benchmarking’, epoch.ai/benchmarks (CC BY 4.0)
Sources and attribution
Each score keeps its own source, unit and run conditions. Tokscale does not combine benchmarks into one score, and a missing score means the model was not evaluated, not that it scored zero.