Alibaba launched Qwen 3.8-Max this week and marketed the preview as second solely to Claude Fable 5 (their launch-day desk was extra equivocal: the mannequin leads on certainly one of 12 coding-agent rows). However an impartial harness got here near the alternative conclusion: a benchmark run, apparently utilizing the Preview model, put Qwen 3.8-Max's finest effort setting mid-pack, and its default setting final.
Each outcomes are actual and defensible. The hole between them is about token and time budgets, and that issues as a result of these figures aren’t normally headline numbers. Alibaba's footnotes give its coding numbers a five-hour timeout, and as much as 12 hours per run on PaperBench. The impartial harness, VulcanBench, allowed between 45 and 60 minutes of wall clock time. A time funds between 5 and 16 instances bigger on Alibaba’s facet explains the massive distinction in outcomes.
It’s time to do two issues to begin accounting for these variations when selecting fashions. First, the metric to make use of is price per profitable job: whole spend, together with every little thing you spent on makes an attempt that failed, divided by the duties that really handed your acceptance verify. Second, you’ll want to make time or token budgets an specific a part of your acceptance standards, not a hidden element.
Value per token has stopped predicting the invoice
The comparability everybody revealed in Qwen 3.8-Max's first week was a value comparability, as a result of that was the one knowledge accessible. It’s not an affordable mannequin. DeepSeek-V4-Flash-0731, which entered public API beta on July 31, lists at 14 cents per million enter tokens and 28 cents output. Qwen 3.8-Max lists at $2 and $6. Kimi K3 sits at $3 and $15.
These costs inform you lower than they used to, for a purpose particular to reasoning fashions like Qwen: attending to a outcome prices considering tokens. A mannequin that spends most of its token allowance on reasoning can attain a token cap earlier than it writes the reply, supplying you with an empty outcome indistinguishable from a complete failure at the price of a full run.
Synthetic Evaluation has the cleanest revealed measurement of how this may have an effect on actual agent spend: operating its Intelligence Index on DeepSeek-V4-Flash at most effort took 210 million output tokens in opposition to a category median of 100 million. Absolute price stayed low anyway, as a result of the tokens have been so low-cost. However verbosity prices time, not simply cash, and relying in your use case that may sink you.
What you want is a quantity that counts every little thing you spent, together with the makes an attempt that got here again empty, in opposition to the duties that really obtained finished within the time and token funds you specified. That is what a cost-per-success metric helps you see.
Your failure price is partly a configuration setting
A run that produces a incorrect reply and a run that runs out of funds are totally different occasions with totally different fixes. Virtually no harness distinguishes them, and nearly no leaderboard stories the break up. I hit this constructing an agent benchmark of my very own: the harness logged a failure and nothing about why, and I had so as to add the excellence myself. Whenever you do separate them, funds exhaustion seems to dominate.
Lengthy-Horizon-Terminal-Bench, revealed in July, ran 17 frontier fashions throughout 46 duties via a shared harness with one 90-minute try every. Timeouts accounted for 79% of unresolved runs, in opposition to 19% for brokers that stopped on their very own and three% for harness errors. The authors are cautious about what that does and doesn’t imply: the timed-out runs weren’t near ending, with imply reward between 0.10 and 0.35, so you can’t assume extra time would have resulted in success. However the lesson is: benchmarks are implicitly measuring time effectivity, whether or not or not they shout about that.
The clearest revealed instance of the mechanism comes from VulcanBench, the identical open-source harness behind the Qwen chart. In a report dated July 26, Claude Opus 5's lowest-effort setting was its finest, fixing 20 of 23 duties in opposition to 18 at excessive effort. The additional reasoning wasn’t ineffective: excessive effort returned the fewest incorrect solutions of any setting, one in opposition to three. It ran out of clock as a substitute, and a timeout scores zero. Two of its three regressions have been cutoffs on duties that low effort solves, and given limitless time on each it solely ties its least expensive setting, at 3.1 instances the fee.
That has a direct consequence for anybody constructing a routing ladder. The usual design escalates to extra reasoning when an affordable try fails, on the idea that the subsequent rung is healthier and merely prices extra. For a significant share of mannequin and job mixtures that assumption is incorrect, and also you pay the upper rung's value to escalate right into a timeout or hitting a cap.
Who’s already measuring this
A number of teams have landed on price per profitable job independently in the previous few months, which is the strongest sign it's changing into normal.
VulcanBench stories {dollars} per solved job as a headline column and has since its earliest stories. Lengthy-Horizon-Terminal-Bench publishes per-task price subsequent to accuracy, and its most instructive row is GPT-5.4 at roughly $26 per job with a a lot decrease cross price than Grok 4.5 at about $11. TestEvo-Bench runs brokers beneath a price cap, and Claude Code's test-generation rating falls from 71% to 44% on the tighter cap.
Distributors are already on board with the thought of measuring per profitable job. HubSpot moved its Breeze Buyer Agent in April to 50 cents per resolved dialog, down from $1 per dealt with dialog. Zendesk payments per automated decision. Fin fees 99 cents per end result and payments solely on end-to-end decision.
What to alter this week
Emit a failure purpose on each agent run as a required area, with funds exhaustion, verifier failure and harness error as distinct values relatively than one failure flag. Till you possibly can separate a timeout from a incorrect reply, your cross price is measuring two issues directly and you can’t inform which one to repair.
Compute price per profitable job per effort stage, not simply per mannequin. Whole spend together with failed makes an attempt, divided by duties that handed your acceptance verify. The rating won’t match the speed card, and the most affordable setting might nicely win.
Cap on tokens relatively than wall clock except latency is genuinely in your service stage goal. A wall-clock cap scores your supplier's serving pace as mannequin high quality.
Test the default effort setting on every little thing you have got deployed. Qwen 3.8-Max runs at its highest reasoning setting when the trouble area is unset, and its highest setting was its worst performer in impartial testing. A group that by no means touches that parameter is operating the configuration that prices probably the most per solved job.
