Trending AI Insights: Why Can't the Best AI Agent Finish the Job?
SemiAnalysis tested 13 frontier models on 206 real professional tasks. The winner finished 12% of them, at about $51 a task. The same week, Anthropic's filing showed what that arithmetic does to a business.
Two documents landed within twenty-four hours of each other this week, and neither was a product launch. The first was a benchmark. The second was an SEC filing. Read separately, one is a scoreboard and the other is an income statement. Read together, they describe the same constraint from two ends of the same pipe.
On 2 November, the semiconductor research firm SemiAnalysis published GDPx, a benchmark for agentic work in the eleven domains that pay the most: chip design, hardware design, cybersecurity, biology, finance, law, quantitative trading, materials science, software engineering, alignment research and recursive self-improvement. The headline result is that the best model completes about 12% of the tasks.
The same day, Anthropic filed an amended S-1 ahead of its listing. Revenue for the September quarter came in at $16.96 billion, up 1,127% year on year. Compute and infrastructure spending rose 77% in a single quarter, to $9.9 billion, and for the first time the company earned less revenue per dollar of that spending than it had the quarter before.
What GDPx actually measures
The design choice is the story. Older benchmarks handed the model a tidy packet — all attachments included, conditions stated, one correct answer — which made results comparable and quietly deleted the hardest part of real work. The GDPx authors say this directly: previous benchmarks did not give each task a domain-specific simulated working environment. Their example is a human expert asked to recommend one of two investments. Step one is spending hours gathering information. That gathering is the difficult part, and it was absent from the tests.
So GDPx builds the environment instead. Tasks come from real settings: private-equity deal work for the finance items, actual zero-day vulnerabilities for the security items, and out-of-sample daily profit and loss judged by a paired t-test for the trading items — no language model sitting in the judge's chair. Environments run to roughly 4,000 files. The default time limit is eight hours. Models work inside their native harnesses — Claude Code, Codex, Antigravity — and deliver an XLSX, PPTX, DOCX or PDF, or edit an existing file in place. Grading follows the standard a senior reviewer in a corporate network would apply. The first version has 206 tasks, built with 14 specialist data partners, and scores 13 frontier models.
That is a much harder exam than a multiple-choice suite, and the scores look like it.
12%, 10.9%, 9.6%
The top of the first leaderboard: Claude Opus 5.5 at 12.0%, Gemini 4 Carbon at 10.9%, GPT-6 Astra at 9.6%. Reported secondary coverage puts the per-task cost at roughly $51 for Opus 5.5, $16.29 for Gemini 4 Carbon and $53.99 for GPT-6 Astra.
Those cost figures are the more interesting half. Gemini trails the leader by about one percentage point and costs roughly a third as much per attempt. Astra scores below both and spends more than either. At a 12% success rate, a task that needs to succeed once will take multiple attempts, so the effective price of one delivered result is not the per-task figure — it is that figure divided by the success rate, and then multiplied by however many tries you allow. At these numbers, one completed professional task costs hundreds of dollars in inference before a human has touched it.
There is also a genuine specialisation result buried in the table. On cybersecurity tasks, Meta's Muse Wonder Full leads by what the benchmark describes as a clear margin, ahead of both Google models, which in turn beat OpenAI's and Anthropic's. No single model is best at everything, which is exactly what you would expect from a test built out of eleven different professions — and exactly what a single composite score hides.
The ranking flips when you take the safety guardrails out
SemiAnalysis ran a second cut of the same data with cybersecurity, biology and alignment removed. GPT-6 Astra moves to first, and GPT-6.1 Sol and Claude Sonnet 5.5 both move ahead of Gemini 4 Carbon. The authors say this second ranking matches their subjective impression of the models better than the headline one does.
The reason is mundane and important: on the sensitive domains, some models refuse the task outright. A refusal is scored as a failure, so a model with tighter guardrails looks weaker than a model that simply attempts everything. That is not a flaw unique to GDPx — every capability benchmark with a safety layer inherits it — but it means the headline number is measuring capability and policy at the same time, and the two cannot be separated from the outside.
Two further results deserve the same scepticism. Finance and law appear to be near-total failures across the board, which says as much about grading a legal memo as it does about the models. And Ethan Mollick of Wharton noted the obvious missing control: there is no human baseline in the table. Without one, 12% is a number, not a verdict. It could mean models are far from professional delivery, or it could mean the rubric is harsher than a human reviewer would be. Nobody has run the comparison yet.
Two numbers from the same forty-eight hours
Anthropic's amended S-1 is the other half of the argument, and it is unusually candid for a pre-IPO document. Third-quarter revenue reached $16.956 billion against $1.382 billion a year earlier. Operating expenses reached $17.523 billion, up 292%, which is more than the revenue. The company posted a $567 million operating loss one quarter after its first operating profit of $974 million. Net loss widened to $23.291 billion, though that line is dominated by non-cash items from convertible notes and derivative liabilities and should not be read as cash burn.
The number worth pinning to the wall is the ratio. In the second quarter, every dollar of compute and infrastructure spending produced $2.08 of revenue. In the third quarter it produced $1.72 — the first decline the company has recorded. Demand outran reserved capacity, so Anthropic bought about $5 billion of short-term compute at spot prices, which is more expensive than the twelve-to-eighteen-month contracts it normally signs. New model price cuts and contract discounts cost another $600 million of revenue. Cost up, price down, in the same quarter.
Set that beside GDPx and the shape of the problem becomes clear. The industry is selling agentic work at prices set by competition, while paying for tokens at prices set by a compute shortage, and the work itself succeeds about one time in eight. Everyone in the chain is being squeezed by the same gap between what a task costs to attempt and what a completed task is worth.
The filing also carries two details that will matter to buyers. Commitments for compute and data-centre leases total about $895 billion, running to 2045, with roughly $448 billion of that expected before the end of 2030. And 5% to 7% of revenue in the first nine months came from customers violating the company's usage policies — a share Anthropic has been actively reducing since August, with the stated expectation that growth will slow as a result. Enterprise traction is real: 11,900 customers above $100,000 a year, 1,900 above $1 million, 250 above $10 million.
What this means if you are the one buying
The practical lesson from GDPx is not that agents are useless. It is that the unit of accounting has moved. A per-token price tells you what a thought costs. A per-task cost tells you what an attempt costs. Neither tells you what a delivered result costs, and that is the only number a budget can be built on.
Three working rules follow. First, measure success rate on your own tasks before you measure anything else, because a 30% cheaper model that fails twice as often is not cheaper. Second, stop buying the top of the leaderboard by default: Gemini 4 Carbon is within a point of the best score at roughly a third of the cost per attempt, and for most workflows that trade is obviously correct. Third, keep a human in the loop at the step where the failure is unrecoverable — which for eight-hour tasks is usually not the last step but the fifth.
That last point is why the comparison work matters more than the ranking. Once models differ by cost per attempt, per-task success and domain specialisation, the useful question stops being “which model is best” and becomes “which model for this job, at what cost per delivered result.” That is the question aibookmarks.net is built around: a human-curated directory of AI tools (one of the most comprehensive AI tool directories available) organised by the job you need done rather than by launch date, with side-by-side comparisons that put running cost next to capability. If you are choosing between a frontier agent and a cheaper specialised tool this month, start from the task you need finished, not from the scoreboard.
Four caveats before you quote any of this
The original did not name the models. SemiAnalysis's own write-up states that even the best model reached 12% without attaching a name in the text. The model-by-model ordering, including Opus 5.5 at 12.0%, comes from secondary reporting and social transcription of the chart, not from a confirmed per-row table. Treat the ranking as indicative and the 12% as the firm figure.
Success is pass@1 with three attempts. That is a reasonable bar for one-shot delegation and a harsh one for a workflow that lets the agent retry. Re-run the numbers under your own retry policy before drawing conclusions about your use case.
Capability and policy are mixed. Refusals score zero. Any comparison that includes cybersecurity, biology or alignment is partly a comparison of guardrail strictness, which SemiAnalysis acknowledges by publishing the second cut.
No human baseline exists yet. Until a panel of senior professionals sits the same 206 tasks under the same eight-hour limit, the absolute number cannot be interpreted. The benchmark is scheduled to be refreshed every three to six months, so this may change.
What to do this week
- If you are budgeting an agent deployment, replace your per-seat or per-token estimate with cost per completed task. Multiply attempts by cost, divide by success rate, then add the review time.
- If you are choosing a model for long-horizon work, test Gemini 4 Carbon against your incumbent on a small set of your own tasks. A one-point capability gap is not worth a threefold cost gap on most workloads.
- If you are selling agentic work to clients, price the retries. The 12% figure is a warning about margin, not about capability.
- If you are evaluating vendors, ask for their success rate on tasks like yours, at pass@1. Anyone quoting a composite benchmark score without a cost column has not done the work.
---
Original author: Dylan Patel and the SemiAnalysis team, authors of the GDPx benchmark, with financial figures from Anthropic's amended Form S-1 filed with the US Securities and Exchange Commission on 2 November 2026.
Published by AI Bookmarks · More AI news →