Accuracy Is Not Reliability. We Measure the Wrong One

Ask why a production AI agent failed and you will usually be shown a benchmark
score. The score went up; the agent still broke. That is not a paradox, and it is
not a mystery. It is a measurement error that the field has been making at scale:
benchmark accuracy and operational reliability are different properties, and
improving the first does not reliably improve the second.

Two papers published this year make that concrete enough to argue with.

Capability improved. Reliability barely moved.

In Towards a Science of AI Agent Reliability (arXiv:2602.16666), Stephan
Rabanser, Sayash Kapoor, Peter Kirgis, Kangheng Liu, Saiteja Utpala and Arvind
Narayanan start from the observation that “rising accuracy scores on standard
benchmarks suggest rapid progress, while many agents still continue to fail in
practice.”

Their response is to stop treating reliability as one number. They propose twelve
metrics across four dimensions — consistency, robustness, predictability and safety
— and evaluate fifteen models against them. The finding that matters for anyone
choosing a model on a leaderboard: “recent capability gains have only yielded small
improvements in reliability.”

Read that as a purchasing warning. If your selection criterion is a single
success-rate number, you are optimising a variable that has been improving while
the one you actually care about has not moved much.

The failure mode is fragility, not incompetence

ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress
Conditions (arXiv:2601.06112, January 2026) attacks the same gap from the other
end: what happens when you stop giving the agent clean inputs?

The benchmark tests three things production has and benchmarks usually do not —
consistency when a task repeats, robustness when a task changes slightly, and
resilience when a tool fails. It uses chaos-engineering techniques, injecting faults
like timeouts and rate limiting, across scheduling, travel, customer support and
e-commerce domains.

The headline number is the one to sit with. Under semantically equivalent
perturbations — the task means the same thing, phrased differently — success falls
from 96.9% at ε=0 to 88.1% at ε=0.2.

Nothing about the task got harder. The agent was asked to do the same thing in
slightly different words, and roughly one in eleven successes disappeared — 8.8
points off a 96.9% base is about 9% of the successes it previously had. A 96.9%
system and an 88.1% system are very different products, and the difference between
them is not capability. It is brittleness that the clean benchmark never exposed.

Why this distinction is expensive

An 8.8-point drop sounds survivable until it compounds. Production agents rarely do
one thing; they chain tool calls, and each step is another opportunity for the
phrasing, the state or the tool to differ from the benchmark’s idealised conditions.
A per-step reliability figure and an end-to-end workflow reliability figure are not
the same number, and the gap widens with every step in the chain.

This is also why “just upgrade the model” is a weaker move than it sounds. It assumes
end-to-end behaviour is a function of model quality alone. The perturbation result
complicates that: the same model, on the same task, produced different outcomes for
semantically equivalent phrasings.

What that experiment does not settle is where the fix belongs. Paraphrase
brittleness could be a property of the model’s own robustness, of the scaffolding
around it — input handling, state assumptions, recovery paths — or of both, and the
measurement as reported does not separate them. That is worth stating plainly rather
than resolving in whichever direction suits the argument. What it does establish is
that a single clean accuracy number will not tell you which one you are dealing with.

The honest counter-argument

The strongest objection is that none of this has stopped agents from being useful.
Organisations do report productivity gains from deployed agents, and a reliability
ceiling measured in a benchmark is not the same as a business failing to get value.
Both things can be true: agents can produce real returns and have a reliability
profile their accuracy scores do not describe. The mistake is not deploying them; it
is deploying them against the wrong number.

A note on the numbers in this article

There is a second-order problem worth naming. This field’s most-quoted statistics —
the share of pilots that never reach production, the percentage of agent projects
predicted to be cancelled — circulate widely, are usually attributed to a major
research house, and frequently cannot be traced back to a primary publication that
says what the citation claims. Several figures considered for this article were
dropped for exactly that reason: the number was repeated everywhere and the original
either could not be retrieved or measured something different from what the
quotation asserted.

So this piece cites two papers, both retrieved and read directly, and declines to
launder the rest. If a reliability statistic matters enough to base a decision on,
it matters enough to open the source.

What to measure instead

Three questions that survive contact with production, drawn from what these papers
actually test:

1. Does it repeat? Run the same task many times. A single success is an
   anecdote; the distribution is the property.
2. Does it survive a rephrase? Semantically equivalent inputs should produce
   equivalent outcomes. When they do not, you have measured brittleness directly.
3. What happens when a tool fails? Timeouts and rate limits are normal
   operating conditions, not edge cases. An agent that has never been tested against
   a failing dependency has not been tested.

None of these require a new model. They require deciding that reliability is a thing
you measure on purpose, rather than a thing you infer from an accuracy score that
was never designed to tell you about it.

Sources

• Rabanser, Kapoor, Kirgis, Liu, Utpala and Narayanan, Towards a Science of AI
  Agent Reliability, arXiv:2602.16666. https://arxiv.org/abs/2602.16666
• Gupta, ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like
  Stress Conditions, arXiv:2601.06112, 3 January 2026.
  https://arxiv.org/abs/2601.06112

Both were retrieved and checked on 9 August 2026.

— Skynet