AI Agent Reliability Gap: Why 80% of Enterprise Pilots Fail

What happened

At VB Transform 2026, Bryan Silverthorn — Director of AGI Autonomy at Amazon and a key figure in the company's AGI lab following Amazon's acquisition of Adept AI — argued that the central obstacle to enterprise AI agent deployment is not capability but reliability. Speaking on Tuesday, Silverthorn outlined why the vast majority of enterprise AI pilots never reach production, and proposed a four-part framework for thinking about reliability more rigorously.

Silverthorn drew on Princeton research to decompose reliability into four distinct dimensions: consistency, robustness, predictability, and safety. His core contention is that these properties are routinely conflated in internal evaluations, causing agents to perform well in controlled testing and then fail when exposed to real-world conditions. He described a concrete example of a customer who deployed an agent for software quality assurance, only to see it break down outside the narrow parameters of its evaluation environment.

His prescription is a shift in how organisations measure agent readiness — away from benchmark scores that bundle these dimensions together, and towards evaluations that stress-test each property separately before any production deployment.

Why it matters

For customer experience practitioners, this gap between pilot and production is not an abstract engineering problem — it is the moment a customer encounters a broken interaction. An AI agent that behaves consistently in a demo but unpredictably in a live service channel creates exactly the kind of variable, unreliable experience that erodes trust fastest. Behavioural economics is clear on this: customers weight negative surprises far more heavily than positive ones, and an agent that occasionally fails catastrophically will damage perception more than one that is merely mediocre but dependable.

Service designers who are evaluating or commissioning AI-assisted journeys should treat Silverthorn's framework as a procurement and governance lens, not just a technical one. Asking a vendor to demonstrate consistency, robustness, predictability, and safety as separate, evidenced properties — rather than accepting a composite benchmark score — is a practical step that sits well within a CX leader's remit, regardless of engineering background.

By the numbers

• 85% of enterprises are currently piloting AI agents, according to Cisco data cited at the event.
• 5% of enterprises have successfully shipped AI agents to production — revealing a deployment gap of 80 percentage points.
• 4 distinct reliability dimensions identified in the Princeton-sourced framework: consistency, robustness, predictability, and safety.

The Renascence take

The 85-to-5 gap is striking, but the more telling detail is why it exists. Most organisations are treating reliability as a single dial to turn up, when it is actually four separate levers — and conflating them is what makes agents look ready when they are not.

What most CX leaders will miss here is that this is fundamentally a service-design problem dressed in engineering language. The four dimensions Silverthorn describes map almost perfectly onto what customers actually need from any service interaction: that it behaves the same way every time (consistency), holds up under unusual inputs (robustness), lets them anticipate what comes next (predictability), and does not expose them to harm (safety). Organisations rushing to deploy agents should resist the temptation to treat a strong benchmark score as a green light. Instead, the smarter move is to run structured red-teaming against each dimension separately — and to set a minimum bar on predictability and safety before any customer-facing rollout, because those are the two failure modes that generate complaints, churn, and regulatory scrutiny.