Legacy infrastructure, not the fashions themselves, is what's really slowing AI brokers down. That was the shared conclusion of three infrastructure leaders — from LinkedIn, Walmart, and Zendesk — at VB Rework 2026.
The panel introduced collectively Animesh Singh, senior director of AI platform and infrastructure at LinkedIn, Desiree Gosby, SVP of company expertise providers and expertise technique at Walmart, and Sami Ghoche, VP of utilized AI at Zendesk, every describing what really broke after they moved brokers from pilot to manufacturing. Every arrived on the identical conclusion from a unique start line: Not one of the bottlenecks they hit have been mannequin issues.
What tied their solutions collectively was a shared premise: most enterprise infrastructure was constructed for a way people work, not for a way brokers work. The hole between these two speeds is the place the actual engineering occurred.
Gosby put it plainly when requested what she'd realized scaling brokers inside Walmart's personal workforce. The purpose, she mentioned, is to verify "engineering doesn't as soon as once more develop into the bottleneck for what it’s we're making an attempt to do."
The place the bottleneck really was
Every firm hit a unique model of the identical wall: infrastructure designed for a way individuals work doesn't maintain up as soon as brokers are doing the work as a substitute.
At LinkedIn, the primary bottleneck wasn't a mannequin, it was Kubernetes, which assumes containers spin up on demand, a course of that takes seconds. Singh mentioned that's too gradual for brokers. The repair was shifting from on-demand provisioning to pre-provisioned swimming pools of containers that swap agentic workloads out and in in actual time.
A second, tougher drawback surfaced as soon as LinkedIn let brokers management their very own orchestration. A five-point analysis system seemed clear, however hallucination saved exhibiting up anyway. Singh mentioned the problem was structural, an LLM evaluating one other LLM's output shares the identical failure mode because the factor it's evaluating.
"We constructed our personal harness, our personal management circulation, and pushed the LLMs to the leaf as a substitute of them orchestrating the loop," Singh mentioned. Roughly 80% of the workflow is now scripted, deterministic code, with LLMs used solely the place reasoning is required, and every step's proof is dedicated to disk earlier than the system strikes on.
Walmart's bottleneck got here from success. An agent harness put straight into staff' palms went viral internally, and what Gosby known as "citizen builders" started constructing their very own brokers to unravel issues that when required a proper engineering roadmap. The upside was actual innovation. The draw back was duplication, dozens of overlapping brokers with no coordination. The repair wasn't reining within the harness, it was constructing governance to identify duplication, promote the most effective model of an agent, and get it into manufacturing with out engineering turning into a chokepoint.
Zendesk hit its bottleneck from the information aspect. Ghoche, who joined by means of Zendesk's acquisition of Forethought, which closed in March 2026, described sitting on what he known as a public determine of 20 billion buyer conversations in Zendesk's repository. The intuition is handy that historical past to a big language mannequin with a giant context window and let it generate the brokers a enterprise wants. Ghoche mentioned that doesn't work. "You’ll be able to't actually try this, so as a substitute you must actually spend money on the underlying knowledge pipelines and all the information infrastructure that comes with that," he mentioned.
The function of open supply
On open supply, all three leaders landed on the same intuition: personal what you may, and lean on frontier labs solely the place they nonetheless have a transparent edge.
Ghoche mentioned his personal view is that almost all enterprises would favor to personal their fashions and infrastructure wherever that's doable, and that reasoning is what drives Zendesk's personal method. The exception is frontier reasoning work, the place the labs nonetheless lead, although he mentioned that slice of use circumstances is shrinking relative to all the pieces else enterprises now do with AI.
LinkedIn's reply was to construct two subsystems particularly for independence. The primary is what the corporate calls an AI gateway, a single interface that each outbound name to a mannequin runs by means of no matter supplier. The second part is a reminiscence subsystem constructed to carry context unbiased of any mannequin supplier.
"Each single outbound name going to an LLM, whether or not it's on a public cloud or on-prem in our personal knowledge facilities, follows the identical semantics, the identical API calls. We are able to shortly swap between totally different suppliers," Singh mentioned.
Walmart constructed its personal inside gateway to remain vendor agnostic throughout three workload sorts: absolutely deterministic workflows, planner-and-reasoner workflows for open-ended duties, and a hybrid of the 2. Compliance-heavy work stays deterministic by design; governance, safety and analysis run by means of the gateway no matter which mannequin is on the opposite finish. Gosby mentioned the selection between a frontier mannequin and an open-weight mannequin comes right down to whichever is best for the particular workload, not a hard and fast coverage.
Recommendation for the modernization journey
Three items of recommendation got here up straight, every tied to the wall a pacesetter had already hit.
Spend money on evals earlier than the rest. Ghoche known as it the factor frequent to each use case, inside or buyer dealing with.
"The factor that's frequent to all of those is evals. It'll drive you to interrupt the issue down, and after getting a strong set of evals, you may transfer quite a bit quicker," he mentioned,
Personal your agent harness from day one. Gosby's recommendation was to place the AI harness straight in staff' palms early, paired with the infrastructure to watch what it produces.
"It’ll unlock an enormous quantity of innovation," she mentioned.
Construct for mannequin and context independence. Making certain flexibility is important for achievement.
"Construct for independence, whether or not it's a frontier mannequin of at present versus an open supply mannequin of tomorrow," Singh mentioned. "Hold that context inside your enterprise to be able to reuse it once you ship the mannequin or the harness tomorrow," Singh mentioned.
