Few firms face increased stakes when deploying AI than Waymo, the self-driving automotive firm underneath Alphabet that spun out of Google. Its fashions don’t merely generate textual content or automate back-office duties: They assist automobiles navigate unpredictable streets, reply to human drivers and make split-second choices within the bodily world.
However the strategies Waymo makes use of to handle these dangers — steady analysis, fastidiously curated information, human oversight and clearly outlined enterprise outcomes — supply a broader playbook for enterprises deploying AI brokers in practically any business.
Manasi Joshi, Waymo’s director of engineering for methods intelligence and machine studying, defined at VB Remodel 2026 how the autonomous automobile firm trains, checks and deploys AI at scale. Thus far, Waymo has pushed greater than 220 million absolutely autonomous, or "rider-only," miles, with 17 occasions fewer critical crash accidents than human drivers over the identical distance, in line with the corporate.
To realize these spectacular outcomes, Joshi stated Waymo has adopted what she referred to as “eval-forced growth” or “eval-centric growth,” making analysis a core a part of engineering slightly than a last test carried out earlier than deployment.
“The stage at which our tasks are maturing will be simply sort of transpired based mostly on the eval maturity that they showcase,” Joshi stated.
In apply, Waymo assesses a challenge’s readiness partly by inspecting the maturity of the checks surrounding it. That strategy has clear implications for enterprises constructing customer support brokers, coding assistants, monetary methods or different AI purposes: If an organization can not reliably measure a system’s efficiency, it will not be prepared to put that system into manufacturing.
Evals should proceed after launch
Joshi stated a lot of Waymo’s high quality work has shifted towards evaluations, together with checks performed throughout mannequin coaching, after coaching and inside open-loop and closed-loop simulations.
“Eval is just not a one-time job to launch a mannequin,” she stated.
Waymo as a substitute treats analysis as a steady course of spanning driving, simulation and validation. Its methodology combines datasets, efficiency metrics and infrastructure able to working effectively at scale.
For enterprises, meaning testing an agent earlier than launch is inadequate. Groups should proceed evaluating it as underlying fashions, enterprise processes, person habits and incoming information change. These evaluations must also connect with precise enterprise outcomes slightly than relying solely on broad business benchmarks.
Joshi cautioned that model-quality measurements are solely as reliable because the analysis information behind them. Waymo subsequently pairs its efficiency claims with details about the properties of the datasets used to check its methods.
Testing the uncommon and harmful instances
Waymo’s analysis hierarchy stays grounded in a single overriding goal: security.
The corporate attracts on first-party driving logs, some third-party information and reasonable simulations that expose its methods to eventualities spanning billions of artificial miles. Activity homeowners select specialised information and metrics for conditions involving susceptible highway customers, railroad crossings, building zones and different complicated environments.
The identical precept applies outdoors autonomous driving. Enterprises want to check not solely the routine requests their brokers deal with efficiently, but in addition unusual conditions the place errors may create monetary, authorized, safety or reputational harm.
Joshi emphasised that Waymo doesn’t depart launch choices solely to automated methods. Its production-readiness opinions embrace intensive human oversight, whereas inner security leaders approve software program releases and service-area expansions.
“This isn’t AI-driven and utterly automated and 0 human oversight,” she stated. “Human lives are at stake.”
Effectivity can not come on the expense of reliability
Waymo faces one other downside acquainted to enterprise AI groups: Demand for compute, storage, reminiscence and community capability is rising sooner than the assets out there.
The corporate pursues effectivity throughout information extraction and storage, distributed mannequin coaching, mannequin distillation, simulation and analysis. It additionally emphasizes “information effectivity,” deciding on probably the most helpful coaching examples as a substitute of treating larger quantity as inherently higher.
Waymo started utilizing transformers in 2017 and subsequently expanded into massive language fashions, vision-language fashions and vision-language-action fashions. Joshi stated the corporate now makes use of generative multimodal fashions as a part of its foundation-model technique.
Waymo divides its expertise between onboard methods inside every automobile and off-board infrastructure used for mannequin growth, information processing and simulation. That mixture forces the corporate to optimize each real-time inference and the bigger methods supporting it.
Brokers want their very own evals
Waymo additionally makes use of AI brokers internally as productiveness instruments for engineers. Joshi stated brokers assist analyze information distributions, assess information effectivity and triage issues present in automobile telemetry, coaching runs and failed analysis jobs.
The purpose is to speed up investigative work so engineers can commit extra time to judgment and tough technical issues. However Waymo additionally evaluates these brokers to make sure they produce reliable, correct outcomes slightly than sending workers down unproductive paths.
For enterprise leaders, Waymo’s bigger lesson is that agentic AI requires greater than selecting a strong mannequin. Organizations want a clearly outlined goal, consultant analysis information, steady testing, infrastructure that may function effectively and named human decision-makers who stay accountable for deployment.
"Incomes belief is supremely necessary," Joshi stated.
