The 70% factuality ceiling: why Google’s new ‘FACTS’ benchmark is a wake-up name for enterprise AI

There's no scarcity of generative AI benchmarks designed to measure the efficiency and accuracy of a given mannequin on finishing varied useful enterprise duties — from coding to instruction following to agentic net looking and software use. However many of those benchmarks have one main shortcoming: they measure the AI's capability to finish particular issues and requests, not how factual the mannequin is in its outputs — how effectively it generates objectively right info tied to real-world information — particularly when coping with info contained in imagery or graphics.

For industries the place accuracy is paramount — authorized, finance, and medical — the dearth of a standardized approach to measure factuality has been a vital blind spot.

That adjustments immediately: Google’s FACTS group and its information science unit Kaggle launched the FACTS Benchmark Suite, a complete analysis framework designed to shut this hole.

The related analysis paper reveals a extra nuanced definition of the issue, splitting "factuality" into two distinct operational eventualities: "contextual factuality" (grounding responses in supplied information) and "world data factuality" (retrieving info from reminiscence or the online).

Whereas the headline information is Gemini 3 Professional’s top-tier placement, the deeper story for builders is the industry-wide "factuality wall."

In line with the preliminary outcomes, no mannequin—together with Gemini 3 Professional, GPT-5, or Claude 4.5 Opus—managed to crack a 70% accuracy rating throughout the suite of issues. For technical leaders, this can be a sign: the period of "belief however confirm" is much from over.

Deconstructing the Benchmark

The FACTS suite strikes past easy Q&A. It’s composed of 4 distinct exams, every simulating a unique real-world failure mode that builders encounter in manufacturing:

1. Parametric Benchmark (Inner Information): Can the mannequin precisely reply trivia-style questions utilizing solely its coaching information?

2. Search Benchmark (Instrument Use): Can the mannequin successfully use an internet search software to retrieve and synthesize stay info?

3. Multimodal Benchmark (Imaginative and prescient): Can the mannequin precisely interpret charts, diagrams, and pictures with out hallucinating?

4. Grounding Benchmark v2 (Context): Can the mannequin stick strictly to the supplied supply textual content?

Google has launched 3,513 examples to the general public, whereas Kaggle holds a non-public set to stop builders from coaching on the take a look at information—a typical situation referred to as "contamination."

The Leaderboard: A Sport of Inches

The preliminary run of the benchmark locations Gemini 3 Professional within the lead with a complete FACTS Rating of 68.8%, adopted by Gemini 2.5 Professional (62.1%) and OpenAI’s GPT-5 (61.8%).Nevertheless, a better take a look at the information reveals the place the true battlegrounds are for engineering groups.

Knowledge sourced from the FACTS Staff launch notes.

For Builders: The "Search" vs. "Parametric" Hole

For builders constructing RAG (Retrieval-Augmented Technology) methods, the Search Benchmark is probably the most vital metric.

The info exhibits a large discrepancy between a mannequin's capability to "know" issues (Parametric) and its capability to "discover" issues (Search). As an example, Gemini 3 Professional scores a excessive 83.8% on Search duties however solely 76.4% on Parametric duties.

This validates the present enterprise structure commonplace: don’t depend on a mannequin's inside reminiscence for vital information.

In case you are constructing an inside data bot, the FACTS outcomes recommend that hooking your mannequin as much as a search software or vector database isn’t elective—it’s the solely approach to push accuracy towards acceptable manufacturing ranges.

The Multimodal Warning

Essentially the most alarming information level for product managers is the efficiency on Multimodal duties. The scores listed below are universally low. Even the class chief, Gemini 2.5 Professional, solely hit 46.9% accuracy.

The benchmark duties included studying charts, deciphering diagrams, and figuring out objects in nature. With lower than 50% accuracy throughout the board, this means that Multimodal AI isn’t but prepared for unsupervised information extraction.

Backside line: In case your product roadmap includes having an AI routinely scrape information from invoices or interpret monetary charts with out human-in-the-loop overview, you’re doubtless introducing important error charges into your pipeline.

Why This Issues for Your Stack

The FACTS Benchmark is more likely to develop into a typical reference level for procurement. When evaluating fashions for enterprise use, technical leaders ought to look past the composite rating and drill into the particular sub-benchmark that matches their use case:

• Constructing a Buyer Assist Bot? Take a look at the Grounding rating to make sure the bot sticks to your coverage paperwork. (Gemini 2.5 Professional truly outscored Gemini 3 Professional right here, 74.2 vs 69.0).

• Constructing a Analysis Assistant? Prioritize Search scores.

• Constructing an Picture Evaluation Instrument? Proceed with excessive warning.

Because the FACTS group famous of their launch, "All evaluated fashions achieved an general accuracy under 70%, leaving appreciable headroom for future progress."For now, the message to the {industry} is evident: The fashions are getting smarter, however they aren't but infallible. Design your methods with the idea that, roughly one-third of the time, the uncooked mannequin may simply be incorrect.