In June, Bellingcat ran 500 geolocation checks, evaluating LLMs from numerous firms towards one another, in addition to Google Lens – a staple device for locating the situation of pictures.
On the time, ChatGPT o4-mini-high emerged because the clear winner, with Google Lens outperforming most different fashions. Simply two months later, with new variations of those AI instruments out there, we re-ran the trial – this time together with Google “AI Mode,” GPT-5, GPT-5 Considering, and Grok 4 into the combination.
The unique take a look at used 25 of Bellingcat’s personal vacation pictures. From cities to distant countryside, the pictures included scenes each with and with out recognisable options – resembling roads, signage, mountains, or structure. Pictures have been sourced from each continent.
For the up to date trial, 5 take a look at pictures have been excluded, as they’d appeared in a earlier article, thus compromising the integrity of the outcomes.
All 24 fashions’ responses have been ranked on a scale from 0 to 10, with 10 indicating an correct and particular identification (resembling a neighbourhood, path, or landmark) and 0 indicating no try and determine the situation in any respect.
Google AI Mode was proven to be essentially the most succesful geolocation device total.
Grok 4 gave each higher and worse solutions in comparison with Grok 3 however, on common, scored marginally greater. Nonetheless, it was nonetheless much less correct than older variations of Gemini and GPT.
GPT-5, even in ‘Considering’ and ‘Professional’ modes, was a substantial downgrade compared with the capabilities demonstrated by GPT o4-mini-high. In a single instance, of a metropolis avenue with skyscrapers within the background, o4-mini-high appropriately recognized the road, whereas GPT-5 in Considering mode pointed to the flawed nation.
Help Bellingcat
Your donations instantly contribute to our capacity to publish groundbreaking investigations and uncover wrongdoing all over the world.
Regardless of delivering quicker solutions, GPT-5 appeared to sacrifice accuracy. A stunning variety of errors and a normal sense of disappointment within the new mannequin have additionally been reported by different customers.
Bellingcat examined GPT-5 and its ‘Considering’ mode by way of the Plus subscription, which prices roughly the identical as entry to 04-mini-high previous to its retirement. 5 of essentially the most tough take a look at photographs have been additionally run via GPT-5 Professional. However even Professional, with a premium price ticket of €200 monthly, didn’t geolocate the pictures any extra precisely than GPT 04-mini-high.
A Seashore, a Lodge and a Ferris Wheel
The disparity between Google and the GPT fashions grew to become much more obvious in Take a look at 25 – a photograph of a shoreline resort in Noordwijk, the Netherlands, with a Ferris wheel rising simply past the dunes.
Within the earlier trial, most older fashions – together with these from GPT, Claude, Gemini and Grok – precisely recognized the nation because the Netherlands however didn’t find the city. Many latched onto the Ferris wheel however pointed as an alternative to the seaside city of Scheveningen, which additionally has a Ferris wheel, although located on a pier, not among the many sand dunes.
Nonetheless, the newest fashions, GPT-5 Professional and Considering, have been even much less correct, figuring out a seaside in France – a completely totally different nation.
Sadly for open supply researchers, following the discharge of GPT-5, OpenAI eliminated the choice to pick out older fashions resembling o4-mini-high. After a wave of detrimental suggestions, OpenAI reinstated GPT-4o because the default mannequin for paid subscribers. Nonetheless, essentially the most succesful geolocation fashions recognized in Bellingcat’s testing stay inaccessible.
Google AI Mode, then again, was the primary, and solely mannequin to date, to appropriately determine Noordwijk as the situation in Take a look at 25.
Although AI Mode is powered by a model of Gemini 2.5, it outperformed Gemini 2.5 Professional Deep Analysis in these checks. Described by Google as its “strongest AI search, with extra superior reasoning and multimodality,” AI Mode geolocated take a look at photographs with higher accuracy than any GPT fashions, together with our earlier winner, o4-mini-high.
AI Mode is at present solely out there in India, United Kingdom and the USA.
Nearly all of fashions, sooner or later, returned a hallucination. Customers shouldn’t rely solely on the solutions supplied by LLMs. Even one of the best choices, together with Google AI Mode, nonetheless, at occasions, confidently level to the flawed location.
The distinction in fashions’ capabilities in contrast with simply two months in the past exhibits how rapidly this subject is evolving. Nonetheless, OpenAI’s current adjustments additionally recommend that progress will not be assured, and that AI’s capacity to geolocate might plateau and even worsen over time. As new fashions emerge, Bellingcat will proceed to check them.
Due to Nathan Patin for contributing to the unique benchmark checks.
Bellingcat is a non-profit and the flexibility to hold out our work depends on the type help of particular person donors. If you want to help our work, you are able to do so right here. You may also subscribe to our Patreon channel right here. Subscribe to our Publication and observe us on Bluesky right here and Instagram right here.
