AI search engines cite incorrect sources at an alarming 60% rate, study says | CJR study shows AI search services misinform users and ignore publisher exclusion requests.
https://arstechnica.com/ai/2025/03/ai-search-engines-give-incorrect-answers-at-an-alarming-60-rate-study-says/
2 Comments
From the article: A new study from Columbia Journalism Review’s Tow Center for Digital Journalism finds serious accuracy issues with generative AI models used for news searches. The research tested eight AI-driven search tools equipped with live search functionality and discovered that the AI models incorrectly answered more than 60 percent of queries about news sources.
Researchers Klaudia Jaźwińska and Aisvarya Chandrasekar noted in their report that roughly 1 in 4 Americans now use AI models as alternatives to traditional search engines. This raises serious concerns about reliability, given the substantial error rate uncovered in the study.
Error rates varied notably among the tested platforms. Perplexity provided incorrect information in 37 percent of the queries tested, whereas ChatGPT Search incorrectly identified 67 percent (134 out of 200) of articles queried. Grok 3 demonstrated the highest error rate, at 94 percent.
For the tests, researchers fed direct excerpts from actual news articles to the AI models, then asked each model to identify the article’s headline, original publisher, publication date, and URL. They ran 1,600 queries across the eight different generative search tools.
The study highlighted a common trend among these AI models: rather than declining to respond when they lacked reliable information, the models frequently provided confabulations—plausible-sounding incorrect or speculative answers. The researchers emphasized that this behavior was consistent across all tested models, not limited to just one tool.
As a person who frequently googles things I’m about 80% confident in, just to ensure my accuracy, this has been glaringly obvious for some time now. We’ve reached an inflection point where my dogshit memory routinely exceeds the accuracy of google’s dogshit answers. I’m gradually migrating away from google for precisely this reason. Duckduckgo is tedious, but no information is more helpful than misinformation.
What’s odd is that other AI is often pretty accurate for this purpose. Like.. GPT is pretty helpful for figuring out what to search for. Maybe Google’s AI would be too, if I prompted it like an AI instead of a search engine. But if I wanted to do that, I would go to a dedicated AI, not a search engine.
It’s like someone realized how much people like cookies, chips, and salsa and decided to mix them all together to try to create the ultimate party food. The resultant soggy mess doesn’t really seem to work for anybody except the kind of person who never really cared about what they were eating in the first place.