Google’s AI Search for Science Dumps Impact Factors

According to The Verge, Google has announced it’s testing a new AI-powered search tool called Scholar Labs designed to answer detailed research questions. The tool uses AI to identify main topics and relationships in user queries and is currently available to a limited set of logged-in users through a waitlist. In a demo, the tool answered a question about brain-computer interfaces and surfaced a 2024 review paper from Applied Sciences journal, which self-reports an impact factor of 2.5 compared to Nature’s 48.5. Google spokesperson Lisa Oguike confirmed the new tool will not sort or limit results based on citation counts or journal impact factors, arguing these metrics can miss key papers in interdisciplinary fields or recently published articles.

The science vetting problem

Here’s the thing: scientists have relied on citation counts and journal impact factors for decades as quick quality checks. It’s basically the academic equivalent of checking a restaurant’s Yelp rating before booking a table. But Google is arguing these metrics are flawed – and they’re not entirely wrong. Citation counts favor older papers that have had time to accumulate references, while impact factors can exclude important research from less prestigious journals.

And yet, even the experts admit they can’t help using these shortcuts. James Smoliga, a Tufts University professor, told The Verge “I’m guilty of it just like everybody else is” despite having debunked methods in a study with thousands of citations. When you’re diving into a new field, what else are you going to do? The existing system isn’t perfect, but it’s what we’ve got.

search”>AI versus traditional search

The contrast becomes really clear when you compare Scholar Labs with traditional academic search tools. I tried the same BCI research query in PubMed, the NIH’s biomedical database, and got completely different results. PubMed lets you filter by study type, exclude preprints, and focus on human clinical research from specific timeframes. Scholar Labs? It’s basically throwing you into the deep end with AI as your only life preserver.

Google says their approach weighs “the full text of each document, where it was published, who it was written by, as well as how often and how recently it has been cited.” But without showing those citation numbers or letting users filter by them, researchers are left taking Google’s word that the AI understands context better than they do. That’s a big ask in fields where even Nobel winners need corrections and data fabrication happens.

Broader implications

Matthew Schrag, a Vanderbilt neurology professor who’s flagged dubious research, thinks AI search has potential to cast a wider net and surface overlooked papers. But he emphasizes that scientists need to remain “the final arbiters” of quality. The question is whether busy researchers will actually do that deep reading when an AI serves up seemingly relevant results.

Google is calling this a “new direction” and plans to incorporate user feedback. But fundamentally, they’re betting that language model understanding can replace decades of established academic signaling. It’s a bold move that could either democratize research discovery or make it harder to separate signal from noise.

The Scholar Labs experiment raises bigger questions about how we evaluate scientific quality in an AI-driven world. When traditional metrics are imperfect but AI is opaque, where does that leave researchers? Probably still checking citation counts on the side, if we’re being honest.