AI Revolutionizes Evidence-Based Medical Reviews — But Experts Advise Caution

In a striking example of AI in healthcare, a group of researchers has dramatically accelerated the process of conducting systematic medical reviews, completing 12 high-quality analyses in just two days. These reviews, known for their role in shaping evidence-based emergency medicine and clinical policies, traditionally take months or even years to finalize.

The breakthrough, detailed in a recent medical AI preprint on medRxiv, was made possible by leveraging large language models (LLMs) like GPT-4.1 to automate two key stages: identifying relevant studies and extracting data. In one case, a review update that would typically require weeks was completed in just 20 minutes.

“By the time I get a coffee, the AI finishes an entire review,” said Christian Cao, a University of Toronto medical student and co-creator of the system. He and his team have since launched Otto Review, a startup focused on commercializing this artificial intelligence in healthcare solution.

A Leap Forward in Healthcare Innovation—But Not Fully Autonomous

While this marks a promising advancement in artificial intelligence clinical research, experts urge restraint. The system currently handles only screening and data extraction, leaving other vital steps such as review design, literature search, risk of bias assessment, meta-analysis, and writing to human experts.

“These are just two steps out of many,” said Justin Clark, a specialist in AI-assisted systematic reviews at Bond University. “It’s a significant start, but far from a complete solution”.

Even so, the tool matched—and in some cases surpassed—the accuracy of experienced human reviewers. Notably, in three of the 12 reviews, the AI for biotechnology and healthcare found new studies that altered the original conclusions. Without automation, updating all 12 reviews would have taken an estimated 12 years of manual labor.

Toward AI-Powered Living Reviews

The team envisions a future where medical AI tools can keep all published reviews up to date in real time—a concept known as “living reviews.” Although certain aspects remain under human oversight for quality control, the developers are working to expand automation throughout the entire process.

Meanwhile, global institutions like Cochrane and researchers such as James Thomas (University College London) are pushing for ethical guidelines to ensure transparency and accuracy in AI-powered medical tools.

“There are AI and healthcare tools emerging everywhere,” Thomas noted. “But the science to validate them must keep up”.

Conclusion

This project highlights the transformative potential of artificial intelligence in medicine, especially in reducing time and cost in systematic reviews. However, it also underscores the importance of human oversight and methodological rigor. While AI in healthcare can serve as a valuable tool, its role is best viewed as a collaborator—not a replacement—in clinical research.