Harvard research reveals AI surpasses ER doctors in diagnosis, reaching performance limits: ‘We’re already at the ceiling’

(SeaPRwire) –   AI has already made its way into your workplace, your child’s classroom, and the courtroom. Now, it is expanding into hospitals.

A recent study led by a team of physicians and computer scientists from Harvard Medical School and Beth Israel Deaconess Medical Center compared emergency room diagnoses from OpenAI’s o1-preview with those provided by two attending internal medicine physicians.

The two sets of competing diagnoses were then evaluated by two other attending physicians, who had no idea whether results came from humans or AI. The final results leaned in AI’s favor.

“We tested the AI model against nearly every benchmark, and it outperformed both previous models and our physician baselines,” Arjun Manrai, senior co-author of the study and an assistant professor of biomedical informatics at Harvard’s Blavatnik Institute, said in a statement.

AI has already begun reshaping the medical field. Google DeepMind’s Alphafold is pushing biological research forward. Several emergency rooms have rolled out generative AI to take notes and build medical records. And in Utah, one AI system even prescribes medication to patients without a physician involved in the process (though physicians have pointed out this practice could put patients at risk).

The Harvard study is the latest evidence that AI models are growing increasingly capable of completing critical tasks for the medical profession, delivering results that even shocked the study’s own researchers.

“I thought it would just be a fun experiment, but I didn’t expect it to work this well. That was definitely not what ended up happening,” Adam Rodman, a senior author of the study and a doctor at Beth Israel, said in a statement.

How AI is transforming medicine

This study’s findings were especially striking because unlike older AI medical studies, researchers did not clean up the input data. Every case was presented exactly as it appeared in an electronic health record.

Peter Brodeur, a study co-author and a Harvard clinical fellow in medicine at Beth Israel Deaconess, said in a statement that AI models are growing more capable all the time.

“We used to evaluate models with multiple-choice tests; now they consistently score close to 100%, and we can’t track progress anymore because we’re already at the ceiling,” he said.

To be clear, researchers noted that the results do not mean AI is ready to replace physicians. Brodeur said that while AI excels at diagnosing conditions, it also often suggests unnecessary testing that can actually do more harm than good.

Even so, AI is already influencing how physicians make decisions. In a separate study published last December, a team of researchers found that 67% of physicians who initially recommended against a patient getting treatment changed their decision after AI suggested the opposite.

This shift happens despite the fact that there is no formal accountability framework for AI diagnoses, and the technology still has not earned full patient trust, Rodman told The Guardian.

Patients still “want humans to guide them through life or death decisions [and] to guide them through challenging treatment decisions,” he said.