The New England Journal of Medicine is adding to its library of top tier publications with the launch of a new journal focused on artificial intelligence – NEJM AI – and it’s gearing up for the January debut with a sneak peek at a few early-release articles.
Use of GPT-4 to Diagnose Complex Clinical Cases was a standout study from the preview, finding that GPT-4 correctly diagnosed over half of complex clinical cases.
Researchers asked GPT-4 to provide a diagnosis for 38 clinical case challenges that each included a medical history along with six multiple choice options. The most common diagnoses included 15 cases related to infectious disease (39.5%), five cases in endocrinology (13.1%), and four cases in rheumatology (10.5%).
- GPT-4 was given the plain unedited text from each case, and solved each one five times to evaluate reproducibility.
- Those answers were compared to over 248k answers from online medical-journal readers, which were used to simulate 10k complete sets of human answers.
GPT-4 correctly diagnosed an average of 21.8 cases (57%), while the medical-journal readers correctly diagnosed an average of 13.7 cases (36%). Not too shabby considering the LLM could only leverage the case text and not the included graphics.
- Based on the simulation, GPT-4 also performed better than 99.98% of all medical-journal readers, with high reproducibility across all five tests (lowest score was 55.3%).
A couple caveats to consider are that medical-journal readers aren’t licensed physicians, and that real-world medicine doesn’t provide convenient multiple choice options. That said, a separate study found that GPT-4 performed well even without answer options (44% accuracy), and these models will only grow more precise as multimodal data gets incorporated.
The Takeaway
The race to bring AI to healthcare is on, and it’s generating a stampede of new research investigating the boundaries of the tech’s potential. As the hype of the first lap starts to give way to more measured progress, NEJM AI will most likely be one of the best places to keep up with the latest advances.
