Real-world comparisons of AI recommendations and doctors’ clinical decisions have been few and far between, but a new study in the Annals of Internal Medicine gave us a great look at how performance stacks up with actual patients.
The early verdict? AI came out on top, but that doesn’t mean doctors should pack their bags quite yet.
Researchers from Cedars-Sinai and Tel Aviv University compared recommendations made by K Health’s AI Physician Mode to the final decisions made by physicians for 461 virtual urgent care visits. Here’s what they found:
- In 68% of cases, the AI and physician recommendations were rated as equal
- AI rated better on 21% of cases, versus just 11% for physicians
- AI recommendations were rated “optimal” in 77% of cases, versus 67% for physicians
Although AI takes the cake with the top line numbers, unpacking the data reveals some not-too-surprising strengths and weaknesses. AI was primarily rated better when physicians:
- Missed important lab tests (22.8%)
- Didn’t follow clinical guidelines (16.3%)
- Failed to refer patients to specialists or the ED if needed (15.2%)
- Overlooked risk factors and red flags (4.4%)
Physicians beat out AI when the human elements of care delivery came into play, such as adapting to new information or making nuanced decisions. Physicians were rated better when:
- AI made unnecessary ED referrals (8.0%)
- There was evolving or inconsistent information during consultations (6.2%)
- They made necessary referrals that the AI missed (5.9%)
- They correctly adjusted diagnoses based on visual examinations (4.4%)
While the study focused on the exact types of common conditions that AI excels at diagnosing (respiratory, urinary, vaginal, eye, and dental), it’s still impressive to see the outperformance in the messy trenches of a real clinical setting – a far cry from the static medical exams that have been the go-to for similar evaluations.
The Takeaway
For AI to truly transform healthcare, it’ll need to do a lot more than automate administrative work and back office operations. This study demonstrates AI’s potential to enhance decision-making in actual medical practice, and points toward a future where delivering high-quality patient care becomes genuinely scalable.
