The difference between benchmark performance and real-world performance has been the theme of the year in AI research, so Google was right on cue with its first prospective clinical trial for AMIE using actual patients.
Meet the Articulate Medical Intelligence Explorer. AMIE is Google’s flagship medical AI, and it teamed up with Beth Israel Deaconess Medical Center to see how well it performed in real clinical workflows.
- 100 patients completed an AMIE interaction before their PCP visit, with AMIE taking histories and giving patients potential diagnoses to discuss with their PCP.
- PCPs received the transcript and summary prior to the visit, and all interactions were monitored live by physicians trained to intervene if safety criteria weren’t met.
AMIE got a gold star. Not only were there zero safety interventions across all 100 cases, patient attitudes toward AI significantly increased after interacting with AMIE.
- PCPs using AMIE reported increased visit preparedness in 75% of cases, as well as potential behavior change in nearly 60%.
- AMIE included the correct final diagnosis in 90% of cases, and 75% of AMIE’s differentials had the top 3.
- The quality of AMIE’s differential diagnosis and management plan appropriateness and safety was similar to PCPs, although PCPs won on management plan practicality and cost-effectiveness.
Other highlights were between-the-lines. PCPs had the chart, the physical exam, and the AMIE transcript, yet AMIE still matched them on differential quality and management safety.
- That speaks to the ceiling (or lack there-of) for structured AI history-taking, and shows that AI is ready to improve patient care in more ways than just making predictions.
- The fact that PCPs reported improved preparedness and potential behavior change in over half of the cases highlights how AI can augment – not replace – clinical reasoning.
The Takeaway
The distance between the bench and bedside is getting shorter, and Google’s AMIE results suggest that conversational AI in primary care is closer to reality than most people might think.
