The peer-reviewed version of AMIE’s real-world primary care study was just printed in The Lancet, and the headline safety result made it through intact – a few of the preprint’s bolder claims did not.
ICYMI. The setup is identical to the preprint.
- 114 adults with urgent care appointments at Beth Israel Deaconess chatted with AMIE up to five days before the visit while an internal medicine physician watched every conversation live.
- PCPs got the transcript and summary ahead of time, and blinded evaluators graded the outputs afterward.
AMIE passed with flying colors. Key highlights:
- Zero conversations needed a safety stop. Supervisors logged one hallucination (AMIE described a past surgery as upcoming) and added clinical information in five interactions.
- Clinical evaluators rated the conversations favorably in 87-100% of cases across 17 criteria, and patient attitudes toward AI improved after the chat and stayed up after the visit.
- AMIE’s differential included the final diagnosis in its top seven 90% of the time and its top three 75% of the time.
- Of the 44 PCPs who read the transcript before the visit, 33 found it helpful for prep and 25 said it may have changed their approach.
Now the preprint comparison. In March, Google reported AMIE’s differentials were statistically indistinguishable from the PCPs’. The Lancet version drops that comparison entirely because of confounding: the PCPs had read AMIE’s transcript before writing their own assessments.
The management plan comparison survived, with no significant gap on appropriateness or safety and PCPs winning on practicality and cost-effectiveness.
- Supervisor interventions grew from three in the preprint to five in the final paper, three of them because AMIE put a malignancy on the differential.
- One of those listed lymphoma, and the treating clinician later judged the interaction somewhat harmful in hindsight for the anxiety it may have caused. The supervisor saw no visible distress and let it run.
- The surgery-timing error was folded into the intervention count in March and is now labeled a hallucination and counted on its own.
Other points worth mentioning: Google funded the study, AMIE ran on Gemini 2.5 with no medical fine-tuning, and Google already has a nationwide virtual care study underway with Included Health.
The Takeaway
Zero safety stops in a real clinic is the number that matters, and it survived peer review. The doctor comparison didn’t, which is a decent reminder that a preprint is a press release with a methods section.
