Tag: amie

Google’s AMIE Survives Peer Review, Mostly

Posted on by Jason Barry

The peer-reviewed version of Google’s study on AMIE’s real-world performance was just printed in The Lancet, and the headline safety result made it through intact. Some of the preprint’s bolder claims did not.

ICYMI. The setup is identical to the March preprint on the Articulate Medical Intelligence Explorer, Google’s “medical AI researcher” that’s better known as AMIE 

  • 114 urgent care patients at Beth Israel Deaconess Medical Center chatted with AMIE, which then created a list of potential diagnoses and management plans for PCPs to review prior to the appointments.
  • AMIE’s conversations were supervised live in case anyone needed to step in, and blinded evaluators graded the outputs afterward.

AMIE passed with flying colors. Key highlights:

  • Zero conversations needed a safety stop. Supervisors logged just one hallucination, where AMIE described a past surgery as upcoming.
  • AMIE’s differential included the final diagnosis in its top seven candidates for 90% of cases, and its top three for 75%.
  • Of the 44 PCPs who had AMIE’s outputs before their visits, 33 found it helpful for prep and 25 said it may have changed their approach.
  • Evaluators rated the conversations favorably in 87-100% of cases, and patient attitudes toward AI improved after chatting with AMIE.

Here’s what got cut. In March, Google claimed that AMIE’s differentials and management plans were the same quality as those produced by the PCPs – “without significant differences.”

  • The Lancet version drops that comparison entirely because of confounding: the clinicians had read AMIE’s transcript before writing their own assessments, which could have biased their diagnoses.
  • The management plan comparison survived, with no significant gap on appropriateness or safety, but PCPs taking the lead on practicality and cost-effectiveness.

That wasn’t the only change. Supervisor interventions grew from three in the preprint to five in the final paper, primarily because AMIE included a “malignancy” in three of its potential diagnoses.

  • One of those listed lymphoma, and the treating clinician believed “the patient might have had anxiety related to the output and assessed the interaction as somewhat harmful in hindsight.” The supervisor saw no visible distress and let it run.

The Takeaway

Zero safety stops in a real clinical setting is a nice victory for AMIE, and it will be even nicer if it can scale to more patients. Luckily, Google already has a nationwide study underway with Included Health to show it can. 

AMIE Graduates From Chatbot to Video Visits

Posted on by Jason Barry

The moment has arrived. AI is now as good as physicians at conducting real-time video consultations – so long as patients stick to the script.

There’s a new expert in town. It’s Google’s flagship medical AI researcher named the Articulate Medical Intelligence Explorer, but its friends call it AMIE for short. 

AMIE gets its eyes and ears from a three-agent architecture built on Gemini and Project Astra, which splits the job so no single model has to think and respond at the same time:

  • A Talker agent keeps the conversation flowing with low-latency responses.
  • A Planner agent continuously updates the differential diagnosis and clinical goals.
  • A Perception agent watches the live audio-video stream for clinically relevant cues.

Google put AMIE through the wringer. A randomized study pitted AMIE against 10 board-certified PCPs across 100 telehealth scenarios enacted by 15 professional patient actors, with 20 independent physicians grading the encounters.

AMIE matched or beat the PCPs across the board:

  • Overall clinical rubric score: 83% vs. 68%
  • Top-1 diagnostic accuracy: 91% vs. 77%
  • Perception and examination: 74% vs. 47%

The physical exam gap was the eye-opener. AMIE proactively coached patient actors through maneuvers like range-of-motion tests and self-palpation, while the human PCPs mostly fell back on verbal history-taking.

  • Patient actors preferred AMIE for assessing and explaining their conditions, but still preferred humans for rapport, with interviews describing AMIE as having awkward pauses and odd conversational cadence.

Here’s the disclaimer. These were patient actors following scripts, not real patients. The physicians had to keep their cameras off, and AMIE still whiffed on subtle signals like tremors, nystagmus, and affect – the exact cues where perception matters most.

  • That’s not exactly real-world validation, but that’s already underway through a feasibility study with Beth Israel Deaconess and a nationwide study with Included Health.

The Takeaway

When AMIE first prompted predictions that patients would soon see an “AI doctor” before a human one, the missing physical exam was the go-to rebuttal. That gap just got noticeably smaller, and it doesn’t feel like we’ll be able to write too many more of these before it disappears completely.

Google AMIE Shines in First Real-World Study

Posted on by Jason Barry

The gap between benchmark scores and real-world performance has been the theme of the year in AI research, so Google was right on cue with its first prospective clinical trial for AMIE using actual patients. 

Meet the Articulate Medical Intelligence Explorer. AMIE is Google’s flagship “medical AI researcher,” and it teamed up with Beth Israel Deaconess Medical Center to gauge performance in real clinical workflows.

  • 100 patients completed an AMIE interaction before their primary care visit, with AMIE taking medical histories and equipping patients with potential diagnoses to discuss with their PCP.
  • PCPs received the transcript, summary, and AMIE’s management plan prior to the visit. All interactions were monitored live by physicians trained to intervene if safety criteria weren’t met.

AMIE got a gold star. Not only were there zero safety stops across all 100 interactions, patients reported that their attitudes toward AI significantly improved after chatting with AMIE.

  • AMIE’s differential included the correct final diagnosis in 90% of cases (per chart review 8 weeks post-encounter), with 75% top-3 accuracy.
  • PCPs using AMIE reported increased visit preparedness in 75% of cases, as well as potential behavior change in nearly 60%.
  • The quality of AMIE’s differential diagnosis and management plan appropriateness was similar to PCPs, although PCPs won on management plan practicality and cost-effectiveness.

Other findings were less obvious. PCPs had the chart, the physical exam, and the pre-visit transcript, yet AMIE still matched them on differential quality and management safety without taking a single peak at the EHR.

  • That speaks to the ceiling (or lack there-of) for structured AI history-taking, and shows that AI is gearing up to improve patient care in more ways than just making predictions.
  • The fact that PCPs reported better visit preparedness and potential behavior change in over half of cases also highlights how AI can augment – not just replace – clinical reasoning.

The Takeaway

The distance between the bench and bedside is getting shorter, and Google’s AMIE results suggest that conversational AI in primary care is closer to reality than most people might think.

Get the top digital health stories right in your inbox