A recent study out of Stanford explored whether doctors would revise their medical decisions in light of new AI-generated information, finding that docs are more than willing to change their minds despite being just as vulnerable to cognitive biases as the rest of us.
Here’s the set up published in Nature Communications Medicine:
- 50 physicians were randomized to watch a short video of either a white male or black female patient describing their chest pain with an identical script.
- The physicians made triage, diagnosis, and treatment decisions using any non-AI resource.
- The physicians were then given access to GPT-4 (which they were told was an AI system that had not yet been validated) and allowed to change their decisions.
The initial scores left plenty of room for improvement.
- The docs achieved just 47% accuracy in the white male patient group.
- The docs achieved a slightly better 63% accuracy in the black female patient group.
The physicians were surprisingly willing to change their minds based on the AI advice.
- Accuracy scores with AI improved from 47% to 65% in the white male group.
- Accuracy scores with AI improved from 63% to 80% in the black female group.
Not only were the physicians open to modifying their decisions with AI input, but doing so made them more accurate without introducing or exacerbating demographic biases.
- Both groups showed nearly identical magnitudes of improvement (18%), suggesting that AI can augment physician decision-making while maintaining equitable care.
- It’s worth noting that the docs used AI as more than a search engine, asking it to bring in new evidence, compare treatments, and even challenge their own beliefs [Table].
The Takeaway
Although having the doctors go first means that AI didn’t save them any time in this study – and actually increased time per patient – it showed that flipping the paradigm from “doctors checking AI’s work” to “AI helping doctors check their own work” has the potential to improve clinical decisions without amplifying biases.
