nature Archives - Digital Health Wire

General-Purpose LLMs Outperform Healthcare-Specific Models

Posted on June 17, 2026June 17, 2026 by Jason Barry

We might have just gotten our spiciest study of the year after Nature published results suggesting that general-purpose LLMs outperform fine-tuned healthcare models straight out of the box.

It was a battle of the bots. Researchers pitted OE and UTD Expert AI against three general-purpose frontier models that anyone with a web browser can pull up in two seconds: GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6.

The models were tested across three domains:

medical knowledge (MedQA)
expert clinician alignment (HealthBench)
100 real physician queries (RCQ) scored by 12 blinded clinicians

It was a clean sweep. The general-purpose LLMs outperformed the specialized models on all three evals, and by a healthy margin. This chart gets the point across.

On MedQA, Gemini led the pack with 97.4% accuracy (vs. 89.6% for OE and 88.4% for UTD). Fun fact, the frontier models are trained on these exact questions (and answers).

On HealthBench, GPT-5.2 dominated with a score of 88%. It’s almost like OpenAI invented the benchmark.

The RCQs were probably the most clinically meaningful component, and all three frontier models took the podium here as well. It was a bit odd that the researchers didn’t share the exact questions, and OE thought so too.

OpenEvidence hit back hard and fast. It quickly took to socials to let the world know that the study was not only poorly designed and biased, but that the authors had reached out for API access to build a competing product. Request denied.

Outside of pointing out the data contamination issue with MedQA, OE also critiqued HealthBench for scoring responses based on subjective stylistic choices. It gave an example where OE scored 20% “worse” because it didn’t use a specific email header.

The cherry on top was OE revealing that the real-world clinician queries were only added after peer reviewers flagged the study for having weak evidence. Big if true.

Obligatory disclaimer: the models were tested back in February, and the performance gap is most likely even wider today.

That said, OE and UTD didn’t become this successful by being better AI developers than OpenAI and Anthropic. They did it by curating sources for verifiable evidence, wrapping them in an interface that docs love, and earning the trust of clinicians.

The Takeaway

Frontier models might eventually eat the world, and they’ll probably get pretty dang good at answering clinical questions in the process. In the meantime, benchmarks still aren’t medicine, and this study still isn’t the final word on clinical AI.

AI Can Help Doctors Change Their Minds

Posted on April 27, 2025April 28, 2025 by Jason Barry

A recent study out of Stanford explored whether doctors would revise their medical decisions in light of new AI-generated information, finding that docs are more than willing to change their minds despite being just as vulnerable to cognitive biases as the rest of us.

Here’s the set up published in Nature Communications Medicine:

50 physicians were randomized to watch a short video of either a white male or black female patient describing their chest pain with an identical script.

The physicians made triage, diagnosis, and treatment decisions using any non-AI resource.

The physicians were then given access to GPT-4 (which they were told was an AI system that had not yet been validated) and allowed to change their decisions.

The initial scores left plenty of room for improvement.

The docs achieved just 47% accuracy in the white male patient group.
The docs achieved a slightly better 63% accuracy in the black female patient group.

The physicians were surprisingly willing to change their minds based on the AI advice.

Accuracy scores with AI improved from 47% to 65% in the white male group.
Accuracy scores with AI improved from 63% to 80% in the black female group.

Not only were the physicians open to modifying their decisions with AI input, but doing so made them more accurate without introducing or exacerbating demographic biases.

Both groups showed nearly identical magnitudes of improvement (18%), suggesting that AI can augment physician decision-making while maintaining equitable care.

It’s worth noting that the docs used AI as more than a search engine, asking it to bring in new evidence, compare treatments, and even challenge their own beliefs [Table].

The Takeaway

Although having the doctors go first means that AI didn’t save them any time in this study – and actually increased time per patient – it showed that flipping the paradigm from “doctors checking AI’s work” to “AI helping doctors check their own work” has the potential to improve clinical decisions without amplifying biases.

AI Misses the Mark on Detecting Critical Conditions

Posted on March 17, 2025March 17, 2025 by Jason Barry

Most health systems have already begun turning to AI to predict if patient health conditions will deteriorate, but a new study in Nature Communications Medicine suggests that current models aren’t cut out for the task.

Virginia Tech researchers looked at several popular machine learning models cited in medical literature for predicting patient deterioration, then fed them datasets about the health of patients in ICUs or with cancer.

They then created test cases for the models to predict potential health issues and risk scores in the event that patient metrics were changed from the initial dataset.

AI missed the mark. For in-hospital mortality prediction, the models tested using the synthesized cases failed to recognize a staggering 66% of relevant patient injuries.

In some instances, the models failed to generate adequate mortality risk scores for every single test case.

That’s clearly not great news, especially considering that algorithms that can’t recognize critical patient conditions obviously can’t alert doctors when urgent action is needed.

The study authors point out that it’s extremely important for technology being used in patient care decisions to incorporate medical knowledge, and that “purely data-driven training alone is not sufficient.”

Not only did the study unearth “alarming deficiencies” in models being used for in-hospital mortality predictions, but it also turned up similar concerns with models predicting the prognosis of breast and lung cancer over five-year periods.

The authors conclude that a significant gap exists between raw data and the complexities of medical reality, so models trained solely on patient data are “grossly insufficient and have many dangerous blind spots.”

The Takeaway

The promise of AI remains just as immense as ever, but studies like this provide constant reminders that we need a diligent approach to adoption – not just for the technology itself but for the lives of the patients it touches. Ensuring that medical knowledge gets incorporated into clinical AI models also seems like a theme that we’re about to start hearing more often.

Study: AI is in the Eye of the Beholder

Posted on October 23, 2023October 23, 2023 by Jason Barry

At a time when new healthcare AI solutions are getting unveiled every week, a study in Nature Machine Intelligence found that the way people are introduced to these models can have a major effect on their perceived effectiveness.

Researchers from MIT and ASU had 310 participants interact with a conversational AI mental health companion for 30 minutes before reviewing their experience and determining whether they would recommend it to a friend.

Participants were divided into three groups, which were each given a different priming statement about the AI’s motives:

No motives: A neutral view of the AI as a tool
Caring motives: A positive view where the AI cares about the user’s well-being
Manipulative motives: A negative view where the AI has malicious intentions

The results revealed that priming statements certainly influence user perceptions, and the majority of participants in all three groups reported experiences in line with expectations.

88% of the “caring” group and 79% of the “no motive” group believed the AI was empathetic or neutral – despite the fact that they were engaging with identical agents.

Only 44% of the “manipulative” group agreed with the primer. As the authors put it, “If you tell someone to be suspicious of something, then they might just be more suspicious in general.”

As might be expected, participants who believed the model was caring also gave it higher effectiveness scores and were more likely to recommend it to a friend. That’s obviously relevant for those developing similar mental health chatbots, but a key insight for presenting any AI agent to new users.

An interesting feedback loop was also found between the priming and the conversation’s tone. People who believed the AI was caring tended to interact with it in a more positive way, making the agent’s responses drift positively over time. The opposite was true for those who believed it was manipulative.

The Takeaway

The placebo effect is a well documented cornerstone of medical literature, but this might be the first study to bridge the phenomenon from sugar pill to AI chatbot. Although AI is often thought of as primarily an engineering problem, this research does a great job highlighting how human factors and the power of belief play a huge role in the perceived effectiveness of the technology.

Can Wearables Help Measure Patient Outcomes?

Posted on November 14, 2021October 13, 2023 by Jason Barry

Researchers from the University of Edinburgh published a systematic review in Nature that aimed to determine the current evidence base and reporting quality for mobile digital health interventions (DHI) in the postoperative period following surgery.

Methodology – After screening 6,969 articles for patients undergoing surgeries where postoperative outcomes were measured using DHIs (defined as mobile technologies to improve health system efficiency and health outcomes), 44 studies were included in the final review.

Results – The review indicated that several types of mobile phone- or wearables-generated data can improve the assessment of postoperative recovery:

patient-reported outcome data (from validated self-report tools)
continuous activity data (from wearables)
combining remote assessment with active clinical prompts or patient advice

DHI Shortcomings – Studies included in the analysis demonstrated that DHIs may facilitate patient recovery following major operations and reduce inappropriate service use, although they also revealed issues with the current evidence base that should be addressed:

patients are rarely engaged in the development of DHIs
only one study was designed to engage patients in reviewing their own data
high levels of exclusion exist for patients without relevant mobile technology

Discussion

The increasing availability of high quality mobile technologies provides a new bridge between clinical services and patients’ homes, and while the authors of the study are optimistic about the technology, they stress the importance of improving reporting standards if its potential is to be fulfilled.

Going forward, the researchers suggest that studies of DHIs in postoperative settings seek to provide meaningful comparisons to non-DHI care in order to demonstrate clinical value, with particular attention paid to reporting quality so that equitable comparisons can be made to existing research.

Get the top digital health stories right in your inbox

This content is exclusive to subscribers