Risk scores are used to help predict everything from hospital readmissions to medication adherence, but more-often-than-not these valuable metrics are still generated by adding up a few key predictors in a simple model. That makes the difficult task of selecting the right variables extremely important, which is why researchers out of Duke-NUS Medical School in Singapore developed a machine learning system to help optimize the process.
Most current risk scores are generated from regression models that are popular for their simplicity and predictive power. The widely-used LACE index predicts unplanned hospital readmissions using only four components: inpatient length of stay, acute admission, number of ED visits in the past 6 months, and comorbidities.
There have been several efforts to improve risk scores using advances in AI, but these models are usually “black boxes,” making it notoriously difficult to interpret variable selection and importance.
The Duke-NUS researchers proposed an AutoScore-ShapleyVIC machine learning model for improving risk score variable selection while maintaining interpretability, then compared it to the LACE index for predicting 30-day readmissions of patients who visited the ED of Singapore General Hospital between 2009 and 2017.
We’ll leave the model development details to the AI experts, but here’s a look at how AutoScore-ShapleyVIC performed against the LACE index:
- AUC: 0.756 (AutoScore-ShapleyVIC) vs.0.733 (LACE)
- Accuracy: 71% vs. 64%
- Sensitivity: 66% vs. 74%
- Specificity: 72% vs. 62%
The final AutoScore-ShapleyVIC model significantly outperformed the LACE index, but didn’t sacrifice interpretability to achieve its results. Although it was able to narrow 41 candidate variables down to just 6 for the final model (number of ED visits in past 6 months, metastatic cancer, age, sodium, renal disease, and ED triage), the logic behind the variable selection can still be visualized to help with interpretation.
The Takeaway
Although we don’t usually cover AI studies, this research helped illustrate that machine learning models have a lot of potential to improve existing care paths without sacrificing interpretability. The authors emphasized how this particular approach is not limited to any specific clinical application, suggesting that machine learning algorithms are still in the early stages of improving risk scores in other areas such as ED triage or cardiac arrest survival.
