Importance: Machine learning (ML) models for allocating readmission-mitigating interventions are typically selected according to their discriminative ability, which may not necessarily translate into utility in allocation of resources. Objective: To determine whether ML models for allocating readmission-mitigating interventions are ranked differently based on their overall utility and their discriminative ability. Design: A retrospective analysis of ML models using claims data acquired from the Optum Clinformatics Data Mart. Setting: Health plan claims from all 50 states for commercially-insured individuals. Participants: 513,495 patients who were admitted as inpatients over the period January 2016 through January 2017. Main Outcomes and Measures: Maximum utility achieved by three machine learning models for allocating readmission-mitigating interventions, determined using cost accrued in the 90 days post-discharge of an index admission and estimated counterfactual cost. Data were analyzed between April 2019 and March 2020. Results: The study sample consisted of 513,495 patients (mean [SD] age 69  years; 294,895 [57%] Female) mean 90 day cost of $11,552 for the study period. Allocating readmission-mitigating interventions based on a LightGBM model trained to predict readmissions achieved a maximum utility of $-12,645 per patient, and an AUC of 0.74 (95% CI 0.74, 0.75); allocating interventions based on a model trained to predict cost as a proxy achieved a higher maximum utility of $-12,472 per patient, and an AUC of 0.63 (95% CI 0.62, 0.63). A hybrid model combining both intervention strategies achieved a maximum utility of $-12,472, and an AUC of 0.71 (95% CI 0.71, 0.71), comparable with the best models on either metric. Conclusion and Relevance: We demonstrate that machine learning models may be ranked differently based on overall utility and discriminative ability. Machine learning models for allocation of limited health resources should consider directly optimizing for utility.
- Downloaded 338 times
- Download rankings, all-time:
- Site-wide: 86,937
- In health informatics: 318
- Year to date:
- Site-wide: 53,500
- Since beginning of last month:
- Site-wide: 55,183
Downloads over time
Distribution of downloads per paper, site-wide
- 27 Nov 2020: The website and API now include results pulled from medRxiv as well as bioRxiv.
- 18 Dec 2019: We're pleased to announce PanLingua, a new tool that enables you to search for machine-translated bioRxiv preprints using more than 100 different languages.
- 21 May 2019: PLOS Biology has published a community page about Rxivist.org and its design.
- 10 May 2019: The paper analyzing the Rxivist dataset has been published at eLife.
- 1 Mar 2019: We now have summary statistics about bioRxiv downloads and submissions.
- 8 Feb 2019: Data from Altmetric is now available on the Rxivist details page for every preprint. Look for the "donut" under the download metrics.
- 30 Jan 2019: preLights has featured the Rxivist preprint and written about our findings.
- 22 Jan 2019: Nature just published an article about Rxivist and our data.
- 13 Jan 2019: The Rxivist preprint is live!