Random effects adjustment in machine learning models for cardiac surgery risk prediction: a benchmarking study (original) (raw)
Related papers
Risk stratification plays a major role in the clinical decision-making process, patient consent and clinical governance analysis. However, the calibration of current risk scores (e.g., European System for Cardiac Operative Risk Evaluation (EuroSCORE), The Society of Thoracic Surgeons (STS) risk score) has been shown to deteriorate over time – a process known as calibration drift. The introduction of new clinical scores with different variable sets typically result in disparate datasets due to different levels of missingness. This is a barrier to the full insight and predictive capability of datasets across all potentially available time ranges. Little is known about the use of ensemble learning with ensemble metrics to mitigate the effects of calibration drift and changing risk across siloed datasets and time. In this study, we evaluated the effect of various combinations of Machine Learning (ML) models in improving model performance. The National Adult Cardiac Surgery Audit dataset...
JACC: Cardiovascular Interventions, 2019
OBJECTIVES This study sought to determine whether machine learning can be used to better identify patients at risk for death or congestive heart failure (CHF) rehospitalization after percutaneous coronary intervention (PCI). BACKGROUND Contemporary risk models for event prediction after PCI have limited predictive ability. Machine learning has the potential to identify complex nonlinear patterns within datasets, improving the predictive power of models. METHODS We evaluated 11,709 distinct patients who underwent 14,349 PCIs between January 2004 and December 2013 in the Mayo Clinic PCI registry. Fifty-two demographic and clinical parameters known at the time of admission were used to predict in-hospital mortality and 358 additional variables available at discharge were examined to identify patients at risk for CHF readmission. For each event, we trained a random forest regression model (i.e., machine learning) to estimate the time-to-event. Eight-fold cross-validation was used to estimate model performance. We used the predicted time-to-event as a score, generated a receiver operating characteristic curve, and calculated the area under the curve (AUC). Model performance was then compared with a logistic regression model using pairwise comparisons of AUCs and calculation of net reclassification indices. RESULTS The predictive algorithm identified a high-risk cohort representing 2% of all patients who had an in-hospital mortality of 45.5% (95% confidence interval: 43.5% to 47.5%) compared with a risk of 2.1% for the general population (AUC: 0.925; 95% confidence interval: 0.92 to 0.93). Advancing age, CHF, and shock on presentation were the leading predictors for the outcome. A high-risk group representing 1% of all patients was identified with 30-day CHF rehospitalization of 8.1% (95% confidence interval: 6.3% to 10.2%). Random forest regression outperformed logistic regression for predicting 30-day CHF readmission (AUC: 0.90 vs. 0.85; p ¼ 0.003; net reclassification improvement: 5.14%) and 180-day cardiovascular death (AUC: 0.88 vs. 0.81; p ¼ 0.02; net reclassification improvement: 0.02%). CONCLUSIONS Random forest regression models (machine learning) were more predictive and discriminative than standard regression methods at identifying patients at risk for 180-day cardiovascular mortality and 30-day CHF rehospitalization, but not in-hospital mortality. Machine learning was effective at identifying subgroups at high risk for postprocedure mortality and readmission.
Machine learning algorithms for predicting mortality after coronary artery bypass grafting
Frontiers in Cardiovascular Medicine, 2022
BackgroundAs the era of big data analytics unfolds, machine learning (ML) might be a promising tool for predicting clinical outcomes. This study aimed to evaluate the predictive ability of ML models for estimating mortality after coronary artery bypass grafting (CABG).Materials and methodsVarious baseline and follow-up features were obtained from the CABG data registry, established in 2005 at Tehran Heart Center. After selecting key variables using the random forest method, prediction models were developed using: Logistic Regression (LR), Support Vector Machine (SVM), Naïve Bayes (NB), K-Nearest Neighbors (KNN), Extreme Gradient Boosting (XGBoost), and Random Forest (RF) algorithms. Area Under the Curve (AUC) and other indices were used to assess the performance.ResultsA total of 16,850 patients with isolated CABG (mean age: 67.34 ± 9.67 years) were included. Among them, 16,620 had one-year follow-up, from which 468 died. Eleven features were chosen to train the models. Total ventil...
Predicting Surgical Risk: How Much Data is Enough?
AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium, 2010
As medicine becomes increasingly data driven, caregivers are required to collect and analyze an increasingly copious volume of patient data. Although methods for studying these data have recently evolved, the collection of clinically validated data remains cumbersome. We explored how to reduce the amount of data needed to risk stratify patients. We focused our investigation on patient data from the National Surgical Quality Improvement Program (NSQIP) to study how the accuracy of predictive models may be affected by changing the number of variables, the categories of variables, and the times at which these variables were collected. By examining the implications of creating predictive models based on the entire variable set in NSQIP and smaller selected variable groups, our results show that using far fewer variables than traditionally done can lead to similar predictive accuracy.
2017
Coronary Artery Bypass Graft (CABG) surgery is the most common cardiac operation and its complications are associated with increased long-term mortality rates. Although many factors are known to be linked to this, much remains to be understood about their exact influence on outcome. In this study we used Machine Learning (ML) algorithms to predict long-term mortality in CABG patients using data from routinely measured clinical parameters from a large cohort of CABG patients (n=5868). We compared the accuracy of 5 different ML models with traditional Cox and Logistic Regression, and report the most important variables in the best performing models. In the validation dataset, the Gradient Boosted Machine (GBM) algorithm was the most accurate (AUROC curve [95%CI] of 0.767 [0.739-0.796]), proving to be superior to traditional Cox and logistic regression (p <0.01) for long-term mortality prediction. Measures of variable importance for outcome prediction extracted from the GBM and Rand...
Applying machine learning methods to predict operative mortality after tricuspid valve surgery
The Cardiothoracic Surgeon
Background EuroSCORE stratifies surgical risk in cardiac surgery; however, it is not explicitly for tricuspid valve surgery. Therefore, we aimed to apply machine learning (ML) methods to predict operative mortality after tricuspid valve surgery and compare the predictive ability of these models to EuroSCORE. This retrospective analysis included 1161 consecutive patients who underwent tricuspid valve surgery at a single center from 2009 to 2021. The study outcome was operative mortality (n=112), defined as mortality occurring within 30 days of surgery or the same hospital admission. Random forest, LASSO, elastic net, and logistic regression were used to identify predictors of operative mortality. Results EuroSCORE was significantly higher in patients who had operative mortality [8.52 (4.745–20.035) vs.4.11 (2.29–6.995), P<0.001] [AUC=0.73]. Random forest identified eight variables predicting operative mortality with an accuracy of 92% in the test set (age≥70 years, heart failure, ...
British Journal of Anaesthesia, 2019
Background: Rapid, preoperative identification of patients with the highest risk for medical complications is necessary to ensure that limited infrastructure and human resources are directed towards those most likely to benefit. Existing risk scores either lack specificity at the patient level or utilise the American Society of Anesthesiologists (ASA) physical status classification, which requires a clinician to review the chart. Methods: We report on the use of machine learning algorithms, specifically random forests, to create a fully automated score that predicts postoperative in-hospital mortality based solely on structured data available at the time of surgery. Electronic health record data from 53 097 surgical patients (2.01% mortality rate) who underwent general anaesthesia between April 1, 2013 and December 10, 2018 in a large US academic medical centre were used to extract 58 preoperative features. Results: Using a random forest classifier we found that automatically obtained preoperative features (area under the curve [AUC] of 0.932, 95% confidence interval [CI] 0.910e0.951) outperforms Preoperative Score to Predict Postoperative Mortality (POSPOM) scores (AUC of 0.660, 95% CI 0.598e0.722), Charlson comorbidity scores (AUC of 0.742, 95% CI
Journal of Cardiothoracic and Vascular Anesthesia, 2021
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
BMC Medical Research Methodology, 2022
Background: This study illustrates the use of logistic regression and machine learning methods, specifically random forest models, in health services research by analyzing outcomes for a cohort of patients with concomitant peripheral artery disease and diabetes mellitus. Methods: Cohort study using fee-for-service Medicare beneficiaries in 2015 who were newly diagnosed with peripheral artery disease and diabetes mellitus. Exposure variables include whether patients received preventive measures in the 6 months following their index date: HbA1c test, foot exam, or vascular imaging study. Outcomes include any reintervention, lower extremity amputation, and death. We fit both logistic regression models as well as random forest models. Results: There were 88,898 fee-for-service Medicare beneficiaries diagnosed with peripheral artery disease and diabetes mellitus in our cohort. The rate of preventative treatments in the first six months following diagnosis were 52% (n = 45,971) with foot exams, 43% (n = 38,393) had vascular imaging, and 50% (n = 44,181) had an HbA1c test. The directionality of the influence for all covariates considered matched those results found with the random forest and logistic regression models. The most predictive covariate in each approach differs as determined by the t-statistics from logistic regression and variable importance (VI) in the random forest model. For amputation we see age 85 + (t = 53.17) urban-residing (VI = 83.42), and for death (t = 65.84, VI = 88.76) and reintervention (t = 34.40, VI = 81.22) both models indicate age is most predictive. Conclusions: The use of random forest models to analyze data and provide predictions for patients holds great potential in identifying modifiable patient-level and health-system factors and cohorts for increased surveillance and intervention to improve outcomes for patients. Random forests are incredibly high performing models with difficult interpretation most ideally suited for times when accurate prediction is most desirable and can be used in tandem with more common approaches to provide a more thorough analysis of observational data.