Performance of administrative database frailty instruments in predicting clinical outcomes and cost for patients undergoing transcatheter aortic valve implantation: a historical cohort study (original) (raw)

Abstract

Purpose

Frailty instruments may improve prognostic estimates for patients undergoing transcatheter aortic valve implantation (TAVI). Few studies have evaluated and compared the performance of administrative database frailty instruments for patients undergoing TAVI. This study aimed to examine the performance of administrative database frailty instruments in predicting clinical outcomes and costs in patients who underwent TAVI.

Methods

We conducted a historical cohort study of 3,848 patients aged 66 yr or older who underwent a TAVI procedure in Ontario, Canada from 1 April 2012 to 31 March 2018. We used the Johns Hopkins Adjusted Clinical Group (ACG) frailty indicator and the Hospital Frailty Risk Score (HFRS) to assign frailty status. Outcomes of interest were in-hospital mortality, one-year mortality, rehospitalization, and healthcare costs. We compared the performance of the two frailty instruments with that of a reference model that adjusted baseline covariates and procedural characteristics. Accuracy measures included c-statistics, Akaike information criterion (AIC), Bayesian information criterion (BIC), integrated discrimination improvement (IDI), net reclassification index (NRI), bias, and accuracy of cost estimates.

Results

A total of 863 patients (22.4%) were identified as frail using the Johns Hopkins ACG frailty indicator and 865 (22.5%) were identified as frail using the HFRS. Although agreement between the frailty instruments was fair (Kappa statistic = 0.322), each instrument classified different subgroups as frail. Both the Johns Hopkins ACG frailty indicator (rate ratio [RR], 1.13; 95% confidence interval [CI], 1.06 to 1.20) and the HFRS (RR, 1.14; 95% CI, 1.07 to 1.21) were significantly associated with increased one-year costs. Compared with the reference model, both the Johns Hopkins ACG frailty indicator and HFRS significantly improved NRI for one-year mortality (Johns Hopkins ACG frailty indicator: NRI, 0.160; P < 0.001; HFRS: NRI, 0.146; P = 0.001) and rehospitalization (Johns Hopkins ACG frailty indicator: NRI, 0.201; P < 0.001; HFRS: NRI, 0.141; P = 0.001). These improvements in NRI largely resulted from classification improvement among those who did not experience the event. With one-year mortality, there was a significant improvement in IDI (IDI, 0.003; P < 0.001) with the Johns Hopkins ACG frailty indicator. This improvement in performance resulted from an increase in the mean probability of the event among those with the event.

Conclusion

Preoperative frailty assessment may add some predictive value for TAVI outcomes. Use of administrative database frailty instruments may provide small but significant improvements in case-mix adjustment when profiling hospitals for certain outcomes.

Résumé

Objectif

L’utilisation d’indicateur de fragilité pourrait améliorer l’évaluation pronostique des patients bénéficiant d’un remplacement valvulaire aortique par voie percutanée (procédure TAVI). Peu d’études ont évalué et comparé la performance des instruments d’évaluation de la fragilité développés à partir de données administratives chez les patients bénéficiant d’un TAVI. Nous avions pour objectif d’examiner la performance des instruments d’évaluation de la fragilité développés à partir de données administratives dans la prédiction des issues cliniques et des coûts chez les patients ayant bénéficié d’un TAVI.

Méthode

Nous avons réalisé une étude de cohorte historique auprès de 3848 patients âgés de 66 ans ou plus qui ont bénéficié d’une procédure TAVI en Ontario, Canada, du 1er avril 2012 au 31 mars 2018. Nous avons utilisé l’indicateur de fragilité ACG (Adjusted Clinical Group) de Johns Hopkins et le score de risque de fragilité à l’hôpital (HFRS) pour définir la fragilité. Les critères d’évaluation étaient la mortalité hospitalière, la mortalité à un an, la réhospitalisation et les coûts des soins de santé. Nous avons comparé la performance des deux instruments d’évaluation de la fragilité à celle d’un modèle de référence qui ajustait les covariables de base et les caractéristiques procédurales. Les mesures d’exactitude comprenaient l’analyse statistique c, le critère d’information d’Akaike (AIC), le critère d’information bayésien (BIC), l’amélioration de la discrimination intégrée (IDI), l’indice NRI (net reclassification index), le biais et l’exactitude des estimations de coûts.

Résultats

Au total, 863 patients (22,4 %) ont été identifiés comme fragiles à l’aide de l’indicateur de fragilité ACG de Johns Hopkins, et 865 (22,5 %) ont été identifiés comme fragiles à l’aide du HFRS. Bien que l’agrément entre les instruments d’évaluation de la fragilité ait été acceptable (statistique de Kappa = 0,322), chaque instrument a classé des sous-groupes différents comme étant fragiles. L’indicateur de fragilité ACG de Johns Hopkins (rapport de taux [RR], 1,13; intervalle de confiance à 95 % [IC], 1,06 à 1,20) et le HFRS (RR, 1,14; IC 95 %, 1,07 à 1,21) étaient associés de façon significative à une augmentation des coûts sur un an. Par rapport au modèle de référence, l’indicateur de fragilité ACG de Johns Hopkins améliorent de façon significative le NRI pour la mortalité (l’indicateur de fragilité ACG de Johns Hopkins: NRI, 0.160; P < 0.001; HFRS: NRI, 0.146; P = 0.001) et la réhospitalisation (l’indicateur de fragilité ACG: NRI, 0.201; P < 0.001; HFRS: NRI, 0.141; P = 0.001) à un an. Ces améliorations du NRI résultent en grande partie de l’amélioration de la classification chez ceux qui n’ont pas bénéficié d’un TAVI. En ce qui a trait à la mortalité à un an, il y a eu une amélioration significative de l’IDI (IDI, 0,003; P < 0,001) avec l’indicateur de fragilité ACG de Johns Hopkins. Cette amélioration de la performance résultait d’une augmentation de la probabilité moyenne de TAVI chez les personnes ayant vécu l’événement.

Conclusion

L’évaluation préopératoire de la fragilité peut ajouter une certaine valeur prédictive aux issues cliniques suivant une procédure de TAVI. L’utilisation d’instruments d’évaluation de la fragilité développés à partir de données administratives peut apporter des améliorations mineures mais significatives pour l’ajustement de risque lors de l’évaluation des hôpitaux en fonction de certaines issues cliniques.

Methods

Study design and data sources

We conducted a historical cohort study in Ontario, the largest province in Canada. All Ontario residents receive publicly funded universal medical coverage financed by the Ontario Ministry of Health (MOH). For this study, we used data from the CorHealth Ontario TAVI registry to identify patients who underwent TAVI at designated hospital centres. The CorHealth TAVI registry contains demographic, comorbidity, and procedural data. Linked administrative databases also included the Continuing Care Reporting System, Canadian Institute for Health Information Discharge Abstract Database, Home Care Database, National Ambulatory Care Reporting System, Ontario Drug Benefit Claims, Ontario Health Insurance Plan Claims Database, Same Day Surgery Database, Registered Persons Database, Assistive Devices Program, Ontario Case Costing Initiative, Ontario Mental Health Reporting System, and Ontario Census Area Profiles. These data sets were linked using unique encoded identifiers and analyzed at ICES (Toronto, ON, Canada).Footnote 1 The use of the data in this project is authorized under Section 45 of Ontario’s Personal Health Information Protection Act and did not require review by a Research Ethics Board. Parts of this material are based on data and/or information from the Canadian Drug Product Database and Data Extract, compiled and provided by Health Canada,Footnote 2 and used by ICES with the permission of the Minister of Health of Canada (2017).

Study cohort

In Ontario, Canada, adults aged 65 yr and older are eligible for the Ontario Drug Benefit program. To consider the effects and costs of medication use, we included patients aged 66 or older who underwent a TAVI procedure in Ontario, Canada from 1 April 2012 to 31 March 2018. For patients who underwent repeat TAVI procedures during the time frame (N = 18), we excluded the second procedure from the data set. Patients without a valid personal identification number were also excluded (N = 63).

Administrative database frailty instruments

We used two administrative database frailty instruments to assign frailty status based on preprocedural patient characteristics reported within the database over a two-year period—the Johns Hopkins ACG System Version 10.0 frailty indicator and the HFRS. The Johns Hopkins ACG frailty indicator is a binary variable that dichotomizes patients as frail and nonfrail based on 12 clusters of frailty-defining diagnoses.16 The Johns Hopkins ACG frailty indicator is a proprietary index. Thus, specific diagnostic codes used to assign the clusters are not publicly available. The Johns Hopkins ACG frailty indicator has been used to identify frailty in patients undergoing TAVI and other surgical populations.16 The HFRS is a frailty score developed and validated based on more than 100 International Statistical Classification of Diseases 10th Revision (ICD-10) diagnostic codes.[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. https://doi.org/10.1016/s0140-6736(18)30668-8

            ") Each diagnostic code is assigned a score to calculate the HFRS. The HFRS score can be used to dichotomize patients as frail (> 5 points) and nonfrail (≤ 5 points).[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. 
              https://doi.org/10.1016/s0140-6736(18)30668-8
              
            ") The HFRS can also categorize patients as high risk (> 15 points), intermediate risk (> 5–15 points) and low risk (≤ 5 points) of frailty.[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. 
              https://doi.org/10.1016/s0140-6736(18)30668-8
              
            ") The HFRS has been validated in older people (≥ 75 yr) with these cut-offs.[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. 
              https://doi.org/10.1016/s0140-6736(18)30668-8
              
            ") To facilitate comparison between the two instruments, this study focused on the dichotomous measure of the HFRS. Both the Johns Hopkins ACG frailty indicator and HFRS have been used as tools to identify high-risk TAVI patients in recent research.[15](/article/10.1007/s12630-022-02354-6#ref-CR15 "Sami F, Ranka S, Shah A, Torres C, Villablanca P. Impact of frailty on outcomes in patients undergoing transcatheter aortic valve replacement: a report from national inpatient sample. J Am Coll Cardiol 2020; 75: 1487. 
              https://doi.org/10.1016/S0735-1097(20)32114-8
              
            "), [17](/article/10.1007/s12630-022-02354-6#ref-CR17 "Malik AH, Yandrapalli S, Zaid S, et al. Impact of frailty on mortality, readmissions, and resource utilization after TAVI. Am J Cardiol 2020; 127: 120–7. 
              https://doi.org/10.1016/j.amjcard.2020.03.047
              
            ") The domains of the HFRS are shown in the Electronic Supplementary Material (ESM) eAppendix 1.

Covariates

We collected demographic data, including age, sex, rural residence, and neighbourhood income quintile, from the Registered Persons Database. Comorbidities, including myocardial infarction, ischemic heart disease, atrial fibrillation, peripheral vascular disease, cerebrovascular disease, cancer, and dialysis, were identified based on ICD-10 diagnostic codes from the Discharge Abstract Database and Ontario Health Insurance Plan Claims Database fee codes in the three years preceding TAVI. Cardiac history, including previous coronary artery bypass graft surgery, valve surgery, percutaneous coronary intervention, valve-in-valve surgery, permanent pacemaker, and implantable cardioverter defibrillator were identified from the Discharge Abstract Database and Ontario Health Insurance Plan Claims Database in the 20 years preceding TAVI. Heart failure, chronic obstructive pulmonary disease, hypertension, diabetes mellitus, dyslipidemia, and dementia were identified based on ICES-validated algorithms.[18](/article/10.1007/s12630-022-02354-6#ref-CR18 "Udell JA, Koh M, Qiu F, et al. Outcomes of women and men with acute coronary syndrome treated with and without percutaneous coronary revascularization. J Am Heart Assoc. 2017; 6(1): 1–10. https://doi.org/10.1161/JAHA.116.004319

            "),[19](/article/10.1007/s12630-022-02354-6#ref-CR19 "Jaakkimainen RS, Bronskill SE, Tierney MC, et al. Identification of physician-diagnosed Alzheimer’s disease and related dementias in population-based administrative data: a validation study using family physicians’ electronic medical records. J Alzheimers Dis 2016; 54: 337–49. 
              https://doi.org/10.3233/jad-160105
              
            ") Preprocedural and procedural data, including serum creatinine, anesthesia, hemoglobin, transvalvular gradient, the Society of Thoracic Surgeons score, access site, and valve type were obtained from the CorHealth database (data sources and codes are available in ESM eTable 1).

Clinical outcomes

The primary clinical outcome was death from any cause at one year and secondary outcomes were in-hospital death and one-year rehospitalization. In-hospital deaths during the index hospitalization were identified from the Discharge Abstract Database. Deaths after discharge were identified from the Registered Persons Database. Rehospitalization was identified from the National Ambulatory Care Reporting System and Discharge Abstract Database.

Cost outcomes

The primary cost outcome was healthcare costs incurred by the Ontario Ministry of Health and Long-Term Care over a one year period from the first day of the index hospitalization for the TAVI procedure and ending 365 days later. The secondary outcome was high-cost patients defined as being in the top 5% of total one-year healthcare cost.[20](/article/10.1007/s12630-022-02354-6#ref-CR20 "Wammes JJ, van der Wees PJ, Tanke MA, Westert GP, Jeurissen PP. Systematic review of high-cost patients’ characteristics and healthcare utilisation. BMJ Open 2018; 8: e023113. https://doi.org/10.1136/bmjopen-2018-023113

            ") Healthcare costs included the cost of inpatient hospitalization, emergency visits, same-day surgeries and other ambulatory treatments, physician services, diagnostic and laboratory services, and outpatient prescriptions covered by public funding. All TAVI-related costs and non-TAVI-related costs were included. All costs were converted to 2017 CAD.

Healthcare costs from the perspective of the Ontario MOH were calculated based on person-level healthcare use and captured in administrative claims and billing data.[21](/article/10.1007/s12630-022-02354-6#ref-CR21 "Wodchis WP, Bushmeneva K, Nikitovic M, Mckillop I. Guidelines on person-level costing using administrative databases in Ontario, 2013. Available from URL: https://tspace.library.utoronto.ca/handle/1807/87373

             (accessed July 2022).") Inpatient costs were estimated based on the resource intensity weight using information including the case mix group, age, comorbidity level, flagged intervention, and out-of-hospital intervention.[21](/article/10.1007/s12630-022-02354-6#ref-CR21 "Wodchis WP, Bushmeneva K, Nikitovic M, Mckillop I. Guidelines on person-level costing using administrative databases in Ontario, 2013. Available from URL: 
              https://tspace.library.utoronto.ca/handle/1807/87373
              
             (accessed July 2022).") Emergency department visits, same-day surgeries, and other ambulatory treatment (i.e., dialysis and oncology) costs were similarly estimated based on a weighting factor (i.e., the Comprehensive Ambulatory Classification System weight).[21](/article/10.1007/s12630-022-02354-6#ref-CR21 "Wodchis WP, Bushmeneva K, Nikitovic M, Mckillop I. Guidelines on person-level costing using administrative databases in Ontario, 2013. Available from URL: 
              https://tspace.library.utoronto.ca/handle/1807/87373
              
             (accessed July 2022).") The cost of physician services, diagnostic services, and outpatient medications were estimated based on the amount reimbursed by the province.[21](/article/10.1007/s12630-022-02354-6#ref-CR21 "Wodchis WP, Bushmeneva K, Nikitovic M, Mckillop I. Guidelines on person-level costing using administrative databases in Ontario, 2013. Available from URL: 
              https://tspace.library.utoronto.ca/handle/1807/87373
              
             (accessed July 2022).") Details on person-level costing using administrative databases have been described by Wodchis _et al_.[21](/article/10.1007/s12630-022-02354-6#ref-CR21 "Wodchis WP, Bushmeneva K, Nikitovic M, Mckillop I. Guidelines on person-level costing using administrative databases in Ontario, 2013. Available from URL: 
              https://tspace.library.utoronto.ca/handle/1807/87373
              
             (accessed July 2022).")

Statistical approaches

We compared characteristics between groups with and without frailty for both the Johns Hopkins ACG frailty indicator and HFRS. Continuous variables are presented as mean (standard deviation [SD]) and median [interquartile range (IQR)]. Categorical variables are presented as proportions. We calculated standardized differences since they are less sensitive to sample sizes and considered values greater than 0.1 meaningful.[22](/article/10.1007/s12630-022-02354-6#ref-CR22 "Austin PC. Using the standardized difference to compare the prevalence of a binary variable between two groups in observational research. Commun Stat Simul Comput 2009; 38: 1228–34. https://doi.org/10.1080/03610910902859574

            ") We also report _P_ values for single variable differences between patients with and without frailty. Both mean and median healthcare costs are presented, along with tests of significance for cost differences between patients with and without frailty.[23](/article/10.1007/s12630-022-02354-6#ref-CR23 "Mihaylova B, Briggs A, O’Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Econ 2011; 20: 897–916. 
              https://doi.org/10.1002/hec.1653
              
            "), [24](/article/10.1007/s12630-022-02354-6#ref-CR24 "Blough DK, Ramsey SD. Using generalized linear models to assess medical care costs. Health Serv Outcomes Res Methodol 2000; 1: 185–202.")

We estimated the prevalence of frailty by dividing the number of patients with frailty by the total number of patients. We calculated the Kappa statistic to assess the agreement between the Johns Hopkins ACG frailty indicator and HFRS.25 To better understand frailty categorization within subgroups, we also classified patients as frail and nonfrail according to their demographic and clinical characteristics.

To account for the two-level hierarchical structure of database (i.e., patients nested within hospitals), multilevel modelling was applied. For clinical outcomes, a hierarchical logistic regression model accounting for clustering within hospitals, and adjusting for demographics, baseline comorbidities, cardiac history, and preprocedural characteristics was fitted as the reference model.[26](/article/10.1007/s12630-022-02354-6#ref-CR26 "Austin PC, Tu JV, Alter DA. Comparing hierarchical modeling with traditional logistic regression analysis among patients hospitalized with acute myocardial infarction: should we be analyzing cardiovascular outcomes data differently? Am Heart J 2003; 145: 27–35. https://doi.org/10.1067/mhj.2003.23

            ") The reference model was prespecified, including variables selected based on their clinical significance for directly influencing the outcomes of interests.[27](/article/10.1007/s12630-022-02354-6#ref-CR27 "Wijeysundera HC, Li L, Braga V, et al. Drivers of healthcare costs associated with the episode of care for surgical aortic valve replacement versus transcatheter aortic valve implantation. Open Heart 2016; 3: e000468. 
              https://doi.org/10.1136/openhrt-2016-000468
              
            ") Covariates were treated as either dichotomous or categorical variables and odds ratios were reported. Frailty instruments were added to the reference model individually to test the significance of frailty in predicting odds of death at one year, in-hospital death, and one-year rehospitalization. The models were:

1. Model 0: The reference prediction model without frailty instruments. The reference model adjusted all baseline covariates and procedural characteristics described above and was consistent with previous studies of this patient population.
1. Model A: Model 0 + the Johns Hopkins ACG frailty indicator.
1. Model B: Model 0 + the HFRS.
1. Model C: Model 0 + the Johns Hopkins ACG frailty indicator + the HFRS.

Multicollinearity in the models was examined using the variance inflation factor, and we considered a variance inflation factor > 5 as an indicator of multicollinearity.28 Sample size was examined based on the number of events per variable.[29](/article/10.1007/s12630-022-02354-6#ref-CR29 "Cundill B, Alexander ND. Sample size calculations for skewed distributions. BMC Med Res Methodol 2015; 15: 28. https://doi.org/10.1186/s12874-015-0023-0

            "),[30](/article/10.1007/s12630-022-02354-6#ref-CR30 "Van Smeden M, de Groot JA, Moons KG, et al. No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Med Res Methodol 2016; 16: 1–12. 
              https://doi.org/10.1186/s12874-016-0267-3
              
            ") Missing data were found in anemia diagnosis (37%), creatinine measures (38%), and anesthesia (2%). Participants with missing data were flagged as “missing” and included as a category.[27](/article/10.1007/s12630-022-02354-6#ref-CR27 "Wijeysundera HC, Li L, Braga V, et al. Drivers of healthcare costs associated with the episode of care for surgical aortic valve replacement versus transcatheter aortic valve implantation. Open Heart 2016; 3: e000468. 
              https://doi.org/10.1136/openhrt-2016-000468
              
            ")

To compare the reference model and each model with frailty instruments, the following performance statistics were reported: likelihood ratio, chi-square test, c-statistic, Akaike information criterion (AIC), Bayesian information criterion (BIC), integrated discrimination improvement (IDI), net reclassification improvement (NRI), and calibration.[31](/article/10.1007/s12630-022-02354-6#ref-CR31 "Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. https://doi.org/10.1002/sim.2929

            "), [32](/article/10.1007/s12630-022-02354-6#ref-CR32 "Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014; 33: 517–35. 
              https://doi.org/10.1002/sim.5941
              
            ") Discrimination was assessed using the c-statistic, which is the probability that the model yields a higher risk for individuals who experience the outcome than for those who do not experience the outcome.[31](/article/10.1007/s12630-022-02354-6#ref-CR31 "Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. 
              https://doi.org/10.1002/sim.2929
              
            "), [33](/article/10.1007/s12630-022-02354-6#ref-CR33 "Pencina MJ, D’Agostino RB Sr. Evaluating discrimination of risk prediction models: the C statistic. JAMA 2015; 314: 1063–4. 
              https://doi.org/10.1001/jama.2015.11082
              
            ") A larger c-statistic indicates improved discrimination.[31](/article/10.1007/s12630-022-02354-6#ref-CR31 "Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. 
              https://doi.org/10.1002/sim.2929
              
            "), [33](/article/10.1007/s12630-022-02354-6#ref-CR33 "Pencina MJ, D’Agostino RB Sr. Evaluating discrimination of risk prediction models: the C statistic. JAMA 2015; 314: 1063–4. 
              https://doi.org/10.1001/jama.2015.11082
              
            ") The improvement in discrimination with the addition of frailty instruments was assessed by the change in the c-statistic and the DeLong test.[34](/article/10.1007/s12630-022-02354-6#ref-CR34 "DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–45. 
              https://doi.org/10.2307/2531595
              
            ") The AIC and BIC allow for simultaneous comparison of multiple models, with more negative values indicating a model better fits the data (goodness-of-fit).[35](/article/10.1007/s12630-022-02354-6#ref-CR35 "Neath AA, Cavanaugh JE. The Bayesian information criterion: background, derivation, and applications. Wiley Interdiscip Rev Comput Stat 2012; 4: 199–203. 
              https://doi.org/10.1002/wics.199
              
            "), [36](/article/10.1007/s12630-022-02354-6#ref-CR36 "Vrieze SI. Model selection and psychological theory: a discussion of the differences between the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Psychol Methods 2014; 17: 228–43. 
              https://doi.org/10.1037/a0027127
              
            ") The IDI and NRI have been proposed to assess improvement in model performance offered by a new predictor.[31](/article/10.1007/s12630-022-02354-6#ref-CR31 "Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. 
              https://doi.org/10.1002/sim.2929
              
            "), [37](/article/10.1007/s12630-022-02354-6#ref-CR37 "Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–38. 
              https://doi.org/10.1097/ede.0b013e3181c30fb2
              
            ") The IDI assesses the ability of a new model to improve average sensitivity without sacrificing average specificity, with more positive values indicating improved performance.[31](/article/10.1007/s12630-022-02354-6#ref-CR31 "Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. 
              https://doi.org/10.1002/sim.2929
              
            "), [37](/article/10.1007/s12630-022-02354-6#ref-CR37 "Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–38. 
              https://doi.org/10.1097/ede.0b013e3181c30fb2
              
            ") The NRI is calculated based on the number of patients correctly reclassified by adding a new predictor minus the number incorrectly reclassified, with larger values indicating improved net prediction.[31](/article/10.1007/s12630-022-02354-6#ref-CR31 "Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. 
              https://doi.org/10.1002/sim.2929
              
            "), [37](/article/10.1007/s12630-022-02354-6#ref-CR37 "Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–38. 
              https://doi.org/10.1097/ede.0b013e3181c30fb2
              
            ") Continuous NRIs were reported. Calibration refers to the degree of agreement between observed and predicted probabilities and was examined using calibration plots and the Hosmer–Lemeshow test.[32](/article/10.1007/s12630-022-02354-6#ref-CR32 "Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014; 33: 517–35. 
              https://doi.org/10.1002/sim.5941
              
            ")

For cost outcomes, we fitted a hierarchical generalized linear model with a logarithmic link and gamma distribution to examine the performance of frailty instruments in predicting one year costs adjusting for prespecified covariates.[38](/article/10.1007/s12630-022-02354-6#ref-CR38 "Austin PC, Ghali WA, Tu JV. A comparison of several regression models for analysing cost of CABG surgery. Stat Med 2003; 22: 2799–815. https://doi.org/10.1002/sim.1442

            ") The model addresses the heavily right-skewed cost distribution.[23](/article/10.1007/s12630-022-02354-6#ref-CR23 "Mihaylova B, Briggs A, O’Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Econ 2011; 20: 897–916. 
              https://doi.org/10.1002/hec.1653
              
            "), [24](/article/10.1007/s12630-022-02354-6#ref-CR24 "Blough DK, Ramsey SD. Using generalized linear models to assess medical care costs. Health Serv Outcomes Res Methodol 2000; 1: 185–202.") The exponential of the coefficient represents the rate ratio (RR). The RR is interpreted as the proportional increase in one-year costs when changing one unit in the predictor. Predictive performance was compared based on bias and mean squared error. Differences in adjusted absolute costs between patients with and without frailty were estimated using the method of recycled predictions[39](/article/10.1007/s12630-022-02354-6#ref-CR39 "Muratov S, Lee J, Holbrook A, et al. Incremental healthcare utilisation and costs among new senior high to cost users in Ontario, Canada: A retrospective matched cohort study. BMJ Open. 2019; 9(10): 1–9. 
              https://doi.org/10.1136/bmjopen-2018-028637
              
            ") (details are presented in eAppendix 2).[38](/article/10.1007/s12630-022-02354-6#ref-CR38 "Austin PC, Ghali WA, Tu JV. A comparison of several regression models for analysing cost of CABG surgery. Stat Med 2003; 22: 2799–815. 
              https://doi.org/10.1002/sim.1442
              
            "),[39](/article/10.1007/s12630-022-02354-6#ref-CR39 "Muratov S, Lee J, Holbrook A, et al. Incremental healthcare utilisation and costs among new senior high to cost users in Ontario, Canada: A retrospective matched cohort study. BMJ Open. 2019; 9(10): 1–9. 
              https://doi.org/10.1136/bmjopen-2018-028637
              
            ") We also fitted logistic regression models to predict high-cost patients. Predictive accuracy was compared between the reference model and each model with frailty instruments. Predictive performance statistics, including c-statistic, AIC, BIC, IDI, and net NRI, were reported.[31](/article/10.1007/s12630-022-02354-6#ref-CR31 "Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. 
              https://doi.org/10.1002/sim.2929
              
            "), [40](/article/10.1007/s12630-022-02354-6#ref-CR40 "Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29–36. 
              https://doi.org/10.1148/radiology.143.1.7063747
              
            "), [41](/article/10.1007/s12630-022-02354-6#ref-CR41 "Tjur T. Coefficients of determination in logistic regression models - a new proposal: the coefficient of discrimination. Am Stat 2009; 63: 366–72. 
              https://doi.org/10.1198/tast.2009.08210
              
            ") Adverse outcomes including death, stroke, bleeding, permanent pacemaker implantation, and rehospitalization between high-cost and non-high-cost patients were compared using chi-square analysis.

Sensitivity analyses

A Cox subdistributional model, with sandwich variance estimators to account for clustering by institution, was built to estimate the effect of frailty on the hazard of rehospitalization with death as a competing risk. Patients who die cannot be hospitalized after the date of death and we accounted for death as a competing risk.[42](/article/10.1007/s12630-022-02354-6#ref-CR42 "Austin PC, Lee DS, Fine JP. Introduction to the analysis of survival data in the presence of competing risks. Circulation 2016; 133: 601–9. https://doi.org/10.1161/circulationaha.115.017719

            "), [43](/article/10.1007/s12630-022-02354-6#ref-CR43 "Lau B, Cole SR, Gange SJ. Competing risk regression models for epidemiologic data. Am J Epidemiol 2009; 170: 244–56. 
              https://doi.org/10.1093/aje/kwp107
              
            ")

We used the Johns Hopkins ACG System Version 10.0 to obtain the Johns Hopkins ACG frailty indicator. SAS Enterprise 7.1 (SAS Institute Inc., Cary, NC, USA) was used for all analyses; we considered P values < 0.05 significant.

Results

Characteristics of the cohort

A total of 3,848 patients (45% females; 87% transfemoral approach) were included (ESM eTable 2). Participants with and without frailty had marked differences in their demographic and procedural characteristics (Table 1). At one-year follow-up, 554 (14%) deaths occurred. In-hospital death and 12-month rehospitalization occurred in 163 (4%) and 2,566 (66%) cases, respectively. High-cost patients were defined as those in the top 5% of one-year cost, which was equal to or greater than CAD 133,941.

Predictive performance of frailty instruments

Frailty instruments did not improve discrimination significantly, with the exception of the Johns Hopkins ACG frailty indicator with rehospitalization (∆ c-statistic, 0.006; DeLong P = 0.04) and the Johns Hopkins ACG frailty indicator and HFRS with rehospitalization (∆ c-statistic, 0.008; DeLong P = 0.02) (Table 2).

Compared with the reference model, both the Johns Hopkins ACG frailty indicator and HFRS significantly improved NRI for one-year mortality and rehospitalization (Table 2). These improvements in NRI largely resulted from classification improvement among those who did not experience the event, at the expense of poorer classification for those who experienced the event. For example, when adding the Johns Hopkins ACG frailty indicator to the model predicting one-year mortality, a greater percentage of patients without the event (78%) moved down in predicted probability and a smaller percentage (22%) moved up, resulting in an NRI among the non-event group of 56%. In contrast, among the event group, a greater percentage moved down in predicted probability than moved up (70% vs 23%) resulting in an NRI among the event group of -40%. (ESM eTable 3). Neither the Johns Hopkins ACG frailty indicator nor the HFRS improved NRI for classifying high-cost patients (Table 2). Nevertheless, the inclusion of both the Johns Hopkins ACG frailty indicator and the HFRS significantly improved the NRI for classifying high-cost patients (NRI, 0.255; P < 0.001) (Table 2).

With one-year mortality, there was a significant improvement in IDI (IDI, 0.003; P < 0.001) with the Johns Hopkins ACG frailty indicator (Table 2). This improvement in performance resulted from an increase in the mean probability of the event among those who experienced the event. The mean probability of the event among those without the event did not change (ESM eTable 4). With rehospitalization, both the Johns Hopkins ACG frailty indicator (IDI, 0.005; P < 0.001) and the HFRS (IDI, 0.003; P < 0.001) significantly improved the IDI (Table 2) owing to small increases in the mean probability of rehospitalization among those rehospitalized and decreased mean probability among those who were not rehospitalized (ESM eTable 4).

Calibration plots and results of Hosmer–Lemeshow tests are shown in ESM eFigure and eTable 5.

Agreement between the two frailty instruments

A total of 863/3,848 (22.4%) patients were identified as frail using the Johns Hopkins ACG frailty indicator and 865/3,848 (22.5%) were identified as frail using the HFRS (Figure). Although the agreement between the Johns Hopkins ACG frailty indicator and the HFRS was fair (Kappa statistic = 0.322; 95% confidence interval [CI], 0.288 to 0.357), each classified different subgroups as frail. Of the 863 patients identified as frail using the Johns Hopkins ACG frailty indicator, 453 were not frail according to the HFRS; of the 865 patients identified as frail using the HFRS, 455 were not frail according to the Johns Hopkins ACG frailty indicator. Comparison of frailty categorization within subgroups of demographic and clinical variables indicated fair agreement between the frailty instruments. Kappa statistics of agreement between the frailty instruments ranged from 0.109 (in the dementia subgroup) to 0.400 (in the subgroup of age 66–70 yr) (ESM eTable 6).

Figure

The alternative text for this image may have been generated using AI.

Full size image

Agreement between the HFRS and Johns Hopkins ACG frailty indicator

ACG = adjusted clinical groups; HFRS = Hospital Frailty Risk Score

Frailty instruments and clinical outcomes

Patients with and without frailty had marked differences in one-year mortality and rehospitalization (ESM eTable 7). Adjusting for demographics and comorbidities, both the Johns Hopkins ACG frailty indicator and HFRS were significantly associated with one-year mortality and rehospitalization at one year (ESM eTables 8 and 9). With death as a competing risk, the subdistribution hazard models did not change the associations between frailty instruments and rehospitalization at one year (ESM eTable 10).

Frailty instruments and cost outcomes

Costs of patients by frailty status are shown in Table 3. Using the Johns Hopkins ACG frailty indicator, the mean (SD) one-year costs were CAD 66,243 (50,469) in 863 patients with frailty compared with CAD 55,583 (42,607) in 2,985 patients without frailty (median [IQR], CAD 50,095 [36,676–79,475] vs CAD 42,736 [32,363–63,784]). Using the HFRS, the mean (SD) one-year costs were CAD 71,775 (56,036) in 865 patients with frailty compared with CAD 53,971 (39,966) in 2,983 patients without frailty (median [IQR], CAD 54,713 [37,702–82,613]) vs CAD 42,055 [32,187–61,632]). Differences in adjusted absolute costs between patients with and without frailty are reported in ESM eTable 11.

Adjusting for demographics and baseline comorbidities, both the Johns Hopkins ACG frailty indicator (RR, 1.13; 95% CI, 1.06 to 1.20; ESM eTable 8) and the HFRS (RR, 1.14; 95% CI, 1.07 to 1.21; ESM eTable 9) were significantly associated with increased one-year costs. Compared with non-high-cost patients, high-cost patients had more adverse outcomes (ESM eTable 12–14). Adjusting for demographics and baseline comorbidities, the Johns Hopkins ACG frailty indicator (OR, 1.44; 95% CI, 0.99 to 2.08) was not significantly associated with being a high-cost patient (ESM eTable 8). The HFRS (OR, 1.81; 95% CI, 1.27 to 2.58) was significantly associated with being a high-cost patient (ESM eTable 9).

Discussion

Principal findings

Drawing on data from administrative databases in Ontario, our study found fair agreement between the two database-derived frailty instruments, the Johns Hopkins ACG frailty indicator and the HFRS. Nevertheless, despite similar proportions of patients with frailty, the instruments identified a different subset of individuals as living with frailty. Adjusting for demographics and baseline comorbidities, hierarchical regression analysis showed that both the Johns Hopkins ACG frailty indicator and the HFRS were significantly associated with one-year mortality and rehospitalization following TAVI. For predicting one-year mortality, the addition of each frailty instrument to the reference model significantly improved the NRI, and this improvement mainly resulted from improved classification among those without the event, with considerable declines among those with the event. Analysis of cost data from the Ontario TAVI cohort showed that patients with frailty incurred significantly higher one-year healthcare costs. The HFRS was a significant predictor of high-cost patients but neither instrument improved model performance in predicting high-cost patients.

Comparisons with other studies

This study adds to prior population-based findings of predictive performance of the Johns Hopkins ACG frailty indicator and HFRS in TAVI patients.[15](/article/10.1007/s12630-022-02354-6#ref-CR15 "Sami F, Ranka S, Shah A, Torres C, Villablanca P. Impact of frailty on outcomes in patients undergoing transcatheter aortic valve replacement: a report from national inpatient sample. J Am Coll Cardiol 2020; 75: 1487. https://doi.org/10.1016/S0735-1097(20)32114-8

            "), [17](/article/10.1007/s12630-022-02354-6#ref-CR17 "Malik AH, Yandrapalli S, Zaid S, et al. Impact of frailty on mortality, readmissions, and resource utilization after TAVI. Am J Cardiol 2020; 127: 120–7. 
              https://doi.org/10.1016/j.amjcard.2020.03.047
              
            ") Kundi _et al_.[44](/article/10.1007/s12630-022-02354-6#ref-CR44 "Kundi H, Valsdottir LR, Popma JJ, et al. Impact of a claims-based frailty indicator on the prediction of long-term mortality after transcatheter aortic valve in Medicare beneficiaries. Circ Cardiovasc Qual Outcomes 2018; 11: 1–8. 
              https://doi.org/10.1161/circoutcomes.118.005048
              
            ") characterized the prognostic importance of the Johns Hopkins ACG frailty indicator in 52,338 patients undergoing TAVI and the addition of this frailty marker improved four-year mortality prediction (c-statistic, 0.700; IDI, 0.019; _P_ < 0.001). Further to this, Kundi _et al_.[14](/article/10.1007/s12630-022-02354-6#ref-CR14 "Kundi H, Popma JJ, Reynolds MR, et al. Frailty and related outcomes in patients undergoing transcatheter valve therapies in a nationwide cohort. Eur Heart J 2019; 40: 2231–9. 
              https://doi.org/10.1093/eurheartj/ehz187
              
            ") used the HFRS to identify frailty prevalence and related outcomes in 28,531 patients undergoing TAVI in the USA. They found 13,593 (48%) patients with frailty and an IDI of 0.024 (_P_ < 0.001) with addition of the HFRS to a reference model predicting all-cause one-year mortality.[14](/article/10.1007/s12630-022-02354-6#ref-CR14 "Kundi H, Popma JJ, Reynolds MR, et al. Frailty and related outcomes in patients undergoing transcatheter valve therapies in a nationwide cohort. Eur Heart J 2019; 40: 2231–9. 
              https://doi.org/10.1093/eurheartj/ehz187
              
            ") Although these studies also examined the predictive performance of frailty instruments among TAVI patients, the results of these studies may not be comparable with our results. Prior studies had a less inclusive reference model, so their reference models are not directly comparable with our reference model. Further, their patient populations came from the USA, while our study used data from a Canadian TAVI registry.

Clinical implications

Our study shows that, even though the Johns Hopkins ACG frailty indicator and HFRS were developed based on the same frailty concept, these frailty instruments should not be considered interchangeable. While our study found similar proportions of patients with frailty and a fair agreement between the Johns Hopkins ACG frailty indicator and the HFRS, each identified different subgroups as frail. These findings are consistent with previous research that showed a wide range of agreement between frailty instruments.[45](/article/10.1007/s12630-022-02354-6#ref-CR45 "Aguayo GA, Donneau AF, Vaillant MT, et al. Practice of epidemiology agreement between 35 published frailty scores in the general population. Am J Epidemiol 2017; 186: 420–34. https://doi.org/10.1093/aje/kwx061

            ") An important reason might be the Johns Hopkins ACG frailty indicator and the HFRS quantify different health deficits. The Johns Hopkins ACG frailty indicator assesses malnutrition, dementia, impaired vision, decubitus ulcer, incontinence of urine, loss of weight, poverty, barriers to access to care, difficulty in walking, and falls, whereas the HFRS includes a total of 109 health conditions.[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. 
              https://doi.org/10.1016/s0140-6736(18)30668-8
              
            ") Although some deficits are included in both frailty instruments, such as dementia, unspecified fall, blindness and low vision, decubitus ulcer, falling from bed, and falling on and down stairs and steps, most deficits included in the HFRS are not included in the Johns Hopkins ACG frailty indicator.[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. 
              https://doi.org/10.1016/s0140-6736(18)30668-8
              
            ") Moreover, the weights assigned to each deficit in calculating the frailty score might differ. Of the 109 health conditions, dementia in Alzheimer’s disease, hemiplegia, Alzheimer’s disease, sequelae of cerebrovascular disease, and other symptoms and signs involving the nervous and musculoskeletal systems are the five deficits assigned with highest weights.[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. 
              https://doi.org/10.1016/s0140-6736(18)30668-8
              
            ") Due to the proprietary nature of the Johns Hopkins ACG system, specific codes are not publicly available so it is not possible to compare the weighting between the two codes. Notably, both the Johns Hopkins ACG frailty indicator and HFRS were developed based on the frailty index,[46](/article/10.1007/s12630-022-02354-6#ref-CR46 "Searle SD, Mitnitski A, Gahbauer EA, Gill TM, Rockwood K. A standard procedure for creating a frailty index. BMC Geriatr 2008; 8: 1–10. 
              https://doi.org/10.1186/1471-2318-8-24
              
            ") which may not align with the Fried frailty phenotype[1](/article/10.1007/s12630-022-02354-6#ref-CR1 "Fried LP, Tangen CM, Walston J, et al. Frailty in older adults: evidence for a phenotype. J Gerontol A Biol Sci Med Sci 2001; 56: M146–56. 
              https://doi.org/10.1093/gerona/56.3.m146
              
            ") that considers frailty by physical characteristics (shrinking, low strength, exhaustion, slow walking speed, and low physical activity).

Our historical cohort study findings suggest that, from a prediction perspective, frailty instruments are associated with TAVI patient outcomes, and adding frailty instruments to a reference model may achieve better model fit, classification, and integrated discrimination in predicting patient outcomes. Nevertheless, due to a lack of ideal methods to evaluate the added value of new biomarkers, the clinical use for these frailty instruments to improve the prediction of clinical outcomes remains unclear and needs further research. Although both the Johns Hopkins ACG frailty indicator and HFRS led to improved NRI and IDI when predicting one-year mortality and rehospitalization, improvements in NRI were at the expense of poorer classification for those who experienced the event, suggesting that sensitivity may be sacrificed to improve specificity. Moreover, magnitudes of improvement in IDI were small. Even if the Johns Hopkins ACG frailty indicator and HFRS performed slightly better than the prespecified reference model, the utility of adding these instruments to a well-specified risk adjustment model is unclear. Further research may use decision curve analysis across a range of threshold probabilities to quantify the net benefit of using these frailty instruments in a clinical setting.

Our findings also contribute to the emerging literature on frailty and healthcare costs of TAVI and suggest that the association of frailty and healthcare costs might be largely determined by the higher rates of adverse outcomes in patients with frailty. In analysis of healthcare costs, we found a marked cost difference between patients with and without frailty. Adjusting for patient and procedure characteristics, our study found that the HFRS was significantly associated with increased healthcare costs.

While both the Johns Hopkins ACG frailty indicator and HFRS improved prediction of one-year mortality and rehospitalization compared with the prespecified reference model, neither the Johns Hopkins ACG frailty indicator nor the HFRS improved performance in predicting being a high-cost patient. These findings suggest that the performance of frailty instruments vary when predicting different outcomes. Therefore, clinicians or risk-adjustment modelers may choose frailty instruments based on predictive perfromance for outcomes of interest.

Study limitations

Our study has some important limitations. First, the HFRS was developed and validated based on a cohort of patients aged 75 yr and older with elective, nonelective, and day-case admissions to hospitals in the UK,[11](/article/10.1007/s12630-022-02354-6#ref-CR11 "Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. https://doi.org/10.1016/s0140-6736(18)30668-8

            ") while our study cohort was restricted to patients aged 66 yr and older who underwent a TAVI procedure. Although most frailty instruments were developed in a patient population over the age of 75 yr, previous literature has examined these instruments in a younger (≥ 65 yr) population.[14](/article/10.1007/s12630-022-02354-6#ref-CR14 "Kundi H, Popma JJ, Reynolds MR, et al. Frailty and related outcomes in patients undergoing transcatheter valve therapies in a nationwide cohort. Eur Heart J 2019; 40: 2231–9. 
              https://doi.org/10.1093/eurheartj/ehz187
              
            ") Second, we only tested the performance of frailty instruments in predicting outcomes for up to one year. Long-term outcomes were not examined. Third, restricted by the availability of data, we were unable to adjust for all variables that have been shown to affect TAVI outcomes, such as the New York Heart Association classification and left ventricular ejection fraction.[47](/article/10.1007/s12630-022-02354-6#ref-CR47 "Henning KA, Ravindran M, Qiu F, et al. Impact of procedural capacity on transcatheter aortic valve replacement wait times and outcomes: A study of regional variation in Ontario, Canada. Open Heart 2020; 7: e001241. 
              https://doi.org/10.1136/openhrt-2020-001241
              
            "), [48](/article/10.1007/s12630-022-02354-6#ref-CR48 "Czarnecki A, Qiu F, Henning KA, et al. Comparison of 1-year pre- and post-transcatheter aortic valve replacement hospitalization rates: a population-based cohort study. Can J Cardiol 2020; 36: 1616–23. 
              https://doi.org/10.1016/j.cjca.2020.01.009
              
            ") Including these variables may give a more inclusive reference model and lead to lower prognostic impacts of frailty. We were unable to distinguish costs associated with specific interventions or episodes of care. Fourth, given the limitations underlying linked health administrative data, our study may be subject to possible inaccuracies in administrative database codes. For example, there might be variations or errors in documentation and coding, leading to measurement errors. Fifth, for the purpose of comparing frailty instruments, our study only focused on the predictive value of binary frailty measures. Use of a binary variable may lose information offered by a continuous variable. Sixth, due to resource limitations, we were unable to study the performance of other frailty instruments, such as the Preoperative Frailty Index, which was developed and validated to predict outcomes of general surgical patients.[12](/article/10.1007/s12630-022-02354-6#ref-CR12 "McIsaac DI, Wong CA, Huang A, Moloo H, van Walraven C. Derivation and validation of a generalizable preoperative frailty index using population-based health administrative data. Ann Surg 2019; 270: 102–8. 
              https://doi.org/10.1097/sla.0000000000002769
              
            ")

Conclusions

In conclusion, preoperative assessment of frailty may add some predictive value for outcomes after TAVI. Incorporating administrative database frailty instruments may provide small but significant benefits in case-mix adjustment for profiling hospitals for certain outcomes. Further validation of administrative database frailty instruments is required for this patient population.

Table 1 Characteristics of patients undergoing TAVI frail versus nonfrail assessed with administrative database frailty indices

Full size table

Table 2 Predictive performance of administrative database frailty indices compared with the reference model

Full size table

Table 3 Cost outcomes of frail versus nonfrail patients undergoing TAVI assessed with administrative database frailty indices

Full size table

Notes

ICES (formerly Institute for Clinical Evaluative Sciences). Available from URL: https://www.ices.on.ca (accessed August 2022).

References

Fried LP, Tangen CM, Walston J, et al. Frailty in older adults: evidence for a phenotype. J Gerontol A Biol Sci Med Sci 2001; 56: M146–56. https://doi.org/10.1093/gerona/56.3.m146
Article CAS PubMed Google Scholar
Afilalo J, Alexander KP, Mack MJ, et al. Frailty assessment in the cardiovascular care of older adults. J Am Coll Cardiol 2014; 63: 747–62. https://doi.org/10.1016/j.jacc.2013.09.070
Article PubMed Google Scholar
Li Z, Dawson E, Moodie J, et al. Measurement and prognosis of frail patients undergoing transcatheter aortic valve implantation: a systematic review and meta-analysis. BMJ Open 2021; 11: e040459. https://doi.org/10.1136/bmjopen-2020-040459
Article PubMed PubMed Central Google Scholar
Afilalo J, Lauck S, Kim DH, et al. Frailty in older adults undergoing aortic valve replacement the FRAILTY-AVR study. J Am Coll Cardiol 2017; 70: 689–700. https://doi.org/10.1016/j.jacc.2017.06.024
Article PubMed Google Scholar
Green P, Woglom AE, Genereux P, et al. The impact of frailty status on survival after transcatheter aortic valve replacement in older adults with severe aortic stenosis: a single-center experience. JACC Cardiovasc Interv 2012; 5: 974–81. https://doi.org/10.1016/j.jcin.2012.06.011
Article PubMed PubMed Central Google Scholar
Goldfarb M, Bendayan M, Rudski LG, et al. Cost of cardiac surgery in frail compared with nonfrail older adults. Can J Cardiol 2017; 33: 1020–6. https://doi.org/10.1016/j.cjca.2017.03.019
Article PubMed Google Scholar
Zampieri FG, Iwashyna TJ, Viglianti EM, et al. Association of frailty with short-term outcomes, organ support and resource use in critically ill patients. Intensive Care Med 2018; 44: 1512–20. https://doi.org/10.1007/s00134-018-5342-2
Article PubMed Google Scholar
Nishimura RA, Otto CM, Bonow RO, et al. 2017 AHA/ACC focused update of the 2014 AHA/ACC guideline for the management of patients with valvular heart disease: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. J Am Coll Cardiol 2017; 70: 252–89. https://doi.org/10.1016/j.jacc.2017.03.011
Article PubMed Google Scholar
Kim DH, Kim CA, Placide S, Lipsitz LA, Marcantonio E. Preoperative frailty assessment and outcomes at 6 months or later in older adults undergoing cardiac surgical procedures: a systematic review. Ann Intern Med 2016; 165: 650–60. https://doi.org/10.7326/m16-0652
Article PubMed PubMed Central Google Scholar
Dent E, Kowal P, Hoogendijk EO. Frailty measurement in research and clinical practice: a review. Eur J Intern Med 2016; 31: 3–10. https://doi.org/10.1016/j.ejim.2016.03.007
Article PubMed Google Scholar
Gilbert T, Neuburger J, Kraindler J, et al. Development and validation of a hospital frailty risk score focusing on older people in acute care settings using electronic hospital records: an observational study. Lancet 391: 1775–82. https://doi.org/10.1016/s0140-6736(18)30668-8
McIsaac DI, Wong CA, Huang A, Moloo H, van Walraven C. Derivation and validation of a generalizable preoperative frailty index using population-based health administrative data. Ann Surg 2019; 270: 102–8. https://doi.org/10.1097/sla.0000000000002769
Article PubMed Google Scholar
Alkadri J, Hage D, Nickerson LH, et al. A systematic review and meta-analysis of preoperative frailty instruments derived from electronic health data. Anesth Analg 2021; 133: 1094–1106. https://doi.org/10.1213/ane.0000000000005595
Article PubMed Google Scholar
Kundi H, Popma JJ, Reynolds MR, et al. Frailty and related outcomes in patients undergoing transcatheter valve therapies in a nationwide cohort. Eur Heart J 2019; 40: 2231–9. https://doi.org/10.1093/eurheartj/ehz187
Article PubMed PubMed Central Google Scholar
Sami F, Ranka S, Shah A, Torres C, Villablanca P. Impact of frailty on outcomes in patients undergoing transcatheter aortic valve replacement: a report from national inpatient sample. J Am Coll Cardiol 2020; 75: 1487. https://doi.org/10.1016/S0735-1097(20)32114-8
Article Google Scholar
Sternberg SA, Bentur N, Abrams C, et al. Identifying frail older people using predictive modeling. Am J Manag Care 2012; 18: 392–7.
Google Scholar
Malik AH, Yandrapalli S, Zaid S, et al. Impact of frailty on mortality, readmissions, and resource utilization after TAVI. Am J Cardiol 2020; 127: 120–7. https://doi.org/10.1016/j.amjcard.2020.03.047
Article PubMed Google Scholar
Udell JA, Koh M, Qiu F, et al. Outcomes of women and men with acute coronary syndrome treated with and without percutaneous coronary revascularization. J Am Heart Assoc. 2017; 6(1): 1–10. https://doi.org/10.1161/JAHA.116.004319
Jaakkimainen RS, Bronskill SE, Tierney MC, et al. Identification of physician-diagnosed Alzheimer’s disease and related dementias in population-based administrative data: a validation study using family physicians’ electronic medical records. J Alzheimers Dis 2016; 54: 337–49. https://doi.org/10.3233/jad-160105
Article PubMed Google Scholar
Wammes JJ, van der Wees PJ, Tanke MA, Westert GP, Jeurissen PP. Systematic review of high-cost patients’ characteristics and healthcare utilisation. BMJ Open 2018; 8: e023113. https://doi.org/10.1136/bmjopen-2018-023113
Wodchis WP, Bushmeneva K, Nikitovic M, Mckillop I. Guidelines on person-level costing using administrative databases in Ontario, 2013. Available from URL: https://tspace.library.utoronto.ca/handle/1807/87373 (accessed July 2022).
Austin PC. Using the standardized difference to compare the prevalence of a binary variable between two groups in observational research. Commun Stat Simul Comput 2009; 38: 1228–34. https://doi.org/10.1080/03610910902859574
Article Google Scholar
Mihaylova B, Briggs A, O’Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Econ 2011; 20: 897–916. https://doi.org/10.1002/hec.1653
Article PubMed Google Scholar
Blough DK, Ramsey SD. Using generalized linear models to assess medical care costs. Health Serv Outcomes Res Methodol 2000; 1: 185–202.
Article Google Scholar
Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37: 360–3.
PubMed Google Scholar
Austin PC, Tu JV, Alter DA. Comparing hierarchical modeling with traditional logistic regression analysis among patients hospitalized with acute myocardial infarction: should we be analyzing cardiovascular outcomes data differently? Am Heart J 2003; 145: 27–35. https://doi.org/10.1067/mhj.2003.23
Article PubMed Google Scholar
Wijeysundera HC, Li L, Braga V, et al. Drivers of healthcare costs associated with the episode of care for surgical aortic valve replacement versus transcatheter aortic valve implantation. Open Heart 2016; 3: e000468. https://doi.org/10.1136/openhrt-2016-000468
Article PubMed PubMed Central Google Scholar
Akinwande MO, Dikko HG, Samson A. Variance inflation factor: as a condition for the inclusion of suppressor variable (s) in regression analysis. Open J Stat 2015; 5(07): 754.
Article Google Scholar
Cundill B, Alexander ND. Sample size calculations for skewed distributions. BMC Med Res Methodol 2015; 15: 28. https://doi.org/10.1186/s12874-015-0023-0
Article PubMed PubMed Central Google Scholar
Van Smeden M, de Groot JA, Moons KG, et al. No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Med Res Methodol 2016; 16: 1–12. https://doi.org/10.1186/s12874-016-0267-3
Article Google Scholar
Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008; 27: 157–72. https://doi.org/10.1002/sim.2929
Article PubMed Google Scholar
Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014; 33: 517–35. https://doi.org/10.1002/sim.5941
Article PubMed Google Scholar
Pencina MJ, D’Agostino RB Sr. Evaluating discrimination of risk prediction models: the C statistic. JAMA 2015; 314: 1063–4. https://doi.org/10.1001/jama.2015.11082
Article CAS PubMed Google Scholar
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–45. https://doi.org/10.2307/2531595
Article CAS PubMed Google Scholar
Neath AA, Cavanaugh JE. The Bayesian information criterion: background, derivation, and applications. Wiley Interdiscip Rev Comput Stat 2012; 4: 199–203. https://doi.org/10.1002/wics.199
Article Google Scholar
Vrieze SI. Model selection and psychological theory: a discussion of the differences between the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Psychol Methods 2014; 17: 228–43. https://doi.org/10.1037/a0027127
Article Google Scholar
Steyerberg EW, Vickers AJ, Cook NR, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 2010; 21: 128–38. https://doi.org/10.1097/ede.0b013e3181c30fb2
Article PubMed PubMed Central Google Scholar
Austin PC, Ghali WA, Tu JV. A comparison of several regression models for analysing cost of CABG surgery. Stat Med 2003; 22: 2799–815. https://doi.org/10.1002/sim.1442
Article PubMed Google Scholar
Muratov S, Lee J, Holbrook A, et al. Incremental healthcare utilisation and costs among new senior high to cost users in Ontario, Canada: A retrospective matched cohort study. BMJ Open. 2019; 9(10): 1–9. https://doi.org/10.1136/bmjopen-2018-028637
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 1982; 143: 29–36. https://doi.org/10.1148/radiology.143.1.7063747
Article CAS PubMed Google Scholar
Tjur T. Coefficients of determination in logistic regression models - a new proposal: the coefficient of discrimination. Am Stat 2009; 63: 366–72. https://doi.org/10.1198/tast.2009.08210
Article Google Scholar
Austin PC, Lee DS, Fine JP. Introduction to the analysis of survival data in the presence of competing risks. Circulation 2016; 133: 601–9. https://doi.org/10.1161/circulationaha.115.017719
Article PubMed PubMed Central Google Scholar
Lau B, Cole SR, Gange SJ. Competing risk regression models for epidemiologic data. Am J Epidemiol 2009; 170: 244–56. https://doi.org/10.1093/aje/kwp107
Article PubMed PubMed Central Google Scholar
Kundi H, Valsdottir LR, Popma JJ, et al. Impact of a claims-based frailty indicator on the prediction of long-term mortality after transcatheter aortic valve in Medicare beneficiaries. Circ Cardiovasc Qual Outcomes 2018; 11: 1–8. https://doi.org/10.1161/circoutcomes.118.005048
Article Google Scholar
Aguayo GA, Donneau AF, Vaillant MT, et al. Practice of epidemiology agreement between 35 published frailty scores in the general population. Am J Epidemiol 2017; 186: 420–34. https://doi.org/10.1093/aje/kwx061
Article PubMed PubMed Central Google Scholar
Searle SD, Mitnitski A, Gahbauer EA, Gill TM, Rockwood K. A standard procedure for creating a frailty index. BMC Geriatr 2008; 8: 1–10. https://doi.org/10.1186/1471-2318-8-24
Article Google Scholar
Henning KA, Ravindran M, Qiu F, et al. Impact of procedural capacity on transcatheter aortic valve replacement wait times and outcomes: A study of regional variation in Ontario, Canada. Open Heart 2020; 7: e001241. https://doi.org/10.1136/openhrt-2020-001241
Article PubMed PubMed Central Google Scholar
Czarnecki A, Qiu F, Henning KA, et al. Comparison of 1-year pre- and post-transcatheter aortic valve replacement hospitalization rates: a population-based cohort study. Can J Cardiol 2020; 36: 1616–23. https://doi.org/10.1016/j.cjca.2020.01.009
Article PubMed Google Scholar

Download references

Author contributions

Zhe Li, Harindra Wijeysundera, Rodrigo Bagur, Davy Cheng, Janet Martin, Bob Kiaii, Feng Qiu, Jiming Fang, and Ava John-Baptiste contributed to the conception and design of the study. Zhe Li, Harindra Wijeysundera, Feng Qiu, Jiming Fang and Ava John-Baptiste contributed to the acquisition of data. Data analysis was primarily conducted by Zhe Li. Ava John-Baptiste provided primary academic supervision. Zhe Li, Harindra Wijeysundera, and Ava John-Baptiste contributed to the interpretation of data. Zhe Li drafted the manuscript. All coauthors have contributed to the revision of the manuscript.

Acknowledgements

This study was supported by the ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC). Parts of this material are based on data and information compiled and provided by the Ontario MOH and the Canadian Institute for Health Information. The analyses, conclusions, opinions, and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred. The clinical registry data used in this study is from participating hospitals through CorHealth Ontario, which serves as an advisory body to the Ministry of Health and Long-Term Care (MOHLTC), is funded by the MOHLTC, and is dedicated to improving the quality, efficiency, access, and equity in the delivery of the continuum of adult cardiac, vascular, and stroke services in Ontario, Canada. We thank IQVIA Solutions Canada Inc. for use of their Drug Information File.

Disclosures

There is no competing interest to declare.

Funding statement

This study was supported by the ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long-Term Care (MLTC) and the Academic Medical Organization of Southwestern Ontario (grant number: INN 17-001) of the Innovation Fund from the Ontario Ministry of Health and Ontario Medical Association. Bob Kiaii has received speaker and consultant honoraria from and has served as a proctor to Medtronic. Bob Kiaii has received speaker and consultant honoraria from Johnson & Johnson. Bob Kiaii has received consultant honoraria from Abbott. Bob Kiaii has served as a proctor to Boston Scientific.

Editorial responsibility

This submission was handled by Dr. Stephan K. W. Schwarz, Editor-in-Chief, Canadian Journal of Anesthesia/Journal canadien d’anesthésie.

Author information

Authors and Affiliations

Department of Epidemiology and Biostatistics, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
Zhe Li PhD, MPH, Rodrigo Bagur MD, PhD, FAHA, FSCAI, Janet Martin PharmD, MSc & Ava John-Baptiste PhD
Centre for Medical Evidence, Decision Integrity & Clinical Impact, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
Zhe Li PhD, MPH, Janet Martin PharmD, MSc & Ava John-Baptiste PhD
Department of Anesthesia & Perioperative Medicine, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
Zhe Li PhD, MPH, Janet Martin PharmD, MSc & Ava John-Baptiste PhD
ICES, Toronto, ON, Canada
Harindra C. Wijeysundera MD, PhD, Feng Qiu MSc & Jiming Fang PhD
Sunnybrook Health Sciences Centre, Toronto, ON, Canada
Harindra C. Wijeysundera MD, PhD
Division of Cardiology, Department of Medicine, Schulich School of Medicine and Dentistry, London, Canada
Rodrigo Bagur MD, PhD, FAHA, FSCAI
School of Medicine, The Chinese University of Hong Kong, Shenzhen, China
Davy Cheng MD
Division of Cardiac Surgery, Department of Surgery, UC Davis Medical Center, Sacramento, CA, USA
Bob Kiaii MD
Schulich Interfaculty Program in Public Health, Western University, London, ON, Canada
Ava John-Baptiste PhD
ICES Western, London, ON, Canada
Zhe Li PhD, MPH & Ava John-Baptiste PhD

Authors

Zhe Li PhD, MPH
Harindra C. Wijeysundera MD, PhD
Rodrigo Bagur MD, PhD, FAHA, FSCAI
Davy Cheng MD
Janet Martin PharmD, MSc
Bob Kiaii MD
Feng Qiu MSc
Jiming Fang PhD
Ava John-Baptiste PhD

Corresponding author

Correspondence toAva John-Baptiste PhD.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article

Li, Z., Wijeysundera, H.C., Bagur, R. et al. Performance of administrative database frailty instruments in predicting clinical outcomes and cost for patients undergoing transcatheter aortic valve implantation: a historical cohort study.Can J Anesth/J Can Anesth 70, 116–129 (2023). https://doi.org/10.1007/s12630-022-02354-6

Download citation

Received: 14 January 2022
Revised: 11 June 2022
Accepted: 07 July 2022
Published: 28 December 2022
Version of record: 28 December 2022
Issue date: January 2023
DOI: https://doi.org/10.1007/s12630-022-02354-6

Performance of administrative database frailty instruments in predicting clinical outcomes and cost for patients undergoing transcatheter aortic valve implantation: a historical cohort study (original) (raw)

Abstract

Purpose

Methods

Results

Conclusion

Résumé

Objectif

Méthode

Résultats

Conclusion

Similar content being viewed by others

Methods

Study design and data sources

Study cohort

Administrative database frailty instruments

Covariates

Clinical outcomes

Cost outcomes

Statistical approaches

Sensitivity analyses

Results

Characteristics of the cohort

Predictive performance of frailty instruments

Agreement between the two frailty instruments

Frailty instruments and clinical outcomes

Frailty instruments and cost outcomes

Discussion

Principal findings

Comparisons with other studies

Clinical implications

Study limitations

Conclusions

Notes

References

Author contributions

Acknowledgements

Disclosures

Funding statement

Editorial responsibility

Author information

Authors and Affiliations

Corresponding author

Additional information

Publisher's Note

Supplementary Information

Rights and permissions

About this article

Cite this article

Keywords