Empirical evidence of the impact of study characteristics on the performance of prediction models: a meta-epidemiological study (original) (raw)
Related papers
Caution when using prognostic models: A prospective comparison of 3 recent prognostic models
Journal of Critical Care, 2012
Purpose: Prognostic models have been developed to estimate mortality and to compare outcomes in different intensive care units. However, these models need to be validated before their use in different populations. In this study, we assessed the performance of 3 recently developed general prognostic models (Acute Physiologic and Chronic Health Evaluation [APACHE] IV, Simplified Acute Physiology Score [SAPS] 3 and Mortality Probability Model III [MPM 0 -III]) in a population admitted at 3 medicalsurgical Brazilian intensive care units. Materials and Methods: All patients admitted from July 2008 to December 2009 were evaluated for inclusion in the study. Standardized mortality ratios were calculated for all models. Calibration was assessed by the Hosmer-Lemeshow goodness-of-fit test. Discrimination was evaluated using the area under the receiver operator curve. Results: A total of 5780 patients were included. Inhospital mortality was 9.1%. Discrimination was very good for all models (area under the receiver operator curve for APACHE IV, SAPS 3 and MPM 0 -III was 0.883, 0.855 and 0.840, respectively). APACHE IV showed better discrimination than SAPS 3 and MPM 0 -III (P b .001 for both comparisons). All models calibrated poorly and overestimated hospital mortality (Hosmer-Lemeshow statistic was 53.7, 134.2, 226.6 for APACHE IV, MPM 0 -III, and SAPS 3, respectively; P b .001 for all). Conclusions: In this study, all models showed poor calibration, while discrimination was very good for all of them. As this has been a common finding in validation studies, caution is warranted when using prognostic models for benchmarking.
Prognosis Research Strategy (PROGRESS) 3: prognostic model research
PLoS medicine, 2013
Prognostic models are abundant in the medical literature yet their use in practice seems limited. In this article, the third in the PROGRESS series, the authors review how such models are developed and validated, and then address how prognostic models are assessed for their impact on practice and patient outcomes, illustrating these ideas with examples.
Statistical methods to assess the prognostic value of risk prediction rules in clinical research
Aging Clinical and Experimental Research, 2020
Prognosis aims at estimating the future course of a given disease in probabilistic terms. As in diagnosis, where clinicians are interested in knowing the accuracy of a new test to identify patients affected by a given disease, in prognosis they wish to accurately identify patients at risk of a future event conditional to one or more prognostic factors. Thus, accurate risk predictions play a primary role in all fields of clinical medicine and in geriatrics as well because they can help clinicians to tailor the intensity of a treatment and to schedule clinical surveillance according to the risk of the concerned patient. Statistical methods able to evaluate the prognostic accuracy of a risk score demand the assessment of discrimination (the Harrell's C-index), calibration (Hosmer-May test) and risk reclassification abilities (IDI, an index of risk reclassification) of the same risk prediction rule whereas, in spite of the popular belief that traditional statistical techniques providing relative measures of effect (such as the hazard ratio derived by Cox regression analysis or the odds ratio obtained by logistic regression analysis) could be per se enough to assess the prognostic value of a biomarker or of a risk score. In this paper we provide a brief theoretical background of each statistical test and a practical approach to the issue. For didactic purposes, in the paper we also provide a dataset (n = 40) to allow the reader to train in the application of the proposed statistical methods.
Meta-analysis of clinical prediction models
2013
This PhD thesis explores statistical methods for adopting evidence synthesis in the development and validation of risk prediction models. The ultimate aim is to improve the future performance and to enhance understanding of their potential generalizability across different settings and populations.
Diagnostic and Prognostic Research Evidence synthesis in prognosis research
Diagnostic and Prognostic Research, 2019
Over the past few years, evidence synthesis has become essential to investigate and improve the generalizability of medical research findings. This strategy often involves a meta-analysis to formally summarize quantities of interest, such as relative treatment effect estimates. The use of meta-analysis methods is, however, less straightforward in prognosis research because substantial variation exists in research objectives, analysis methods and the level of reported evidence. We present a gentle overview of statistical methods that can be used to summarize data of prognostic factor and prognostic model studies. We discuss how aggregate data, individual participant data, or a combination thereof can be combined through meta-analysis methods. Recent examples are provided throughout to illustrate the various methods.
Journal of clinical epidemiology, 2016
To compare different methods to handle treatment when developing a prognostic model that aims to produce accurate probabilities of the outcome of individuals if left untreated. Simulations were performed based on two normally distributed predictors, a binary outcome, and a binary treatment, mimicking a randomized trial or an observational study. Comparison was made between simply ignoring treatment (SIT), restricting the analytical data set to untreated individuals (AUT), inverse probability weighting (IPW), and explicit modeling of treatment (MT). Methods were compared in terms of predictive performance of the model and the proportion of incorrect treatment decisions. Omitting a genuine predictor of the outcome from the prognostic model decreased model performance, in both an observational study and a randomized trial. In randomized trials, the proportion of incorrect treatment decisions was smaller when applying AUT or MT, compared to SIT and IPW. In observational studies, MT was ...
Systematic reviews, 2017
Prognosis research is on the rise, its importance recognised because chronic health conditions and diseases are increasingly common and costly. Prognosis systematic reviews are needed to collate and synthesise these research findings, especially to help inform effective clinical decision-making and healthcare policy. A detailed, comprehensive search strategy is central to any systematic review. However, within prognosis research, this is challenging due to poor reporting and inconsistent use of available indexing terms in electronic databases. Whilst many published search filters exist for finding clinical trials, this is not the case for prognosis studies. This systematic review aims to identify and compare existing methodological filters developed and evaluated to identify prognosis studies of any of the three main types: overall prognosis, prognostic factors, and prognostic [risk prediction] models. Primary studies reporting the development and/or evaluation of methodological sea...
Prognostic models in the clinical arena
Aging Clinical and Experimental Research, 2012
Making a prognosis is to predict the course of a disease and estimate the probability (or risk) of the appearance of a given outcome in relationship to clinical or non-clinical characteristics. Prognostic assessment is usually modelled by multivariable mathematic equations (prognostic models). In this article we describe what a prognostic model is, how to build a good one, why and how it is important to evaluate its generalizability and accuracy by means of discrimination, calibration and reclassification.