Net reclassification indices for evaluating risk prediction instruments: a critical review - PubMed (original) (raw)

Review

Net reclassification indices for evaluating risk prediction instruments: a critical review

Kathleen F Kerr et al. Epidemiology. 2014 Jan.

Abstract

Net reclassification indices have recently become popular statistics for measuring the prediction increment of new biomarkers. We review the various types of net reclassification indices and their correct interpretations. We evaluate the advantages and disadvantages of quantifying the prediction increment with these indices. For predefined risk categories, we relate net reclassification indices to existing measures of the prediction increment. We also consider statistical methodology for constructing confidence intervals for net reclassification indices and evaluate the merits of hypothesis testing based on such indices. We recommend that investigators using net reclassification indices should report them separately for events (cases) and nonevents (controls). When there are two risk categories, the components of net reclassification indices are the same as the changes in the true- and false-positive rates. We advocate the use of true- and false-positive rates and suggest it is more useful for investigators to retain the existing, descriptive terms. When there are three or more risk categories, we recommend against net reclassification indices because they do not adequately account for clinically important differences in shifts among risk categories. The category-free net reclassification index is a new descriptive device designed to avoid predefined risk categories. However, it experiences many of the same problems as other measures such as the area under the receiver operating characteristic curve. In addition, the category-free index can mislead investigators by overstating the incremental value of a biomarker, even in independent validation data. When investigators want to test a null hypothesis of no prediction increment, the well-established tests for coefficients in the regression model are superior to the net reclassification index. If investigators want to use net reclassification indices, confidence intervals should be calculated using bootstrap methods rather than published variance formulas. The preferred single-number summary of the prediction increment is the improvement in net benefit.

PubMed Disclaimer

Figures

Figure 1

Figure 1

In each plot the solid line is the ROC curve for the “old” marker and the dotted line is the ROC curve for the “new” risk model that incorporates the new marker. The new marker has identical distribution in all four cases. _NRI>0_=0.622 in all cases, despite the fact that the prediction increment of the new marker decreases as the strength of the old model increases.

Figure 2

Figure 2

The same data as Figure 1 are shown here in terms of the distributions of risks for old and new risk models. Risk distributions are shown on the log odds scale. Solid lines are the risks using the established predictors X, with nonevents tending to have lower risks than events. Dotted lines are risks using the new marker Y together with X.

Comment in

Similar articles

Cited by

References

    1. Wilson PW, D'Agostino RB, Levy D, Belanger AM, Silbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation. 1998;97(18):1837–1847. - PubMed
    1. Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ. Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. Journal of the National Cancer Institute. 1989;81(24):1879–86. - PubMed
    1. 27th Bethesda Conference: Matching the intensity of risk factor management with the hazard for coronary disease events. Journal of the American College of Cardiology. 1996;27(5):957–1047. - PubMed
    1. Pencina MJ, D'Agostino RB, Sr, D'Agostino RB, Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Statistics in Medicine. 2008;27:157–172. - PubMed
    1. Pencina MJ, D'Agostino RB, Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Statistics in Medicine. 2011;30:11–21. - PMC - PubMed

Publication types

MeSH terms

Grants and funding

LinkOut - more resources