A risk-based comparison of classification systems (original) (raw)
Related papers
ESTIMATING THE ROC CURVE AND ITS SIGNIFICANCE FOR CLASSIFICATION MODELS' ASSESSMENT
Article presents a ROC (receiver operating characteristic) curve and its application for classification models' assessment. ROC curve, along with area under the receiver operating characteristic (AUC) is frequently used as a measure for the diagnostics in many industries including medicine, marketing, finance and technology. In this article, we discuss and compare estimation procedures, both parametric and non-parametric, since these are constantly being developed, adjusted and extended.
Kernel estimators of the ROC curve are better than empirical
Statistics & Probability Letters, 1999
The receiver operating characteristic (ROC) is a curve used to summarise the performance of a binary decision rule. It can be expressed in terms of the underlying distributions functions of the diagnostic measurement that underlies the rule. Lloyd (1998) has proposed estimating the ROC curve from kernel smoothing of these distribution functions and has presented asymptotic formulas for the bias and standard deviation of the resulting curve estimator. This paper compares the asymptotic accuracy of the kernel-based estimator with the fully empirical estimator. It is shown that the empirical estimator is deÿcient compared to the kernel estimator and that this deÿciency is unbounded as sample size increases. A simulation study using both unimodal and bimodal distributions indicates that the gains in accuracy are signiÿcant for realistic sample sizes. Kernel-based ROC estimators can now be recommended. c 1999 Elsevier Science B.V. All rights reserved MSC: primary 62G05; 60F17; secondary 62E20; 62G20
An introduction to ROC analysis
Receiver operating characteristics (ROC) graphs are useful for organizing classifiers and visualizing their performance. ROC graphs are commonly used in medical decision making, and in recent years have been used increasingly in machine learning and data mining research. Although ROC graphs are apparently simple, there are some common misconceptions and pitfalls when using them in practice. The purpose of this article is to serve as an introduction to ROC graphs and as a guide for using them in research.
ROC curve estimation: an overview
• This work overviews some developments on the estimation of the Receiver Operating Characteristic (ROC) curve. Estimation methods in this area are constantly being developed, adjusted and extended, and it is thus impossible to cover all topics and areas of application in a single paper. Here, we focus on some frequentist and Bayesian methods which have been mostly employed in the medical setting. Although we emphasize the medical domain, we also describe links with other fields where related developments have been made, and where some modeling concepts are often known under other designations.
Three-Class ROC Analysis—Toward a General Decision Theoretic Solution
IEEE Transactions on Medical Imaging, 2000
Multiclass receiver operating characteristic (ROC) analysis has remained an open theoretical problem since the introduction of binary ROC analysis in the 1950s. Previously, we have developed a paradigm for three-class ROC analysis that extends and unifies decision theoretic, linear discriminant analysis, and probabilistic foundations of binary ROC analysis in a three-class paradigm. One critical element in this paradigm is the equal error utility (EEU) assumption. This assumption allows us to reduce the intrinsic space of the three-class ROC analysis (5-D hypersurface in 6-D hyperspace) to a 2-D surface in the 3-D space of true positive fractions (sensitivity space). In this work, we show that this 2-D ROC surface fully and uniquely provides a complete descriptor for the optimal performance of a system for a three-class classification task, i.e., the triplet of likelihood ratio distributions, assuming such a triplet exists. To be specific, we consider two classifiers that utilize likelihood ratios, and we assumed each classifier has a continuous and differentiable 2-D sensitivity-space ROC surface. Under these conditions, we proved that the classifiers have the same triplet of likelihood ratio distributions if and only if they have the same 2-D sensitivity-space ROC surfaces. As a result, the 2-D sensitivity surface contains complete information on the optimal three-class task performance for the corresponding likelihood ratio classifier.
A Comparison of the ROC Curve Estimators
The ROC (Receiver Operating Chracteristic) curves are frequently used for difierent diagnostic purposes. There are several difierent approaches how to flnd the suitable estimate of the ROC curve in binormal model. The efiective methods which can be used when the sample sizes are small are still very demanded in difierent applications. In the paper the binormal model is assumed and the parametric, semiparametric and nonparametric estimators are compared by simulation study. The parametric approach is based on the method of weighted least squares, the semiparametric approach is based on the functional modelling and the nonparametric approach is based on the sample or empirical cumulative distributive function (cdf). 2. The ROC and ODC curves The receiver operating characteristic (ROC) curve is used for classiflcation between two groups of subjects. One will be called the group with condition and the other will be called the group without condition. The ROC is deflned as a plot of prob...
Cost-Sensitive Classifier Selection Using the ROC Convex Hull Method
One binary classifier may be preferred to another based on the fact that it has better prediction accuracy than its competitor. Without additional information describing the cost of a misclassification, accuracy alone as a selection criterion may not be a sufficiently robust measure when the distribution of classes is greatly skewed or the costs of different types of errors may be significantly different. The receiver operating characteristic (ROC) curve is often used to summarize binary classifier performance due to its ease of interpretation, but does not include misclassification cost information in its formulation. Provost and Fawcett [5, 7] have developed the ROC Convex Hull (ROCCH) method that incorporates techniques from ROC curve analysis, decision analysis, and computational geometry in the search for the optimal classifier that is robust with respect to skewed or imprecise class distributions and disparate misclassification costs. We apply the ROCCH method to several datasets using a variety of modeling tools to build binary classifiers and compare their performances using misclassification costs. We support Provost, Fawcett, and Kohavi’s claim [6] that classifier accuracy, as represented by the area under the ROC curve, is not an optimal criterion in itself for choosing a classifier, and that by using the ROCCH method, a more appropriate classifier may be found that realistically reflects class distribution and misclassification costs.
ROC analysis of classifiers in machine learning: A survey
The use of ROC (receiver operating characteristics) analysis as a tool to evaluate performance of classification models in machine learning has been increasing in the last decade. Among the most notable advances in this area is that the two-class ROC analysis has been extended to multi-class problems and that the ROC analysis in cost-sensitive learning has been considered, as well. Methods exist which take instance-varying costs into account. The purpose of our paper is to present a survey of this field with the aim to gather important achievements in one place. In the paper, we present application areas of the ROC analysis in machine learning, describe its problems and challenges and provide a summarized list of alternative approaches to ROC analysis. In addition to presented theory, we also provide a couple of examples intended to illustrate the described approaches.
Have we overestimated the value of ROC analysis under varying class distributions?
2003
We counsel caution in the application of ROC analysis for prediction of classifier accuracy under varying class distributions. We argue that it is only reasonable to expect ROC analysis to provide accurate error prediction under varying class distributions if the attributes are causally dependent upon the classes and the classes do not contain causally relevant subclasses whose frequency may vary at different rates.
2013
A receiver operating characteristic (ROC) curve is a plot of predictive model probabilities of true positives (sensitivity) as a function of probabilities of false positives (1 � specificity) for a set of possible cutoff points. Some of the SAS/STAT procedures do not have built-in options for ROC curves and there have been a few suggestions in previous SAS forums to address the issue by using either parametric or non-parametric methods to construct those curves. This study shows how a simple concave function verifies the properties necessary to provide good fits to the ROC curve of diverse predictive models, and has also the advantage of giving an exact solution for the estimation of