A note on the choice of a weighting function to give an efficient method for estimating the probability of misclassification (original) (raw)

On Misclassification Probabilities of Linear and Quadratic Classifiers

Afrika Statistika, 2016

We study the theoretical misclassification probability of linear and quadratic classifiers and examine the performance of these classifiers under distributional variations in theory and using simulation. We derive expression for Bayes errors for some competing distributions from the same family under location shift. Résumé. Nousétudions les probabilités théoriques de malclassification de méthodes de classification linéaires et quadratiques. Ensuite, nous examinons les performance de ces méthodes de classifications selon différentes distributions, sur le plan théorique en lesétayant avec deś etudes de simulations. Nous exprimons l'erreur de Bayes pour des distributions de même famille en compétition selon le changement du paramètre de centralisation.

An analysis on the relationship between uncertainty and misclassification rate of classifiers

Information Sciences, 2020

This paper provides new insight into the analysis on the relationship between uncertainty and misclassification of a classifier. We formulate the relationship explicitly by taking entropy as a measurement of uncertainty and by analyzing the misclassification rate based on the membership degree difference. Focusing on binary classification problems, this study theoretically and experimentally validates that the misclassification rate will definitely be upgrading with the increase of uncertainty if two conditions are satisfied: (1) the distributions of two classes based on membership degree difference are unimodal, and (2) these two distributions attain peaks when the membership degree difference is less and larger than zero, respectively. This work aims to provide some practical guidelines for improving classifier performance through clearly expressing and understanding the relationship between uncertainty and misclassification of a classifier.

Cost-Based Classifier Evaluation for Imbalanced Problems

Lecture Notes in Computer Science, 2004

A common assumption made in the field of Pattern Recognition is that the priors inherent to the class distributions in the training set are representative of the true class distributions. However this assumption does not always hold, since the true class-distributions may be different, and in fact may vary significantly. The implication of this is that the effect on cost for a given classifier may be worse than expected. In this paper we address this issue, discussing a theoretical framework and methodology to assess the effect on cost for a classifier in imbalanced conditions. The methodology can be applied to many different types of costs. Some artificial experiments show how the methodology can be used to assess and compare classifiers. It is observed that classifiers that model the underlying distributions well are more resilient to changes in the true class distribution than weaker classifiers.

Reliability estimation of a statistical classifier

Pattern Recognition Letters, 2008

Pattern classification techniques derived from statistical principles have been widely studied and have proven powerful in addressing practical classification problems. In real-world applications, the challenge is often to cope with unseen patterns i.e., patterns which are very different from those examined during the training phase. The issue with unseen patterns is the lack of accuracy of the classifier output in the regions of pattern space where the density of training data is low, which could lead to a false classification output. This paper proposes a method for estimating the reliability of a classifier to cope with these situations. While existing methods for quantifying the reliability are often based on the class membership probability estimated on global approximations, the proposed method takes into account the local density of training data in the neighborhood of a test pattern. The calculations are further simplified by using the Gaussian mixture model (GMM) to calculate the local density of the training data. The reliability of a classifier output is defined in terms of a confidence interval on the class membership probability. The lower bound of a confidence interval or the local density of training data may be used to detect the unseen patterns. The effectiveness of the proposed method is demonstrated using real data sets and performance is compared with other reliability estimation methods.

Classifier quality definition on the basis of the estimation calculation approach

The generalized problem of classifiers training, using training sets has been considered. The accents on the principle moments, reasoning the classifiers overtraining, the ways of impact estimation of these effects degree and the methods of their control have been made. The approach, giving the possibility to estimate the effect of training set reduction on the basis of probabilistically combinatorial method has been considered. The training set reduction effect estimation gives the possibility to establish the redundancy of the training data and define the relative of the negative training objects such as spikes and spurious objects.

Notes on the H-measure of classifier performance

Advances in Data Analysis and Classification, 2022

The H-measure is a classifier performance measure which takes into account the context of application without requiring a rigid value of relative misclassification costs to be set. Since its introduction in 2009 it has become widely adopted. This paper answers various queries which users have raised since its introduction, including questions about its interpretation, the choice of a weighting function, whether it is strictly proper, its coherence, and relates the measure to other work.

Misclassification analysis for the class imbalance problem

2010

In classification, the class imbalance issue normally causes the learning algorithm to be dominated by the majority classes and the features of the minority classes are sometimes ignored. This will indirectly affect how human visualise the data. Therefore, special care is needed to take care of the learning algorithm in order to enhance the accuracy for the minority classes. In

On the Suitability of Numerical Performance Measures for Class Imbalance Problems

2012

The class imbalance problem has been reported as an important challenge in various fields such as Pattern Recognition, Data Mining and Machine Learning. A less explored research area is related to how to evaluate classifiers on imbalanced data sets. This work analyzes the behaviour of performance measures widely used on imbalanced problems, as well as other metrics recently proposed in the literature. We perform two theoretical analysis based on Pearson correlation and operations for a 2× 2 confusion matrix with the aim to show the strengths and weaknesses of those performance metrics in the presence of skewed distributions.

Resampling methods for quality assessment of classifier performance and optimal number of features

Signal Processing, 2013

We address two fundamental design issues of a classification system: the choice of the classifier and the dimensionality of the optimal feature subset. Resampling techniques are applied to estimate both the probability distribution of the misclassification rate (or any other figure of merit of a classifier) subject to the size of the feature set, and the probability distribution of the optimal dimensionality given a classification system and a misclassification rate. The latter allows for the estimation of confidence intervals for the optimal feature set size. Based on the former, a quality assessment for the classifier performance is proposed. Traditionally, the comparison of classification systems is accomplished for a fixed feature set. However, a different set may provide different results. The proposed method compares the classifiers independently of any pre-selected feature set. The algorithms are tested on 80 sets of synthetic examples and six standard databases of real data. The simulated data results are verified by an exhaustive search of the optimum and by two feature selection algorithms for the real data sets.

The influence of prior knowledge on the expected performance of a classifier

Pattern Recognition Letters, 2003

In this paper, we study the probabilistic properties of pattern classifiers in discrete feature space. The principle of Bayesian averaging of recognition performance is used for this analysis. We consider two cases: (a) prior probabilities of classes are unknown, and (b) prior probabilities of classes are known. The misclassification probability is represented as a random value, for which the characteristic function (expressed via Kummer hypergeometric function) and absolute moments are analytically derived. For the case of unknown priors, an approximate formula for calculation of sufficient learning sample size is obtained. The comparison between the performances for two considered cases is made. As an example, we consider the problem of mutational hotspots classification in genetic sequences.