Representing classifier confidence in the safety critical domain: an illustration from mortality prediction in trauma cases (original) (raw)

Abstract

This work proposes a novel approach to assessing confidence measures for software classification systems in demanding applications such as those in the safety critical domain. Our focus is the Bayesian framework for developing a model-averaged probabilistic classifier implemented using Markov chain Monte Carlo (MCMC) and where appropriate its reversible jump variant (RJ-MCMC). Within this context we suggest a new technique, building on the reject region idea, to identify areas in feature space that are associated with “unsure” classification predictions. We term such areas “uncertainty envelopes” and they are defined in terms of the full characteristics of the posterior predictive density in different regions of the feature space. We argue this is more informative than use of a traditional reject region which considers only point estimates of predictive probabilities. Results from the method we propose are illustrated on synthetic data and also usefully applied to real life safety critical systems involving medical trauma data.

Access this article

Log in via an institution

Subscribe and save

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Bishop CM (1998) Neural networks for pattern recognition. Oxford University Press, New York
  2. Bouissou M, Martin F, Ourghanlian A (1999) Assessment of safety critical system including software: a Bayesian belief network for evidence sources. In: Proceedings of the reliability and maintainability symposium, Washington DC, January 1999
  3. Denison DGT, Holmes CC, Mallick BK, Smith AFM (2002) Bayesian methods for nonlinear classification and regression. Wiley, London
  4. Fukunaga K (1990) Introduction to statistical pattern recognition. Morgan Kaufmann
  5. Green PJ (1995) Reversible jump Markov chain Monte Carlo computation and Bayesian model determination. Biometrika 82:711–732
    Article MATH MathSciNet Google Scholar
  6. Little B, Stringini L (1993) Validation of ultra-high dependability for software-based systems. Commun ACM 36(11):69–80
    Article Google Scholar
  7. MacKay DJC (1992) Bayesian interpolation. Neural Comput 4(3):415–447
    Google Scholar
  8. MacKay DJC (1998) Introduction to Monte Carlo methods. In: Jordan MI (ed) Learning in graphical models, NATO Science Series. Kluwer, Dordrecht, pp 175–204
    Google Scholar
  9. Nabney IT (2002) NETLAB: algorithms for pattern recognition. Springer, Berlin Heidelberg New York
    Google Scholar
  10. Ripley BD (1994) Neural networks and related methods for classification. J R Stat Soc B 56(3):409–456
    MATH MathSciNet Google Scholar

Download references

Acknowledgements

This work was supported by grant GR/R24357/01 of the UK Engineering and Physical Sciences Research Council.

Author information

Authors and Affiliations

  1. School of Engineering, Computer Science and Mathematics, University of Exeter, Exeter, UK
    Trevor C. Bailey, Richard M. Everson, Jonathan E. Fieldsend, Wojtek J. Krzanowski, Derek Partridge & Vitaly Schetinin

Authors

  1. Trevor C. Bailey
  2. Richard M. Everson
  3. Jonathan E. Fieldsend
  4. Wojtek J. Krzanowski
  5. Derek Partridge
  6. Vitaly Schetinin

Corresponding author

Correspondence toTrevor C. Bailey.

Rights and permissions

About this article

Cite this article

Bailey, T.C., Everson, R.M., Fieldsend, J.E. et al. Representing classifier confidence in the safety critical domain: an illustration from mortality prediction in trauma cases.Neural Comput & Applic 16, 1–10 (2007). https://doi.org/10.1007/s00521-006-0053-y

Download citation

Keywords