Beyond the five-user assumption: Benefits of increased sample sizes in usability testing (original) (raw)

Abstract

It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.

Article PDF

Similar content being viewed by others

References

• Grosvenor, L. (1999).Software usability: Challenging the myths and assumptions in an emerging field. Unpublished master’s thesis, University of Texas, Austin.
  Google Scholar
• Hudson, W. (2001). How many users does it take to change a website?SIGCHI Bulletin May/June 2001. Retrieved April 15, 2003 from http://www.acm.org/sigchi/bulletin/2001.3/mayjun01.pdf.
• Landauer, T. K., & Nielsen, J. (1993). A mathematical model of the finding of usability problems.Interchi ’93, ACM Computer-Human Interface Special Interest Group.
• Nielsen, J. (1993).Usability engineering. Boston: AP Professional.
  Google Scholar
• Nielsen, J. (2000, March).Why you only need to test with 5 users: Alertbox. Retrieved April 15, 2003 from http://www.useit.com/ alertbox/20000319.html.
• Perfetti, C., & Landesman, L. (2002).Eight is not enough. Retrieved April 14, 2003 from http://world.std.com/\~uieweb/Articles/eight\_is\_ not enough.htm.
• Reason, J. (1997).Managing the risks of organizational accidents. Brookfield, VT: Ashgate.
  Google Scholar
• Spool, J., &Schroeder, W. (2001). Testing web sites: Five users is nowhere near enough. In_CHI 2001 Extended Abstracts_ (pp. 285–286). New York: ACM Press.
  Chapter Google Scholar
• Virzi, R. A. (1992). Refining the test phase of usability evaluation: How many subjects is enough?Human Factors,34, 457–468.
  Google Scholar
• Wentworth, R. J. (1996, March).Group chairman’s factual report (American Airlines Flight 965). Washington, DC: National Transportation Safety Board, Office of Aviation Safety.
  Google Scholar
• Woolrych, A., &Cockton, G. (2001). Why and when five test users aren’t enough. In J. Vanderdonckt, A. Blandford, & A. Derycke (Eds.),Proceedings of IHM-HCI 2001 Conference: Vol. 2 (pp. 105–108). Toulouse, France: Cépadèus.
  Google Scholar

Download references

Author information

Authors and Affiliations

1. Applied Research Laboratories, University of Texas at Austin, P. O. Box 8029, 78713-8029, Austin, TX
   Laura Faulkner

Authors

1. Laura Faulkner
   You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence toLaura Faulkner.

Rights and permissions

About this article

Cite this article

Faulkner, L. Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.Behavior Research Methods, Instruments, & Computers 35, 379–383 (2003). https://doi.org/10.3758/BF03195514

Download citation

• Received: 19 September 2002
• Accepted: 18 May 2003
• Issue Date: August 2003
• DOI: https://doi.org/10.3758/BF03195514

Keywords