Creating a Usability Sample Size Calculator

January 16, 2012 | By Tom Satwicz, Ph.D.

At Blink our clients often ask for advice on the number of participants needed for a usability study. For simple studies, those that test a single design with one group of users, we typically recommend recruiting 8 – 10 users. Client projects are often more complex, however, and additional factors often come into play that necessitate raising the number of study participants. Because adding participants can significantly increase the cost of a study, finding a way to balance those factors can be a challenge. With this in mind, we developed a tool to make it easy to adjust different factors and “what if” scenarios to see how they impact the number of participants needed for testing.

We began the process of building our calculator with a literature review and compared the published reports with our collective experience. We then developed a flow chart that laid down logic considering the different factors that impact the size of a study. In doing so, it became immediately apparent that we were wading into a debate about the appropriateness of a formula that predicts how many participants it takes to uncover the majority of usability issues. The sticking point in the debate is one of pragmatics and experience vs. theory. One camp, articulated in a well-known article by Jakob Nielsen, suggests that “5 users is enough” to uncover the majority of important issues in a single, straightforward usability study.  Nielsen argues that given limited budgets, it is smarter to test with fewer users, iterate on a design, and then test again. Laura Faulkner reported that 10- or even 20-user samples may be needed to consistently uncover the majority of usability problems during testing.  Others have suggested even more.

In our experience, both arguments have merit but should be applied differently, depending on factors such as the precision desired in a study and whether the study is formative (part of a design process) or summative (intended to test an existing product to obtain a baseline for measuring against future efforts). There are also the practical considerations of time and budget.

We chose as a base of recruiting 10 participants (expecting 8 to show) for a simple usability study that tests a single design. From there we considered other factors such as the number of distinct user groups, whether those groups will be performing the same tasks, how the results will be used in the future, the number of designs that will be tested, and whether eye tracking will be used as a method in the study analysis.

Many other issues came into play when designing the calculator, but as we tested its sample-size results against successful studies we have run in the past, we found its results to be amazingly consistent with the numbers of participants we recommended for studies prior to having the tool.

Beyond the calculator and debates over appropriate sample size, perhaps most important of all when running a usability study is engaging with a professional usability team that helps you identify key objectives and factors that influence the study design, provides a usability testing moderator who is expert at recognizing and prioritizing issues, and then offers comprehensive design recommendations and other solutions in its findings that address the problems found.

Please give the usability sample size calculator a try. We would love to hear what you think via the comment form below: