Will your usability test be with a single group of users or more than one group? Most qualitative studies are conducted with a single group of users. Characteristics such as age ranges, gender, technology products owned, etc. do not usually warrant different groups. These qualities are often used to ensure diversity within a group, but not between groups.

Consider testing with more than one group if you want to compare performance or other measures between distinct user types of interest. Assessing the performance of novice vs. expert users, for example, may be of interest if research questions relate to feature learnability or discoverability over time. Assessing performance between groups that use systems differently by role may also be of interest, such as usability testing with IT professionals vs. business decision-makers who extract different reports from the same web application.

Testing with more than one group increases the sample size needed to ascertain whether there are actual differences in the experiences of each group.

If each group is performing the same tasks, task performance may be directly compared to determine if there are differences in the experience as a function of group. If user groups will perform different tasks, then the number of participants needed for the study increases. In some cases, tasks may be so different that it is akin to running separate studies. For example, a system administrator may concentrate on deploying a product once throughout an enterprise while a marketing employee may use the same product repeatedly throughout the day.

Performing a baseline or benchmark study in which performance can be measured against future studies increases the sample size needed. These studies assess usability progress over time, such as in different product releases or iterations over months or years, and a more robust sample is needed to make quantitative comparisons.

In most usability testing, a single design is evaluated to identify issues and opportunities for improvement. Some studies, particularly those early in the design cycle, evaluate more than one design alternative to understand the relative benefits and costs of each and help determine a design direction.

If a study is attempting to compare designs, it is a good practice to increase the number of participants to ensure that the majority of usability issues in each design alternative will be discovered.

Answer "1" if each participant will see only one design (between subjects design) or greater than one if participants will see more than one design (within subjects design or repeated measures design).

If eye tracking will be used simply to complement other usability methods such as qualitative observations and a think-aloud protocol, 12 participants is a good starting point for a single group of users. Eye tracking findings and visualizations in these cases should not be considered statistically valid for making inferences about a broader population.

If eye tracking data is intended to be used as a primary data source to generate statistically valid heat maps and other visualizations, a minimum of 39 participants is required, according to Pernice and Nielsen.