Why your product needs usability testing
Usability testing is key to product quality.
Constant exposure to your customers is key to quality
Most companies nowadays have a mature attitude to data. But I still sometimes hear objections that usability testing is not scientific. Even senior colleagues sometimes voice concerns that usability tests are not “statistically significant”. In an industry where companies live or die by their user experience, these beliefs are really problematic. So let’s deal with this.
“Statistical significance” won’t give you the whole picture
In statistical studies, we need confidence that results are not just random noise. The larger the sample size, the more likely our outcomes had a common underlying effect — significance.
But “significance” is only when something is interesting or important, that is all. Statistical significance does not equal meaning.
Qualitative data can provide insight into what happened and why. So pairing unstructured, qualitative data with quantitative studies can give significance and meaning.
It’s about catching the big fish
We do usability testing to observe user behaviour and find problems. This also helps us see our work through fresh eyes and gives us empathy.
For usability tests, around 15 users is generally considered enough. This is because we are not looking for statistical significance. It’s more like we are fishing for individual problems. Catching even a single problem in a small pool of users is a valuable result. 🐟
Finding a problem in your design means you can eliminate it. Some people call this the design thinking approach, but I prefer calling it the “find and fix” mindset.
Usability testing is about catching the “big fish”. That is, the most frequent problems that most users could encounter. We won’t catch all the problems, but we don’t want to — just the ones that affect most users, because fixing those will have the biggest coverage of users and therefore the highest return on investment.
Source: Nielsen Norman Group
Usability tests aren’t statistically significant, but they are scientific
As you can see above, 15 user tests will ensure we catch 85% of the most common problems that will affect most users. This likelihood is based on “binomial probability”, and has been backed up with 20 years of academic research. You can read more here and here.
Unsurprisingly, you will also see in the chart above that zero users give zero insights. This means you will get far more insight from a 3 person usability study than doing nothing.
(Please note — that what you test really matters. The thought you put into the test affects the quality of the info you get out).
Your product needs usability testing
Usability tests are quick and cheap. Set up your tests well. Adopt the find and fix mindset. Run quant and qual studies in parallel. Then be confident that your product is good enough for users.
Go catch your fish. 🐟
–
