*** On June 28th, our User Experience Director, Steve Tengler, presented a webinar entitled “User Experience vs. User Interface”. This post features a question asked by an attendee.
QUESTION: What’s an appropriate sample size? How many rounds of customer tests should I do before production?
STEVE: Depends on the product. If it’s a garage door opener, it’s one round, and a fairly small sample size. If it’s something a lot more complicated, like an F16 cockpit, it’s going to require many more rounds and larger sample sizes. Let me give you more of an answer. You’ll always want an early fact-finding exercise, probably 8-10 participants. I recommend using a professional company to help you recruit here. Especially so they can find your desired persona or if you need to do ‘drive-alongs’ like I talked about earlier. You do this early to find out their unfulfilled needs and how they might use your product in the real world.
Then when you bake your design a bit more, you might want to do another round, maybe 15-20 participants. This is where you confirm your information architecture. ‘Am I laying out the design in a way that makes sense to people?’ That might be doing card sorting. You probably don’t need to go out in the field a lot for that, does the basic setup seem to make sense? You might have a round once you have more of a direction, it doesn’t have to be large (5-8 participants), it can even be internal. (Just be careful, because internally, you all *want it to work). Just dont use engineers.
But all throughout, you should be using a modeling software so you can do several rounds and you’re iterating and getting it into the product. This way you’re not missing out on Speed to Market.One final round would be a verification testing. I usually recommend right before or right after you launch, just to make sure you get all the learnings and you hopefully set yourself up well for a second generation of the product. Again, it can be anywhere from 1-4 rounds of testing.