Steve Baty — Wed, 16 May 2007 23:42:11 GMT

Iain,

From a statistical stand-point you’d be looking to use binomial point estimators and confidence intervals to estimate the ‘real’ (i.e. population) values for 1st-time completion, overall success etc. 15-30 people will give you a confidence interval that is relatively broad. I’ve worked some of these out previously for reference:

Task with measured ‘success’ rate of 2/3 (66.67%): 47.7% – 81.9% with an expected success ratio of 64.8% (30 users)
Task with measured ‘success’ rate of 4/5 (80%): 61.44% – 91.75% with an expected success ratio of 76.6% (30 users)
Task with measured ‘success’ rate of 3/4 (75%): 56.82% – 87.82% with an expected success ratio of 72.32% (32 users)

What that means is that, if you’re tests gave you an 80% measure of ‘success’ (whatever that was), then you would expect the user population as a whole to perform the same task with a success rate of 76.6%, with a 99% confidence interval (i.e. 1/100 chance that it really lies outside the range) of 61.44% at the low end, and 87.82% at the high end. To put that another way – there’s a half of a percentage point chance that users will actually fare worse than 61.44% if you measured 80% during the test.

I would also reinforce your point about not using the same users again for future iterations – you’d basically invalidate the results – statistically speaking – in that event.

Nicely written article, btw.

Steve Baty — Sun, 30 Mar 2008 10:32:39 GMT

Daniel,

In my recent article on UXMatters I argued the merits of using customer interaction channels as a source of design research. Touch-points such as call centres, customer enquiry forms, feedback forms &etc provide a rich source of insight into unmet customer needs or pain points.

Your points about the relative cost & efficiency of these sources of user insight are well made.

Cheers
Steve

Boxes and Arrows: Comments by Steve Baty