Comments on Measuring the Success Of a Classification System

Iain Barker — Wed, 23 May 2007 01:42:07 GMT

@Miles – I played around with using conditional formatting, but with my limited Excel powers (!) I was unable to make this work is the correct answer could in multiple locations, i.e. the correction location cell has multiple right answers seperated by commas. I suppose there are some clunky workarounds for this (using multiple cells), but if you know of a more clever way please do share!

@Simon – Thanks for your comments. I’ll try to tidy up a version of the spreadsheet and make it available in due course.

Miles Rochford — Sun, 27 May 2007 10:26:26 GMT

Another variation I have also tried is using a wireframe of the home page combined with index cards for lower levels of the hierarchy. The wireframe included navigation for users to ‘jump’ a level deeper, so a rigid top down test would not reflect the way in which users navigated on the site. This approach allowed us to explore the combination of taxonomy and navigation in a single test (kind of a mix between paper prototyping and Donna’s card-based classification evaluation).

- Miles.

Miles Rochford — Sun, 27 May 2007 10:26:12 GMT

One suggestion I would add is the use of ‘conditional formatting’ in Excel to colour code the cells, both in terms of whether the responses were ‘correct’ and in terms of the percentage of correct responses. This saves a LOT of time and reduces the risk of error.

I’ve also used a variation on Donna’s original technique which allowed me to highlight the areas where users experienced difficulty in classifying ‘correctly’ and what their alternative classifications were. This assisted greatly in improving the results in subsequent evaluations.

- Miles.

Simon Johnson — Fri, 18 May 2007 16:02:44 GMT

Iain,

I must say that this is a wonderfully clear representation of data. I am sure Tufte would be proud of you.

To save us all the pain of reinventing the wheel, is there any chance you could provide this spreadsheet for download? Please excuse me if I missed the link if it already exists.

Steve Baty — Wed, 16 May 2007 23:42:11 GMT

Iain,

From a statistical stand-point you’d be looking to use binomial point estimators and confidence intervals to estimate the ‘real’ (i.e. population) values for 1st-time completion, overall success etc. 15-30 people will give you a confidence interval that is relatively broad. I’ve worked some of these out previously for reference:

Task with measured ‘success’ rate of 2/3 (66.67%): 47.7% – 81.9% with an expected success ratio of 64.8% (30 users)
Task with measured ‘success’ rate of 4/5 (80%): 61.44% – 91.75% with an expected success ratio of 76.6% (30 users)
Task with measured ‘success’ rate of 3/4 (75%): 56.82% – 87.82% with an expected success ratio of 72.32% (32 users)

What that means is that, if you’re tests gave you an 80% measure of ‘success’ (whatever that was), then you would expect the user population as a whole to perform the same task with a success rate of 76.6%, with a 99% confidence interval (i.e. 1/100 chance that it really lies outside the range) of 61.44% at the low end, and 87.82% at the high end. To put that another way – there’s a half of a percentage point chance that users will actually fare worse than 61.44% if you measured 80% during the test.

I would also reinforce your point about not using the same users again for future iterations – you’d basically invalidate the results – statistically speaking – in that event.

Nicely written article, btw.

Iain Barker — Mon, 07 May 2007 01:25:36 GMT

Mike,
Thanks for your comment. So far I’ve only used the technique to provide data to support what would otherwise be an entirely subjective assessment of the success/failure of the classification system.

The limited budgets/timescales I typically work within have only enabled me to run one or two days of sessions within each iteration. This means I’ve spoken to between 15 to 30 people, each attempting 10 to 15 tasks. I am not a statistician, but I am guessing that these numbers won’t produce statistically valid data (I am always careful to point this out when using this technique to communicate to clients). Sadly I can’t advise as to what kind of numbers you would need to involve to provide statistically valid data – maybe someone else can help there?

As for repeatedly using the same people, I have always had the luxury of a large pool of potential users, and have always used new users for each iteration. Obviously this could cause some to question the comparison between iterations, but I’ve never had this problem with the clients I’ve worked with. Even if a client wants to include some repeat participants, I would always argue for some new participants with each iteration.

My preference for using new participants is primarily so they don’t take ownership of any solutions/recommendations they may give during the sessions. Also I find that from a political perspective, it is often better to be able to say that 100 people participated in the creation of the classification system, rather than just 15.

I am sure that a statistician may well be wincing at my response – if one is out there reading and wants to recommend how a more rigourous approach could be applied I’d like to hear from them.

Mike Murphy — Fri, 04 May 2007 13:20:23 GMT

Iain, Very nice article. I work in the Process Improvement arena, specifically with CMMI, and capturing measures and managing to them is integral to achieving high level certifications. Measuring the improvement of websites, and proving it, can be tricky. Your spreadsheets provide a simple and understandable way of showing this.

How many people do you interview (ie. is there a magic number), and is it important that the same people be interviewed throughout the process?

Iain Barker — Thu, 03 May 2007 22:26:42 GMT

Thanks Dawn – hopefully that will be corrected very soon.

Dawn Buie — Thu, 03 May 2007 22:25:53 GMT

Hey Ian – did you accidentally repeat the first photo of the index card after this text:

2. On another set of index cards, write and number around 15 common information-seeking tasks. One task per index card.

Iain Barker — Thu, 03 May 2007 01:04:00 GMT

Chad,
So far I’ve always tried to avoid making the user focus too much on their success/failure – I’ve just given them two attempts, but it sounds like what you are suggesting could work. Let me know how it goes.

Chad Wingrave — Fri, 04 May 2007 13:04:49 GMT

Nifty. One additional success metric to apply would be the levels of the hierarchy traversed to get to each item. I.e., average “clicks” to finish each scenario in each redesign. The success rate and “click” count together would really be compatible with each other. I’ll have to give this a go in the future.

henrik persson — Fri, 27 Apr 2007 20:45:26 GMT

at first glance this looks like a great way of improving your ia iteratively. but do the results get better because you adapt to the test users or the other way around?

Ryan Trembath — Fri, 27 Apr 2007 16:15:49 GMT

thank you so much for outlining this. the final spreadsheet really helps show clients the value of the different iterations. I will apply my own flavor to it and report the results!

Michael Beavers — Fri, 27 Apr 2007 13:44:50 GMT

This is great…I like that you also keep testing results visually represented with only three colors. Any more than that would require too much thinking and analysis (think of the American homeland defense color scheme). I hope practitioners try your method and report back in this comment string.

Donna Maurer — Fri, 04 May 2007 21:08:57 GMT

A good enhancement & neat visualisation of outcomes!