Boxes and Arrows: Comments by Iain Barker

Iain Barker — Thu, 03 May 2007 01:04:00 GMT

Chad,
So far I’ve always tried to avoid making the user focus too much on their success/failure – I’ve just given them two attempts, but it sounds like what you are suggesting could work. Let me know how it goes.

Iain Barker — Thu, 03 May 2007 22:26:42 GMT

Thanks Dawn – hopefully that will be corrected very soon.

Iain Barker — Mon, 07 May 2007 01:25:36 GMT

Mike,
Thanks for your comment. So far I’ve only used the technique to provide data to support what would otherwise be an entirely subjective assessment of the success/failure of the classification system.

The limited budgets/timescales I typically work within have only enabled me to run one or two days of sessions within each iteration. This means I’ve spoken to between 15 to 30 people, each attempting 10 to 15 tasks. I am not a statistician, but I am guessing that these numbers won’t produce statistically valid data (I am always careful to point this out when using this technique to communicate to clients). Sadly I can’t advise as to what kind of numbers you would need to involve to provide statistically valid data – maybe someone else can help there?

As for repeatedly using the same people, I have always had the luxury of a large pool of potential users, and have always used new users for each iteration. Obviously this could cause some to question the comparison between iterations, but I’ve never had this problem with the clients I’ve worked with. Even if a client wants to include some repeat participants, I would always argue for some new participants with each iteration.

My preference for using new participants is primarily so they don’t take ownership of any solutions/recommendations they may give during the sessions. Also I find that from a political perspective, it is often better to be able to say that 100 people participated in the creation of the classification system, rather than just 15.

I am sure that a statistician may well be wincing at my response – if one is out there reading and wants to recommend how a more rigourous approach could be applied I’d like to hear from them.

Iain Barker — Wed, 23 May 2007 01:42:07 GMT

@Miles – I played around with using conditional formatting, but with my limited Excel powers (!) I was unable to make this work is the correct answer could in multiple locations, i.e. the correction location cell has multiple right answers seperated by commas. I suppose there are some clunky workarounds for this (using multiple cells), but if you know of a more clever way please do share!

@Simon – Thanks for your comments. I’ll try to tidy up a version of the spreadsheet and make it available in due course.