Boxes and Arrows: Comments by Miles Rochford

Miles Rochford — Sun, 27 May 2007 10:26:12 GMT

One suggestion I would add is the use of ‘conditional formatting’ in Excel to colour code the cells, both in terms of whether the responses were ‘correct’ and in terms of the percentage of correct responses. This saves a LOT of time and reduces the risk of error.

I’ve also used a variation on Donna’s original technique which allowed me to highlight the areas where users experienced difficulty in classifying ‘correctly’ and what their alternative classifications were. This assisted greatly in improving the results in subsequent evaluations.

- Miles.

Miles Rochford — Sun, 27 May 2007 10:26:26 GMT

Another variation I have also tried is using a wireframe of the home page combined with index cards for lower levels of the hierarchy. The wireframe included navigation for users to ‘jump’ a level deeper, so a rigid top down test would not reflect the way in which users navigated on the site. This approach allowed us to explore the combination of taxonomy and navigation in a single test (kind of a mix between paper prototyping and Donna’s card-based classification evaluation).

- Miles.

Miles Rochford — Wed, 23 May 2007 09:11:44 GMT

@Iain – the trick is to use a better separator than a single comma, and make sure your list has starting and ending separators too, like ”,1.1.1,4.2.3,”. They you just check to see if the cell contains the users answer (prefixed and suffixed with commas – to avoid 1.1.1 being matched against 11.1.1). I find it’s a lot easier if you record the data on one sheet and analyse (using formulae) on another.

You can also get a little more sophisticated and do ‘closeness’ matching to assess whether the user was in a nearby section, so if the user selected ‘1.1.3’ but the correct answer was ‘1.1.2’, you can look for ’,1.1,’ (right section, wrong level) or ’,1.1.’ (right level, wrong section) and consider it a ‘closer’ classification than ‘2.8’, say. I’ve found that if users are close ‘enough’, the presentation of the classification on the site can minimise issues with discovery.