Comments on Card-Based Classification Evaluation

Sonya Leaman — Thu, 18 Oct 2007 14:36:30 GMT

This is an excellent technique to uncover any possible issues with labelling or grouping. When it comes to intranets with large amounts of content it can be extremely challenging to get labels right (considering an organisations’ culture, politics etc). So it is always good to also test in the context of a visual interface as this can give additional clues as to what is in each section, aiding a users understanding.

Considering your question evaluating unknown-item searching, I guess that a way to evaluate this is to measure user satisfaction through formal usability testing. So, if in a design there are contextual links and references to also see… then, are users pleasantly surprised at stumbling upon something? Do they leave feeling that it is a really useful site and they learnt more than they expected?

Just a thought, although, not completely scientific or new.

Ben Tremblay — Mon, 11 Jun 2007 22:33:51 GMT

Lateral – a historical note:
In 1975 I needed to index a large collection of resources. Different types (moves, photos, books, magazines, brochures, clippings) and a large number of topics. I devised a “card based” system and the project introduced me to taxonomy.

Does anyone remember COBOL card sorters? The system I used was almost as antique, yet captured the algorithmic essence of the matter: cards with holes punched along their edges, sorted by passing knitting needles through them. Rather elegant, it was … not fun to produce, not easy to maintain, but simple and something like fun for the user!

Javed Anjum Sheikh — Fri, 26 Jan 2007 14:53:42 GMT

How could it be managed without scenario ? Advice please

Nelson Rodríguez-Peña — Fri, 26 Jan 2007 14:52:47 GMT

Hi, I just noticed that the first image, the one with the caption Number the levels of your classification; then transcribe each one onto an index card. (Click to enlarge.), is not pointing to the correct location, it should be http://www.boxesandarrows.com/files/banda/card_based_clas… instead of http://www.boxesandarrows.com/archives/images/040703_card…

Chris Day — Fri, 26 Jan 2007 14:51:18 GMT

cheers for the prompt reply, donna!

thanks for the reassurance – i’d started putting together our first 3 or 4 results in exactly the way you describe, and if it’s good enough for the ‘inventor’, hey, it’s good enough for me *grin*

having thought back over this since posting yesterday, i think it’s probably the best way to go anyway, since having to do it ‘by eye’ forces you to think properly about the implications of the results, rather than relying upon some bit o’ software to spew out answers.

as an aside, one of our database guys (who also dabbles in usability stuff) is going to have a look and see if he can come up with something that could either
1. give us some assistance
or
2. give us some numbers (however meaningless) that we can throw at ROI-obsessed managers!

if we do get anything useful, i’ll post back up here. i guess i wasn’t 100% clear when i posted (at least, not when i read it back) – it’s not so much “magic analysis” i was looking for as just some kind of correlation analysis to give us a concrete figure for ‘confidence’.

hmmm – when i read that paragraph back, i’m not sure that’s clear either – curse the limitations of the written medium!

having just run through the first few users, this method still seems to work great – we’re getting several in next friday for an all day session of testing and usability work, and will be including this evaluation.

it’s pretty exciting stuff to be doing at last – our company (and especially our development dept) is pretty backwards when it comes to user testing, so it’s nice to be able to get some done after months of bitchin’ and moaning at managers!

cheers,
chris

Donna Maurer — Fri, 26 Jan 2007 14:51:18 GMT

I have done ‘by eye’ analysis on a hierarchy with 300 items and 100 scenarios (gosh that was interesting).

I still used the spreadsheet as described in the article, printed it out, stuck it together and looked at it on a big desk (or on the floor!). It would be worth probably shading alternate rows so you can follow a row or column across.

I’m very visual, so this works well for me. Let me know if it doesn’t work for you, and I’ll see if I can think of another way to do it

Chris Day — Fri, 26 Jan 2007 14:51:18 GMT

hi donna

this was an extremely useful article indeed! i work as an analyst programmer, who also gets to do some usability/IA type work if i push hard enough *grin*

we are currently in the process of providing a category browsing interface for our company’s knowledge base, and after having done an open card-sort exercise to assist with initial grouping ideas, we are now testing our proposed taxonomy/hierarchy using your method.

it seems to be working very well (and users certainly enjoy participating), though i do have one question..

the ‘by eye’ method of analysis as outlined in your example seems to work well with smaller numbers of scenarios, but the set we’re currently using has 50-odd cards to be sorted. i’ve tried to keep the number to a minimum, but we have a very wide range of categories and subjects to be validated.

is there any automated analysis method / software that you or anyone else might be able to recommend? for the open card-sort testing we used ezsort, which worked well, but it’s not applicable to ‘known item’ exercises such as this.

or is it the case that we’re just using too many scenarios for this method?

any feedback gratefully received, we’re still relatively new at this and eager to learn more ;o)

cheers,
chris

Donna Maurer — Fri, 26 Jan 2007 14:51:18 GMT

Thanks for all of the nice comments.

Kyle – you are correct – the method could be used on anything at least 2 levels deep, and I do get people to think out loud.

Actually I sometimes do something similar informally for just top-level stuff as well – when someone asks me where I would put something, I give them a related scenario and ask where they would look. Answers their own question usually.

Kyle Pero — Fri, 26 Jan 2007 14:51:18 GMT

This is a very innovative idea! I like how this method is similiar to user testing because you get to watch a user perform a task and listen to their feedback (I assume you encourage the user to think out loud).

I think this may be better than card-sorting. Lyle makes an excellent point that this exercise is natural to a user. It seems more realistic than telling a user to sort a pile of cards (something they normally don’t do online or offline).

It also seems like this method is much more applicable than card-sorting. I’ve found that card-sorting really requires a larger-scale site with lots of content. I assume this method could be used on small or large sites as long as they’re are at least two levels “deep”, right?

I hope to implement this new method soon. Again, very innovative method and a good article.

Kyle Pero — Fri, 26 Jan 2007 14:51:18 GMT

I hope to implement this new method soon. Again, very innovative method and a good article.

samantha bailey — Fri, 26 Jan 2007 14:51:18 GMT

Just yesterday I was in a meeting where we were talking about how to structure our usability testing so as to clarify the categorization issues from the UI issues—so your article is a bit of manna. We’re going to try this technique on a tricky section of our site that we’re re-organizing. I particularly like that you include your insights on ANALYZING the data, something that you don’t always see in card-sorting related articles.

Daniel Szuc — Fri, 26 Jan 2007 14:51:18 GMT

Very good read. Thank you.

Lyle - Croc O' Lyle — Fri, 26 Jan 2007 14:51:18 GMT

Donna,

Thanks for sharing this great technique! The examples were very helpful. The method seems to be a bit of a twist on a closed card sort—kind of a closed card sort mixed with a cognitive walkthrough.

It seems it would be very useful, especially with large taxonomies. A closed sort requires the participant to deal with lots of content examples, finding what category they’d put each into. Your technique is scenario based which is more natural – usually, most end users will use a hierarchy to find information rather than categorize content using it.

For more on cognitive walkthroughs, see:

Usability Evaluation with the Cognitive Walkthrough
http://www.acm.org/sigchi/chi95/proceedings/tutors/jr_bdy…

http://jthom.best.vwh.net/usability/cognitiv.htm

Donna Maurer — Fri, 26 Jan 2007 14:51:17 GMT

Yes, I mean that I only ask twice before moving on. Sorry for the confusion…

I think I could manage to do this with one person, but I have always been lucky and had someone to give me a hand. The tricky bit with doing it alone would be recording the scenario & choice and re-bundling the cards for the next scenario – this may break the flow of the discussion a bit (but probably not much). It would be tricky to note comments at the same time.

Chad Lundgren — Fri, 26 Jan 2007 14:51:17 GMT

Thanks for the great practical article. One thing that threw me a bit was this section “During a scenario, if a participant looks hesitant, ask him if he’d like to choose again. I only offer two choices at any level because I want to see where the person would look first, not get him to hunt through the classification endlessly.”

The first time I read it, I thought you meant you offered only two choices in the hierarchy, even if there were something like six. Now I’m thinking you mean, you only asked twice before moving on.

I’m wondering one other thing: what about one person running this card-based test? I know with usability testing, that is not ideal even for a discount style test, since so much is going on, but does it make more sense for this technique?