Matthias Schreck — Mon, 21 Dec 2009 09:04:28 GMT

Hi Dave,

an excellent article, thank you very much for discussing this in detail. I often find that information architects are a little too complacent when they only do a card-sort in the beginning and then base all their later decisions on the assumptions gained from this initial exercise. Your approach is a nice evaluation tool, and I would love to see it!

There are a few points I wanted to raise:
1.) I often find that with information hierarchies, even a subtle difference in the way you phrase the task can lead to very different outcomes. A verb or adjective in another place can jog a participant’s memory, suddenly “solving” a problem… that’s why I’m a little cautious when you said that 10 tasks were enough. I know that participants get bored really quickly with these tasks, but only 10 tasks can potentially skew the results by relying more on a clever way of posing the question, rather than a good way of presenting the answer. I hope this makes sense.
2.) I think it would be great if your tool could show the results graphically. The great thing about tree hierarchies is that they are so easy to display – and if you then mapped user paths (for example with different colours) over such a tree hierarchy graphic, it might be easier to come to some conclusions. For example, imagine that 5 out of 20 users went down the wrong path for a certain task, but they all went down the same wrong path, the result graphic could show this effectively. The IA could then make a decision to either re-model the structure, or decide on inserting a cross-link in the form of “You might have been looking for this…”
3.) Another useful piece of functionality for your tool would be a register for the “points of incorrect decisions”. Once users are finished with the whole test, the program could display the screens where users went down a different path than what the information architecture had laid out for them. It could show that screen, circle the user decision as “This is what you chose”, and circle the “correct” one as “This is what we had in mind”. You could then ask users what didn’t work for them with your assumptions, and they might actually come up with some pretty good reasons like “I didn’t know what your word meant” or some rather mundane ones, like “Oh, I didn’t see that one.” This might be possible for online tests, too, but I’m rather thinking about the usefulness for a facilitator in a face-to-face interview.

I hope you don’t misunderstand me – I’m really excited about your tool, and I’m just thinking that it has a huge potential to convey even more information about user decisions :-)

Cheers,
Matty

Boxes and Arrows: Comments by Matthias Schreck