Boxes and Arrows: Comments by Dave OBrien

Dave OBrien — Wed, 09 Dec 2009 08:10:58 GMT

Matty:

1. I definitely agree that task phrasing is critical in tree tests, especially the avoidance of “giveaway” words. The problem we’ve seen with more than 10 tasks is that participants learn the tree structure by browsing it during early tasks, making certain later tasks easier. While randomising the task order helps, there’s still a learning effect that unnaturally skews the results (especially if your typical users are not habitual or frequent visitors). If the tree is small, we go with about 8 tasks; if it’s very large, we may go as high as 12. But our findings are admittedly limited so far – what’s needed is more experimentation with the technique to see what its real limits are.

2.) Showing user paths graphically – Yes, results that are more graphical (like what you suggest and a few other ideas we’re playing with) are definitely on the wish list. Being able to show the tree and highlight its strong areas and weak areas (either per task or across the whole test) would be a great way to show what’s important at a glance.

3.) Playback of incorrect decisions – That’s one we haven’t thought of (to my knowledge), but I like its forensic approach – that would help tease out the “why” that we’re currently missing in Treejack and several other online tools.

Good thoughts here. Thanks!

Dave OBrien — Sat, 19 Dec 2009 15:53:08 GMT

Brian: Love the idea of using a nested folder structure in a file browser – use the simplest tool that gets you the answers you’re looking for. Yes, you give up some sophistication, but you may not always need that.

James: I ran some paper-based tests before we had Treejack, partly to get results, and partly to get more familiar with the technique before designing a tool to automate it. While I found the paper version more tedious to prepare than its online counterpart, I really liked how I was able to hear the participants’ thought processes (i.e. think aloud) and ask about their more curious departures from the “correct” paths. I think it might turn out like card sorting, where both paper and online methods will get used depending on what the tester is looking for. What you mention about doing several rounds in 2 days – exactly! That kind of quick turn-around is what I like best about tree testing.

Dave OBrien — Wed, 09 Dec 2009 08:57:22 GMT

Stephanie: That idea of asking participants what they expect to see before doing the test – I’ve done that during standard usability tests, but usually when we didn’t have content behind a clicked link (“We don’t have that page available, but what would you expect to see?”), or at the end of a test, for links that they never clicked (“Tell me more about this one.”).

Probing expectations before the test starts – I agree that this can reveal useful information, but I also wonder if it affects the results of the tasks that follow? That is, if we prompt the participant to think a bit about each top-level heading up front (before they have a task to do), might this improve their performance (or even hurt it, depending on how well their preconceptions matched up with their later thinking)? That would be an interesting study to do.

Dave OBrien — Fri, 25 Dec 2009 18:33:01 GMT

Tedd:

Interesting idea about building dynamic menus based on popularity. Do you know any sites that have tried it?

If the site had one main type of user, I can see this working well. If there were several types of users, I wonder if that causes problems – that is, the popularity of certain sections and pages may vary greatly between user type A and user type B.

If the site had a high return-visitor rate (e.g. an intranet, where a given user visits it daily), we could also try a per-user popularity rating, so my site would have a different organisation than yours. Hmm…

Dave OBrien — Fri, 25 Dec 2009 18:42:17 GMT

Rene:

In general, we try to get at least 30 users of a given type – that seems to be the general statistical number to see medium-sized effects in any study.

However, I prefer to get about 100 users per type if I can. That way, you can spot outliers more easily. For example, if you test 30 people, and 1 chooses a given incorrect topic, that can be safely ignored. But what about 2 or 3 users? At some point those numbers become meaningful.

If I test 100 users, now I can safely ignore topics that get small numbers of hits (say, less than 5). Every time we run a tree test, we get a few seemingly bizarre choices, and it’s good to be able to ignore them with confidence.

Reusing users: Just like traditional usabilty testing, we aim to use fresh users each time, for just the reason you mention.

Time spent recruiting: For this type of unmoderated testing, where the participant can do it remotely on their own time, it’s usually not hard to recruit users. Usually our clients have lists that we can use. Like usability testing, we may also do discount testing with surrogate users, but we always avoid people who know too much (e.g. the project team) or are too different from our target users.

Hope this helps!