Comments on Tree Testing

Dave OBrien — Fri, 25 Dec 2009 18:42:17 GMT

Rene:

In general, we try to get at least 30 users of a given type – that seems to be the general statistical number to see medium-sized effects in any study.

However, I prefer to get about 100 users per type if I can. That way, you can spot outliers more easily. For example, if you test 30 people, and 1 chooses a given incorrect topic, that can be safely ignored. But what about 2 or 3 users? At some point those numbers become meaningful.

If I test 100 users, now I can safely ignore topics that get small numbers of hits (say, less than 5). Every time we run a tree test, we get a few seemingly bizarre choices, and it’s good to be able to ignore them with confidence.

Reusing users: Just like traditional usabilty testing, we aim to use fresh users each time, for just the reason you mention.

Time spent recruiting: For this type of unmoderated testing, where the participant can do it remotely on their own time, it’s usually not hard to recruit users. Usually our clients have lists that we can use. Like usability testing, we may also do discount testing with surrogate users, but we always avoid people who know too much (e.g. the project team) or are too different from our target users.

Hope this helps!

Dave OBrien — Fri, 25 Dec 2009 18:33:01 GMT

Tedd:

Interesting idea about building dynamic menus based on popularity. Do you know any sites that have tried it?

If the site had one main type of user, I can see this working well. If there were several types of users, I wonder if that causes problems – that is, the popularity of certain sections and pages may vary greatly between user type A and user type B.

If the site had a high return-visitor rate (e.g. an intranet, where a given user visits it daily), we could also try a per-user popularity rating, so my site would have a different organisation than yours. Hmm…

Rene Welter — Fri, 25 Dec 2009 18:42:25 GMT

hi dave,

great article! i had some really good ideas while i was reading your article. so i am looking forward to solve a problem i was carrying with me for some time. but i was thinking about some questions also. its about test persons.

i was thinking about your conclusion: doing a better IA, not a different! (btw: a great and easy argument!). how many persons do you test by asking 10 cases? i assume you need a high enough number of test persons to have significant and reliable findings.

if you want to have a comparison between the old structure and a new one, do you choose the same persons? i think you need different groups of test persons, because the first tested group remembers the ols structure and this would falsify the findings…

how much time do you spend in finding test persons that are part of the defined target group? we often use colleague for “discounter testing”.

cheers,
rene

tedd sperling — Fri, 25 Dec 2009 18:42:31 GMT

Interesting article for testing trees.

The tree organization testing problem might be solved by implementing a splay binary tree algorithm approach. An example is shown on my site (Binary-Tree Demo), but the application only runs on Macs.

http://sperling.com/freeware.php

In any event, I propose a “splay menu system” that would have the most popular links appearing in order (top/down, left/right, whatever). Subsequent menus would appear likewise but further down the link tree.

The upside of using this technique would be the most popular pages would appear highest on the menu system. The downside is that the menu system would not be static, but instead always changing dependant upon visitor’s interest and selection.

I would guess that visitor’s interest will focus on certain topics and lessen the amount of menu change in a very short time. However, when new topics are provided, then the menu system would again change to adapt.

This approach would be automatic and allow the visitor to do the menu testing.

Cheers,

tedd

Yuanlin Wu — Wed, 10 Mar 2010 02:53:32 GMT

very inspiring!
we’re considering using this tool for our incoming study~

brooke baldwin — Fri, 25 Dec 2009 18:42:39 GMT

dan
thanks for the article. i started reading it before i looked at your bio and was thinking to myself, “doesn’t this guy know about treejack?”

ha. i guess so.

i’ve started using the tool recently and am pretty pleased with it. keep up the good work.

Dave OBrien — Wed, 09 Dec 2009 08:57:22 GMT

Stephanie: That idea of asking participants what they expect to see before doing the test – I’ve done that during standard usability tests, but usually when we didn’t have content behind a clicked link (“We don’t have that page available, but what would you expect to see?”), or at the end of a test, for links that they never clicked (“Tell me more about this one.”).

Probing expectations before the test starts – I agree that this can reveal useful information, but I also wonder if it affects the results of the tasks that follow? That is, if we prompt the participant to think a bit about each top-level heading up front (before they have a task to do), might this improve their performance (or even hurt it, depending on how well their preconceptions matched up with their later thinking)? That would be an interesting study to do.

Dave OBrien — Sat, 19 Dec 2009 15:53:08 GMT

Brian: Love the idea of using a nested folder structure in a file browser – use the simplest tool that gets you the answers you’re looking for. Yes, you give up some sophistication, but you may not always need that.

James: I ran some paper-based tests before we had Treejack, partly to get results, and partly to get more familiar with the technique before designing a tool to automate it. While I found the paper version more tedious to prepare than its online counterpart, I really liked how I was able to hear the participants’ thought processes (i.e. think aloud) and ask about their more curious departures from the “correct” paths. I think it might turn out like card sorting, where both paper and online methods will get used depending on what the tester is looking for. What you mention about doing several rounds in 2 days – exactly! That kind of quick turn-around is what I like best about tree testing.

Dave OBrien — Wed, 09 Dec 2009 08:10:58 GMT

Matty:

1. I definitely agree that task phrasing is critical in tree tests, especially the avoidance of “giveaway” words. The problem we’ve seen with more than 10 tasks is that participants learn the tree structure by browsing it during early tasks, making certain later tasks easier. While randomising the task order helps, there’s still a learning effect that unnaturally skews the results (especially if your typical users are not habitual or frequent visitors). If the tree is small, we go with about 8 tasks; if it’s very large, we may go as high as 12. But our findings are admittedly limited so far – what’s needed is more experimentation with the technique to see what its real limits are.

2.) Showing user paths graphically – Yes, results that are more graphical (like what you suggest and a few other ideas we’re playing with) are definitely on the wish list. Being able to show the tree and highlight its strong areas and weak areas (either per task or across the whole test) would be a great way to show what’s important at a glance.

3.) Playback of incorrect decisions – That’s one we haven’t thought of (to my knowledge), but I like its forensic approach – that would help tease out the “why” that we’re currently missing in Treejack and several other online tools.

Good thoughts here. Thanks!

Mia Northrop — Wed, 09 Dec 2009 07:28:01 GMT

I’ve used Treejack for several projects now and have found it very insightful. I find it complements card sorting activities very well and also gives a good indication of potential cross-references between pieces of content. I love the fact that by using a consistent set of tasks and simply revising the tree you can measure improvement quantifiably.

Stephanie Heacox — Wed, 09 Dec 2009 07:28:04 GMT

I have done similar exercises in the past, and I think this would be a much more efficient tool for that purpose. In addition to the task-based exploration, I also begin each testing session by simply asking the participant to look at each primary (and sometimes, secondary) navigation category and tell me what they *think* they would find underneath each. it is important to do this before they have developed any preconceived notions as a result of the testing.

James Robertson — Sat, 19 Dec 2009 15:52:56 GMT

Hi Dave, Donna developed this technique when working with us, and we’ve therefore been using it for many years.

While an online tool certainly makes life easier, it’s also very simple to use this technique with handwritten filing cards. An hour to create the cards, and you can fit in several rounds of testing in just two days. Makes a huge difference in refining the IA before moving on to the wireframes…

Brian Hoffman — Tue, 08 Dec 2009 09:50:29 GMT

I’ve used a similar technique in the past and also called it tree testing. Since I wasn’t aware of any software tool for it, I created the tree using a set of Windows nested folders. It was easy to create, cost nothing, and was an interface users were familiar with and found easy to navigate. The only downside was that it took more effort to analyze the data than your tool would. However, I approached it as more of a qualitative method than quantitative, so I didn’t test large numbers of users at once and did it iteratively. Given the choice between what I did and your tool, I’ll probably use your tool next time. But if anyone is on a limited budget, they may find the folder method an acceptable substitute.

Jussi Pasanen — Thu, 17 Dec 2009 14:51:34 GMT

Hi Dave, and thanks for a great article!

We’ve used Treejack since its initial launch early this year and have had good experiences with it, some of which we shared in April:

Using Treejack to test your website structure
http://www.volkside.com/2009/04/using-treejack-for-ia-tes…

There is also another tool in the market called C-Inspector that is specifically designed for tree testing. We posted an article looking at C-Inspector in September, including comparisons with your Treejack tool:

Test your information architecture using C-Inspector
http://www.volkside.com/2009/09/test-your-ia-using-c-insp…

Both articles discuss screen-based IA testing more generally, too.

Cheers, Jussi

Ps. Matthias makes some great points above that are worth taking into consideration in future development.

Matthias Schreck — Mon, 21 Dec 2009 09:04:28 GMT

Hi Dave,

an excellent article, thank you very much for discussing this in detail. I often find that information architects are a little too complacent when they only do a card-sort in the beginning and then base all their later decisions on the assumptions gained from this initial exercise. Your approach is a nice evaluation tool, and I would love to see it!

There are a few points I wanted to raise:
1.) I often find that with information hierarchies, even a subtle difference in the way you phrase the task can lead to very different outcomes. A verb or adjective in another place can jog a participant’s memory, suddenly “solving” a problem… that’s why I’m a little cautious when you said that 10 tasks were enough. I know that participants get bored really quickly with these tasks, but only 10 tasks can potentially skew the results by relying more on a clever way of posing the question, rather than a good way of presenting the answer. I hope this makes sense.
2.) I think it would be great if your tool could show the results graphically. The great thing about tree hierarchies is that they are so easy to display – and if you then mapped user paths (for example with different colours) over such a tree hierarchy graphic, it might be easier to come to some conclusions. For example, imagine that 5 out of 20 users went down the wrong path for a certain task, but they all went down the same wrong path, the result graphic could show this effectively. The IA could then make a decision to either re-model the structure, or decide on inserting a cross-link in the form of “You might have been looking for this…”
3.) Another useful piece of functionality for your tool would be a register for the “points of incorrect decisions”. Once users are finished with the whole test, the program could display the screens where users went down a different path than what the information architecture had laid out for them. It could show that screen, circle the user decision as “This is what you chose”, and circle the “correct” one as “This is what we had in mind”. You could then ask users what didn’t work for them with your assumptions, and they might actually come up with some pretty good reasons like “I didn’t know what your word meant” or some rather mundane ones, like “Oh, I didn’t see that one.” This might be possible for online tests, too, but I’m rather thinking about the usefulness for a facilitator in a face-to-face interview.

I hope you don’t misunderstand me – I’m really excited about your tool, and I’m just thinking that it has a huge potential to convey even more information about user decisions :-)

Cheers,
Matty