Comments on Don't Test Users, Test Hypotheses

Paula Thornton — Fri, 26 Jan 2007 14:51:47 GMT

Great ‘crossover’ piece. While most B&A articles are of little interest for my needs (classification, card sorts, et. al. —all specific to one practical dimension of IA), this was a good summarization of tidbits floating around in my head, but not in any way confimed/solidified. Thanks for organizing my mind—and for contributing to good perspective on activities necessary for any design work.

avi — Fri, 26 Jan 2007 14:51:47 GMT

Brian, two reponses to the issue you raise: 1. framing hypotheses, 2, relating to teams.

1. You can always make your hypothesis a positive one: e.g. “the user *will* find the pricing page.” By doing that you are making explicit the hypothesis that is embedded in the design. Then you all know what you are looking for, and if the user does not find it, the design hypothesis was not proven. Your finding then suggests where rework is needed. (I didn’t mean to imply that you only test for problems you expect to see; rather that you test to see if the users accomplish what they are expected to.)

2. The dynamics of working with teams always requires tact and good communication skills, and will vary, as you know, according to the nature of the your relationship to the team. If it is a close one, then you may be able to get the team involved in spotting the potential problems themselves, essentially helping to build those hypotheses. In a more external, consulting relationship, I try to get the team to define what ‘success’ is for them. Their definition helps me form hypotheses that help me to structure my work, even if I don’t share actual hypotheses with them before them. That way, when I do the testing and report findings, I’m looking at issues that are important to them.

p.s. I think the issue you identify—of the usability person being perceived as a ‘smarty-pants’ know-it-all—is a real problem, and wonder what others do to address it.

Brian R. Krause — Fri, 26 Jan 2007 14:51:47 GMT

Interesting…I’m wondering if you share the hypotheses with the rest of the development team before testing. I think if I told my clients what I was going to find and then proceeded to find it, they would think I was being a smart aleck while I was presenting the hypotheses (“The product is so obviously bad from my looking at it, why are we even testing it?”), looking only to confirm my preconceived ideas while testing (fostering suspicion that the test was designed to make the marketing VP’s ideas look bad), and having wasted their time and money when I told them that I discovered what I said I’d find (the less numerous surprise discoveries seeming less valuable among all the stuff that was already suspected). Consider that the rest of the team usually thinks a system that they can figure out is OK for everybody else, too, and that it’s much easier to argue with the tester’s heuristics and intuitions than it is with reactions of real people discovered through testing.

I think of usability testing as a way to show other members of the team in human terms that design decisions have consequences—delight and productivity vs. confusion and frustration. I think too much of a preview would be upstaging this important phase. In other words, I think it’s more valuable to make the problems with the design that lasting impression than to show that we are so smart we knew they were there all along.

Christopher Fahey — Fri, 26 Jan 2007 14:51:47 GMT

Great article. When we put a prototype through testing, it’s funny how often we talk about it as if we’re testing the users when in fact we are testing the preconceptions we put into our designs.

When designing, we think “this oughtta work!”, or “I wonder if this will work?”. Before testing, we can ask the IA and GUI design team members what aspects of their design they argued about the most, or which aspects they felt diverged from widely-accepted best practices. Writing these down is easy – these are, in effect, the designers’ hypotheses. If testing only had one goal, it would be to confirm or deny the success of these design hypotheses. Perhaps it’s just obvious to many in the community, but somehow I feel that this article proposes a fresh way of looking at the purpose of testing.

avi — Fri, 26 Jan 2007 14:51:47 GMT

You can use a hypothesis-based approach without stipulating tasks. Depends on the hypothesis. And you can even listen to users (in focus groups, in think-alouds, in the previous testing session) to help you form your hypothesis. That’s what I like to do, because it can show up where users and design concepts diverge. I just like to go back and watch the users and see if they do as they say.
AleX… I guess some of the difference in approaches, between what Mark Hurst suggests and I propose,is due to the whens and the whys… when in the development cycle the testing is done and why the research is being conducted.

MM — Fri, 26 Jan 2007 14:51:47 GMT

Rather than hypotheses, I typically define a set of key questions that I hope to answer through any study. These questions help me focus the study and determine the appropriate methodologies I should use to answer the questions. They’re also a useful way to get agreement from the team about the goal of the study. I think that organizing the research around questions instead of hypotheses helps avoid any appearance of bias by the researcher.

ji — Fri, 26 Jan 2007 14:51:47 GMT

Great article!
Yes, prototypes we design are usually based on hypotheses…hopefully based on some good experience and knowledge/systematic approach gained from various resources (requirements, etc.).........

I also wanted to point out various degrees of hypotheses in prototype designs during usability testing…....I wonder if anybody has a good recommenation or comment on getting our hypotheses little closer to the solution…...

thanks.