Comments on Extreme User Research

Daniel Lafreniere — Fri, 28 Mar 2008 00:05:25 GMT

Wow! Thank’s to all of you for your great comments =).

@Cennydd: I do agree with you that there is a risk of bias. As a matter of fact, there is a bias, but, by interviewing more than one surrogate user per job profile, we reduce it is some way. Of course, when it’s possible, I do prefer interviewing + observing real users in their environment (work/home). However, in the projects I was involved in the past years, doing a “real” ethnographic study would have been impossible for different reasons (cost, time and users access).

@Jonathan: you said “When designing a web site, it would seem a bit odd not to account for the obvious possibility that people who call the call center are different from people who prefer to use web.” Absolutely. However, it gives us a good indication of what is mostly needed/asked by users. Questions asked over the phone or email should be aswered on the Website (well, most of the time ;-).

@Ken: you said “I believe this is simply one tool in our tool belt. I don’t think any single source of data is sufficient to make any remotely important design decisions.” Absolutely. I usually combine this approach with web analytics (when there is an existing website), call log analysis, etc.

@Matthew: you said “I still find that I run into barriers of time cost and scope. What have you found to be effective in overcoming these types of concerns?” Well, doing this kind of user research takes usually no more than 20 days (this includes interviews, analysis + report), which is not expensive. The cost of NOT doing user research greatly exceeds the cost of doing it. User research pays by itself just by knowing on what we must focus (what users want/needs). Also, having “hard” facts, as opposed to suppositions, reduces those never-ending discussions during design sessions.

Ken Seto — Thu, 27 Mar 2008 18:16:02 GMT

To add to my point, I believe this is simply one tool in our tool belt. I don’t think any single source of data is sufficient to make any remotely important design decisions.

Ken Seto — Thu, 27 Mar 2008 18:07:53 GMT

Great article Daniel! I think it’s important to be aware of interesting alternatives to the ideal methods of gathering user data. Even though there is some danger involved in relying on this second hand information, the reality is that in most companies, there is still a lot of resistance to budgeting any significant amount of resources towards real user testing.

For myself, this is something I am already practicing to some degree and I find it to be a valuable tool in a pinch.

Thanks again for nice, short, insightful article.

Jonathan Baker-Bates — Thu, 27 Mar 2008 18:04:01 GMT

Oh, and may I add my thanks for writing a mercifully short article. Let’s hope it starts a trend.

Jonathan Baker-Bates — Thu, 27 Mar 2008 22:58:05 GMT

While I think that user research by proxy isn’t a bad idea per se, it seems to me that the article isn’t quite egalitarian enough. When designing a web site, it would seem a bit odd not to account for the obvious possibility that people who call the call center are different from people who prefer to use web. The approach would seem to demand some corresponding research of other channels before making hard design decisions – an online survey, perhaps, seeing as we’re not too bothered by bias.

Matthew Doty — Thu, 27 Mar 2008 17:28:34 GMT

Thank you for speaking out on this! When it comes to this type of user research, I sometimes feel like a lone voice crying in the wilderness. Over the last 5 years, I’ve been trying to convince clients that quality user data IS within their reach thought this type of research. I still find that I run into barriers of time cost and scope. What have you found to be effective in overcoming these types of concerns?

Chris Collingridge — Thu, 27 Mar 2008 17:25:03 GMT

Firstly, the length of this article is great – short and snappy. Much appreciated!

There’s a sliding scale of user involvement, in my opinion. At one end we have an idealised process with unlimited time and resources, in which we conduct ethnographic research across truly representative samples of users or potential users, and continue this engagement throughout product development. At the other end is grabbing someone in a corridor and showing them a screen. This method definitely falls towards the latter end on the spectrum, but utilising information within the organisation is an important area to highlight.

Most organisations that have spent some time developing products for a specific domain do actually contain a lot of good domain information. Using all the information from people who contact customers is a really valuable way to increase the degree of proxy-user input, possibly enabling you to focus your limited time and resources on the areas where real users truly are key.

David Royer — Fri, 28 Mar 2008 10:04:13 GMT

I think we can all agree (author included) that dealing directly with the users is better, but I see situations (no budget/time + hard to access users ) where this technique can be valuable.

Cennydd makes an important point in saying: “By talking to the people who talk to users, we introduce another layer of bias (political, cognitive, whatever) and thus could end up a bit further from facts than if we’d gathered them ourselves.”

I think the key to using this method, or others like it, is to always remember that their is that extra layer of bias, and attempt to take it in to account when working with the information. Designers should think about how the information gathered might be biased and draw design insights accordingly. It is when designers pretend that the information they gain from these once-removed research methods is ‘pure’ that they run in to problems and get blindsided.

Michael Beavers — Thu, 27 Mar 2008 15:30:31 GMT

...and what I neglected to mention above was that these small retailer users were within a 3-block radius of our office. We performed 5 interviews in just a few hours. That sample size was even enough to begin to rough in a persona.

Michael Beavers — Thu, 27 Mar 2008 22:57:09 GMT

Hi, Daniel—nice article. I especially liked your Québec city example as it is a practical reflection of unmet user needs in the architecture. I do think, however, that it is not that difficult to derive great customer insights “on the street”. Even if you walk out of your office or blast an email out to friends and family you can find at least one or two users.

We sometimes are forced by shortages of budget or time (often both) to perform “napkin ethnography”, which is admittedly not as reliable as a fully designed research protocol, but often enough to get a decent direction for what features users would find valuable.

For example, we were looking at a large-scale integration project for a soft drink brand—they wanted to build a site that would make it easy to replenish fountain syrup. We had TONS of features we could have placed on a very crowded homepage, but when we talked to small retailers we found that most ordering/reordering interactions happened over the phone and in-person with the driver (who had a hand-held device). The user was SO busy, they rarely sat down at a computer. That changed how we proposed to design the interface considerably: focusing on automating replenishment phone calls, resolving service issues, and being a lot more selective with new product merchandising. We placed all of the required advanced features in the architecture, but in secondary nav areas that didn’t draw as much visual focus.

All of this is to say that it is always a good idea to talk to users…and it is often easier and faster than we think. Call center employees are definitely great stakeholders to start with.

Sam Dobbin — Thu, 27 Mar 2008 15:47:19 GMT

Interesting article, though I agree with Cennydd that surrogate users can be problematic. As an example, my current project involves a lot of product development where most of the team has frequent access to users, and some of the team would even qualify as end users themselves. That said, the feedback we get from our internal surrogate users is often clouded by their knowledge of business priorities, and whatever ethnographic research we’ve managed to do externally has often revealed dramatically different things.

This is a good approach where no possibility for ethnographic research exists, but it would be interesting to look at a more guerilla approach to observation studies and other first-hand research.

Praveen Kumar Verma — Tue, 01 Apr 2008 13:45:22 GMT

@Jens: I think I should have been more explicit while writing. Anyway, when I said that my intent was to talk to the users about their tasks, needs, aspirations, desires, and motivations. I did not mean to ask them about their interaction with the site. We must keep in mind the famous quote, “Users are not Designers and Designers are not Users.”

Jens Jacobsen — Thu, 27 Mar 2008 13:19:53 GMT

Good article, and it has the perfect length!
@ Praveen: Yes, talking to the users is always a good idea. But: in my view this is also second hand information. Because you are asking people what they think they would do on your site. You never know what they would really do. So the only first hand information we can get our hands on are usability tests and log files.

Praveen Kumar Verma — Tue, 01 Apr 2008 13:42:43 GMT

Daniel, it is a nice read. Use of surrogate sources to gather information is a good idea, however, it takes the discussion back to the relevance of the data collected. What we are gathering here is second hand information and not the first hand had we talked to the users. But sometimes we designers are left with no choice but to use such methods.

Despite of the low cost and less time needed to do extreme user research, we shall always discourage our clients to go for it. Extreme user research should be practiced with caution and care. Nothing could replace the joy of talking to the users and designing for them.

Cennydd Bowles — Thu, 27 Mar 2008 18:11:43 GMT

One comment you make is “We want to make sure we have solid data based on facts, not perceptions.” Agreed. In fact, it’s something of a truism perhaps – but I do wonder whether this method is susceptible to this very problem. By talking to the people who talk to users, we introduce another layer of bias (political, cognitive, whatever) and thus could end up a bit further from facts than if we’d gathered them ourselves.

Don’t get me wrong, your approach can be useful in some circumstances, and I’ve used it myself in the past when I’ve had zero (and I do mean zero) budget to do more thorough research. But, given the choice, face-to-face testing / ethnographic research has to be preferable.

As a postscript, I’d say that anyone who argues that “if I like it, everybody will…” is neither a good designer, nor indeed a designer at all.