Boxes and Arrows: Comments by Dana Chisnell

Dana Chisnell — Wed, 26 Nov 2008 19:42:18 GMT

Paul,

You make a really good point. I am not *assuming* an internal recruiter for the open-ended interviews, but I can imagine that’s the situation it is most likely to occur in. (I’m a consultant and my recruiting consultant does interviews rather than multiple-choice questionnaires.) I’ve seen teams that work with outside agencies be successful with the open-ended approach, but it does take a few times of working together before it can be really smooth. They had to train and coordinate.

In the end, it’s about tradeoffs. Do you have time to do the recruiting? If you don’t have a lot of time or resources to do your own recruiting, are you willing to take what you get from most agencies.

Dana Chisnell — Sat, 29 Nov 2008 16:22:48 GMT

Juan,

I like the idea of weighting the items in a questionnaire. That could definitely resolve some of the issues with multiple-choice answers.

The point I’m trying to emphasize is that we user researchers should be taking the time to do the interviewing (or find someone inside who can) for a few reasons. First, you get better data because it is more likely that the participant fits the profile. Second, you get more data because you learn things in the process of recruiting this way that you don’t learn in a typical usability test session—or if you do, it’s too late to use it. Finally, participants are much more likely to show up because you’ve established a relationship that they’re invested in. It is the beginning of that practice of treating participants like humans rather than data points.

I get that there are trade offs. Outsourcing the recruiting frees up the researcher in a dozen ways. Maybe there’s someone on your team who could help. Teams I know who recruit the way I’m suggesting just get phenomenal results that they know will be valid because they are confident that they have the right user in the room.

Dana

Dana Chisnell — Wed, 04 Feb 2009 00:55:38 GMT

First, thanks for all the great comments! I’m delighted that people care about this topic as much as I do. (Unfortunately, the reviewers for UPA didn’t see value in having a session about recruiting at the conference.)

To everyone who has issues with my suggestion that you move away from using a screener:

I get that it is uncomfortable to move away from using a questionnaire because a) you’re used to thinking of recruiting that way; b) you feel you can’t trust your recruiter to work without one; and c) it seems like more work for you to do recruiting if you don’t codify the recruit with multiple-choice answers.

My first suggestion is that you work on training your recruiter. Teams I know who work with the same recruiter over time (either an in-house person or an agency) find that they can help that person or group learn about the product, the users, and the methods—and that doing so helps the recruiter make better decisions for you. I’ve written about this a couple of times recently on my blog. Check these two posts:

http://usabilitytestinghowto.blogspot.com/2009/01/why-you…
http://usabilitytestinghowto.blogspot.com/2009/01/yes-or-…

Would love to continue the conversation…

Dana

Dana Chisnell — Thu, 29 Jul 2010 16:40:03 GMT

Hi,

Contextual Inquiry is a method developed by Karen Holzblatt and Hugh Beyer. Unfortunately, I see no relationship to their great stuff; no evidence that the authors have looked there for guidance. I wonder just how Karen – because she teaches workshops on the method – feels about the idea of remote contextual inquiry. I always thought the point of contextual inquiry was the *context* part. How do you get that if you’re remote?

What you’re describing here is remote usability testing, which a lot of people are doing, and that Nate Bolt and Tony Tulathimutte have written an excellent book on.

Good luck,
Dana
dana@usabilityworks.net