Boxes and Arrows: Comments by Paul Nuschke

Paul Nuschke — Wed, 01 Aug 2007 12:45:37 GMT

I realized that my timeline might seem modest without the knowledge that I work for a consulting company, so a lot of time is typically spent learning an application, waiting for deliverables, and writing reports. This article would be applicable to anyone wanting to pull off usability testing without much time to prepare or write reports.

Paul Nuschke — Fri, 19 Sep 2008 18:17:02 GMT

Good point, Anthony. One nice thing about consulting work is that clients have typically already gotten the buy-in for our services before they contact us. But for companies doing internal testing, getting buy-in should be a key part of the process.

Paul Nuschke — Tue, 23 Sep 2008 02:29:40 GMT

I’m glad that you shared this Melissa. I had a limited amount of space to describe QTUT, so I chose to leave out testing online and recruiting through friends. When you are working with a consumer-type application, it can be pretty easy to find qualified applicants through social networks. They also have a bit more motivation to pay attention than normal, since they have a connection to you. Internet testing gains you is a bit more flexibility, although I haven’t found that it makes much of difference in terms of recruiting faster. But our (Electronic Ink) testing facility is pretty conveniently located in Philly, so that may be part of it. What do you think?

Paul Nuschke — Tue, 25 Nov 2008 14:09:58 GMT

Nice article, Dana. I’d add an assumption to your recommendations on the screener: that you are using an internal recruiter. To get rid of multiple choice questions, you have to have a recruiter that is very knowledgeable about the participant behavior and demographics that you need. The recruiter also must not “lead” the participants into the correct answer, which is difficult to prevent in people who are not trained in interview techniques. External recruiters do not necessarily have good interviewing skills, they will typically not have little knowledge of your project, and they have a conflict of interest (the less time it takes to recruit someone, the more money they make). So multiple-choice questions, while not perfect, are very difficult to get rid of. However, this article does present a compelling case for utilizing internal recruiting whenever possible.

Paul Nuschke — Sun, 11 Jan 2009 03:27:11 GMT

Hi Andrew. This is off topic, but usability testing does not have to be expensive. While the QTUT method shows you how to make it shorter (which can reduce cost), you can reduce cost even further by not using a lab at all. Any conference room or quiet area will do (as long as your stakeholders are willing to watch video instead of a live session). Regarding reports, we have client confidentiality agreements which prohibit us from sharing reports, but there are a lot out there on the web.

Paul Nuschke — Thu, 15 Jan 2009 18:42:36 GMT

Good question, Andres. The answer depends on the specific question that I’m changing. If it is just a tweak to clarify the task, then I usually make the change after the first participant, so explaining the results is not a big deal. If the task changes significantly later in the study (or is replaced by another task), I treat it like a separate task for analysis. Often, though, many of the issues you find overlap, which actually strengthens your results.

The main problem with switching tasks halfway through is the low N on some issues. It requires some intuition to determine which issues are likely to project to a larger audience, which are not, and which require more evidence. My decision is usually based on the severity of the issue (persistence, impact, and frequency) and the type of participant. If it’s a minor problem, then I might mention it with other low priority issues. If it is a big problem, like a mislabeled button that prevents 25-50% of people from ever finding a webpage, then I might include it with high-priority fixes. The participant type matters if the participant is unreliable or only one type of participant find the issue.

In quick turnaround studies, you have an additional constraint: the release timeline. If the issues can be quickly fixed then you do it. If it is going to take a long time, you delay it.