Boxes and Arrows: Comments by Tony Tulathimutte

Tony Tulathimutte — Tue, 27 Jan 2009 23:52:39 GMT

Hi Andres— Yeah, you’re absolutely right about the mayhem:

We had two moderators dividing up the six participants between each other; otherwise you run the risk of asking redundant questions.

We had the observers chat with us over IM whenever possible, or had them pass questions to us during the (very brief) lulls in testing; we also had a chat room in which the moderators and support staff could all coordinate with one another.

As for who to focus on, that was largely driven by observing what each player was doing. If they were performing a task the developers were interested in, we switched the focus to them; we also encouraged players to thinkaloud while playing, not necessarily anticipating a response. If there was a technical glitch or request, we just had the participants wave at the camera to catch our attention.

Tony Tulathimutte — Tue, 03 Nov 2009 18:17:21 GMT

Sounds great, and good to hear that there’s a distinction being made between automated and moderated remote tools! And right now, some good places to start looking are:

—Elizabeth Bacon’s public Google Docs spreadsheet of remote research tools: http://remoteusability.com/remote-testing-tools-round-up-…

—Our tools page on our website, RemoteUsability.com: http://remoteusability.com/tools/

—Chapter 8 of the forthcoming Remote Research book, by Nate Bolt and yours truly, covers not only moderated and automated tools and services, but also recording, notetaking, and recruiting tools for remote research: http://www.rosenfeldmedia.com/books/remote-research/

Hope that helps!

Tony Tulathimutte — Tue, 03 Nov 2009 18:31:24 GMT

Oh, and one other thing I’d add is that while remote methods do save on certain costs (mostly travel and recruiting), the actual difference in the total project cost isn’t really that huge. I think that the cost savings are a nice plus, but that the main benefits of remote methods are that you’re able to talk to users who are on their actual computers and in their usual computing environment—and if you’re live recruiting (that is, recruiting visitors to your website using a form), then you can contact users right as they’re about to perform a task you’re interested in observing. That means that users are intrinsically motivated to perform their tasks; you’re not just telling them to pretend that they want to do something.