Comments on Remote Online Usability Testing: Why, How, and When to Use It

marv throneberry — Fri, 07 Jan 2011 19:39:45 GMT

There is a new guide to remote usability testing on ok-cancel that has some updates to this article since a lot of new tools have come out.

graham ericksen — Fri, 26 Jan 2007 14:51:25 GMT

I’ve worked with the usability sciences product called WebIQ. It has great analysis tools that let you cut and slice user responses more finely than a traditional spreadsheet of behaviors and results. Although it’s not true usability testing, it is highly useful for determining user intent.

subimage — Fri, 26 Jan 2007 14:51:25 GMT

Come on guys, this is a good article – for a webEx introduction. Where’s the in depth information relating specifically to usability?

Perhaps I’m biased since I’ve done this type of testing in the past. I was really looking for more information relating to how you found testing remotely differs from a one-on-one session.

I’ve found these sessions to produce excellent “why” feedback depending on what types of people I’ve had attend.

When you get too many people attending such meetings it tends to get a bit unmanagable. This seems to be the case especially when the extra people don’t actually use the software you’re testing.

While this is great for upper management to see what’s going on, usually it doesn’t provide the best type of feedback. If you are using this approach, I’d strongly advise limiting your group as much as possible to your target audience – people who will actually be using your application or site.

One specific instance I can remember I had a product manager sitting on my side of the conference call. The users / management mob on the other side of the phone constantly insisted on questioning feature delivery dates and asking if they could add things to the project, even when instructed that the meeting was supposed to be a usability test.

Anyone else with a similar experience or anything to add regarding this type of information? How have you gone about determining who to select for your study?

How have you structured your questions so that you don’t get sidetracked on something that’s not very important?

Ari Paparo — Tue, 11 Jan 2011 03:39:52 GMT

Hey, this article inspired me to write my own about my experience using a different product, Vividence. It can be found here: http://www.aripaparo.com/archive/000367.html.

ji kim — Tue, 11 Jan 2011 03:39:52 GMT

Here is a must have gadget for remote usability testing: Mini Recorder Control

http://www.radioshack.com/product.asp?catalog%5Fname=CTLG…

One of the best $14.99 investment I ever made.

Mabel Ney — Tue, 11 Jan 2011 03:39:52 GMT

The VIA Group in Portland, Maine, also recently completed a usbility evaluation with a similar setup. Here’s what we saw…

> Benefits

The evaluator is in a more familiar and comfortable setting using his/her own system. A formal facility can be intimidating and clinical.

The participant doesn’t have to account for additional travel time or contend with rush hour traffic.

We had fewer drop outs and were able to reschedule when an evaluator needed to attend an important meeting.

> Limitations

Eliminating dial-up participants may impact the audience segmentation and skew the findings.

WebEx impacts the response time. We had to forewarn participants that the scroll bar and click response would be slow.

All viewers must have the same screen resolution, otherwise the window may be clipped and the vertical scroll bar may be hidden. We asked our participants to write down their screen resolution prior to the meeting. Even at the same resolution a few participants were unable to view the vertical scroll bar, and we learned to pull in the right side of the window slightly.

Other programs running in the background can slow the response time. In one case, the mouse pointer was replaced with a resize icon.

A bad phone connection, mumbled or quiet speech or an unresponsive participant can also be very aggravating and inhibit your ability to catch verbal cues.

Be certain your participant’s employer allows them to load software on their work computers. Many large companies have a policy against unapproved installs.

We had several isues and needed to troubleshoot on the fly.

> Recruiting

We supplied a recruitment screener to our recruiters. This helped identify appropriate participants and system requirements, but more importantly, it helped identify people with strong verbal skills.

> Meeting setup

We sent simple instructions discussing the WebEx connection—URL, password, and a link to learn more—the phone conference information, a place to write down the screen resolution, and a document of tasks to be printed prior to the meeting.

We used Cam Studio (http://www.atomixbuttons.com/vsc/) for testing but due to the 1 hour length, the voice and video did not sync. We switched to Camtasia ($350, http://www.techsmith.com/download/studiodefault.asp) which also has editing software.

The recruiter called the participant on the morning of the test as a reminder and went over the steps with them.

> Cost
Webex charges 0.45 per minute per connection or $27.00 per hour per connection

John Bierschwale — Fri, 26 Jan 2007 14:51:24 GMT

The article discusses challenges when using the conference bridge number for both the test conversation and the observers. You can reduce these particular challenges if you have two telephone lines handy. We use two lines/phones in combination with an audio mixer we have in our lab. It’s a bit more complicated but this way there is no chance for the observers to intervene/interfere with the test session.

I’ll try to describe. Line 1 is a direct link to the participant (the participant calls the phone number or you can call the participant, whichever works best for your situation). Line 2 is for the conference bridge. We use the phone tap to send the session audio from Line 1 into the mixer. One output from the mixer goes into our recorder which also has scan-converted video of the session. The other output can go to Line 2 which is the conference bridge (there are phone taps that work either way – record or play into phone – the ones we use are from hellodirect.com).

We often set the conference bridge to “listen only” so we have a captive audience and they don’t have to bother with muting their phones.

Just another option to consider.