Comments on Researching Video Games the UX Way

Nate Bolt — Tue, 17 Mar 2009 18:15:20 GMT

Trevor and Anuj – awesome questions. I *wish* we were allowed to talk about the findings of the study and how it related to game design decisions, but while EA has been absolutely amazing in letting us discuss the research, they have made it very clear we are not allowed to discuss any of the findings publicly. So I guess we can’t officially comment on what the game getting knocked says about the testing, other than that I would love to be able to talk about it someday.

For the remote viewing software, we went through all the same frustrations you listed Anuj, and our final solution was just to run 300ft of VGA cable around the testing area and mirror the gaming laptop output on six seperate LCD monitors (DVI+VGA) that matched each of the participant’s screen’s. Then for recording we used ZD Soft game recorder running on each gaming machine. Pretty duct-tape-ish solution, i know, but it worked for our setup. In the future, I want to look at software+hardware that sends VGA signals over ethernet, since as you’ve noticed the 3d acceleration on games makes the regular software totally FAIL.

Anuj Nanavati — Thu, 05 Mar 2009 23:38:23 GMT

Hi Nate and Tony, this study sounds really fun! I had a quick question about the technical setup. Did you guys use any special remote viewing app for recording the screen running a 3D app or was it through DV cams? We had issues in past with 3D game testing as we could not view the 3D content remotely with a remote viewing client, we tried Morae recorder, Real VNC, Windows Remote Desktop, DameWare Mini Remote Control… but nothing work. As soon as the game starts the remote viewer would go black and it would come back while on regular desktop. i would really appreciate any suggestions here..- Thanks

Trevor Stafford — Thu, 05 Mar 2009 19:44:10 GMT

Uhhh, too bad the game was almost universallly panned…and not just because of the Orwellian DRM, but because it was ridiculously oversimplified.

http://www.amazon.com/Spore-Pc/dp/B000FKBCX4

What does this say about the usability testing?

Demetrius Madrigal — Mon, 02 Mar 2009 23:02:38 GMT

Good article. I’d been wondering for quite some time how research was performed on games considering the fun factor. It sets a great precedent for future research in games, especially how successful Spore has been. It was also great to hear that EA was willing to commit to such a costly research commitment.

Tony Tulathimutte — Tue, 27 Jan 2009 23:52:39 GMT

Hi Andres— Yeah, you’re absolutely right about the mayhem:

We had two moderators dividing up the six participants between each other; otherwise you run the risk of asking redundant questions.

We had the observers chat with us over IM whenever possible, or had them pass questions to us during the (very brief) lulls in testing; we also had a chat room in which the moderators and support staff could all coordinate with one another.

As for who to focus on, that was largely driven by observing what each player was doing. If they were performing a task the developers were interested in, we switched the focus to them; we also encouraged players to thinkaloud while playing, not necessarily anticipating a response. If there was a technical glitch or request, we just had the participants wave at the camera to catch our attention.

Dan Soltzberg — Mon, 26 Jan 2009 18:54:43 GMT

Nate and Tony, your writeup really illustrates the importance of being flexible and responsive with research methods. Thanks for sharing the story.

Andres Zapata — Mon, 26 Jan 2009 14:53:56 GMT

Nate and Tony – AWESOME STUFF! I am trying to understand your technique a little bit better. Did you have two moderators, 6 subjects and 12+ clients all going at the same time? That sounds like mayhem! How did you control the situation – at least from the observation room? For example, I’ve been in many usability tests where it was just me moderating, one participant, and one colleague hanging out with three clients at the same time – and that was tough to control .. granted we have chatty clients, but that’s good… that means they are paying attention and they are engaged with what is being discovered. How did you guys do it? How did you know which subject to focus on at any given point? Write a part II on this! Thanks for sharing.

Chris Blow — Tue, 27 Jan 2009 23:44:37 GMT

love that guy in the first reel: “THE DESIGNING NEVER STOPS, SON.”

ji kim — Thu, 22 Jan 2009 20:47:20 GMT

Good stuff Nate!! Spore did a really nice job with their Spore Creator user experience – big fan!
Methods you guys used sound similar to Listening Lab – http://www.goodexperience.com/2004/12/tips-on-moderating-…

Angie Chang — Thu, 22 Jan 2009 19:37:20 GMT

I love the videos! The gamers are hella funny, LULZ. Makes me think about my own game face :P

Nate Bolt — Mon, 26 Jan 2009 14:54:57 GMT

Thanks Austin glad you liked it. We did have to get permission from both the individual gamers and the Spore team at EA. But the Spore folks were unbelievably cool about giving us permission to talk about the details, with two big caveats:

(1) It all had to be post-launch. While the game was still in development, when all the research took place, everything was super-duper top secret.
(2) We can not ever discuss the actual findings from the study, only the techniques.

The most difficult part was planning this before we even started testing by drafting really rigorous consent forms for participants and sending them info on how we would use their recordings. Some people opted-out, which of course was totally cool. Most were like “sure that’s hella funny lulz”

Austin Govella — Thu, 22 Jan 2009 20:47:24 GMT

I really enjoyed how you pull the Kimono back, even going so far as to post the sample video. The second video felt a little more like marketing, but it shows a lot of the rationale, and I love the shots of the observation room.

Did you have to get permission from EA to post these? Was it difficult?

I’d love to see more write ups and sample videos from other research.