Boxes And Arrows : The Design Behind the Design

Article Idea:

Add Another Measure to the User Experience Toolkit

suggested by Alison J. Head on 2006/08/21

In our daily work, we all apply widely known measures for assessing user experience, whether we specialize in information architecture, usability, or interaction design.

Some of us apply quantitative methods, such as deconstructing transactional logs. Some of us rely on task completion rates, error rates, and the time, or clicks, it takes someone to complete a task. While others of us, count on card sorting, prototyping, or persona building to generate qualitative data that helps inform the process.

Still, most of us in the trenches apply assorted measures; counting on the data we collect to give us insights into how users interact with a site, and in the end, to give us fodder for building ore user-centered designs.

This past summer, we tried something a little different. We collected our typical usability measures for a client, but we began formally collecting a new measure, too. It’s something we’ve seen happening a million times during other tests sessions. We used to think it was just too obvious to formalize and report. We don’t think that anymore.

We call this measure a Site Abandonment Measure (SAM). We define a SAM as the percentage (or number) of participants who give up on a specific task (or set of tasks), leave a site all together, and turn to another source—any source—to get a task done.

Put simply, it’s the “I quit, I’ve had it with your site” rate. And it means that participants give up and go elsewhere to meet their goals—usually another site (we’ve heard “I’d just Google it” a lot), or email, or their good old cell phone to complete the task we have asked them to carry out.

Here are some SAMs from our recent study about a foundation’s extranet that lets donors manage their philanthropic funds:

- When we asked 15 participants in our sample to use the site to fund a certain Special Olympics program for the summer, 40 percent of them abandoned the task all together.

- For all of the tasks we asked participants to conduct during our session (i.e., 10 key tasks), the SAM was 38.6 percent. This means that, on the average, six out of every 15 participants, per task, gave up on the site we were testing—but did not give up on completing the task using other means.

In my shop, we have grown fond of this new measure. It’s relatively straightforward. We believe SAMs provide valuable data beyond what task completion rates and other measures can offer. We have found that SAMs translate well to clients, too. Let’s just say, the abandoned site data we collected resonated down to the tip of our client’s toes.

For practitioners, no matter whether you define yourself by IA, UCD, or UX, a SAM measure may add value to your own evaluation methods, as well as getting the kind buy-in from stakeholders your work requires, too.

###

This proposed article will cover (in greater depth): (1) user experience metrics, techniques, and measures, (2) definition of Site Abandonment Measure (SAM) as a different measure of user experience to combine with other data, and (3) communicating and calculating the return on investment of user experience research. In all likelihood, the final piece will be about 1200 – 1700 words. – AJH