Comments on Metrics for Heuristics: Quantifying User Experience (Part 2 of 2)

Andrea Wiggins — Tue, 31 Jul 2007 13:34:45 GMT

Masood,
Austin has it nailed. My agency work experience has involved the discovery of site goals as a primary part of the site development process. Even if the organization already has goals for their site, they often require adjustment for a new site structure and features. For example, if having 3 or more pageviews in an average visit was a goal metric, a change to a different site technology could interfere by invisibly adding another pageview to every visit, and the original goal metric would need to be adjusted accordingly. In my experience, setting site goals is often a “committee” activity, so I definitely agree that the UX practitioner shouldn’t be defining objectives independently.

Austin Govella — Wed, 07 Mar 2007 14:37:01 GMT

Masood,

Even when they’re not stated, we have assumptions about the goals.

If you don’t have clear goals and metrics, you should definitely discover them for yourself and state them. The business owner can verify whether you’re right or not. And, once that discussion starts, not only do you have clear goals, but you’ve changed the overall project’s conversation.

Masood Nasser — Fri, 26 Jan 2007 14:53:49 GMT

Andrea , think the approach of evlauting site goals and then creating the measures is perfect. The only problem here is that many of the site goals or wishlists are too esoteric to work out measures. In many a case, even the objectives are not clear. So what does the UX practitioner do? Define the objectives independently?. This is something that should not be done, as the true context behind an intitiave can be accurate only when it comes frm , or is endorsed by business owners.

Andrea Wiggins — Wed, 07 Mar 2007 14:34:39 GMT

Jonathan, you’re definitely right about the time-sink potential with web analytics. Not all statistics are useful, decision-making statistics. My favorite heuristic for deciding whether a statistic should be reported is whether it measures something that can be changed or affected by a decision of the people who would receive the report. If they can’t do anything about it, then there’s not necessarily much value in reporting on it; fortunately, we are often able to influence outcomes when we have a goal state in mind.

Instead, my approach is to evaluate the site goals (as established by the site stakeholders) and work out the measures that can be used to reflect those site goals based on the available data set. Sometimes you can get exquisitely precise figures that really tell a tale, and sometimes you can only get the gist of it; either way, the only tenable approach is to know what you’re looking for and how you will find it before you dive in. That’s just good old-fashioned research design. You can’t build a site architecture if you don’t know what the site is intended to do and for whom, and you can’t measure that site’s success if you don’t know how that site works and what constitutes success. The other big problem with mucking about blindly in data is that you may find something of “significance” that really has no face validity.

Jonathan Baker-Bates — Fri, 26 Jan 2007 14:53:40 GMT

Ah – just seen “Edit (for another 15minutes)” – but you can’t delete. Oh well. Sorry about this everyone.

Jonathan Baker-Bates — Fri, 26 Jan 2007 14:53:40 GMT

Hmm – HREFs seem to be stripped out of posts here. Insert this link and the words “SCONE with TEA” between “about” and “CHI” in the above post:

http://vsis-www.informatik.uni-hamburg.de/getDoc.php/publ…

Didn’t B&A used to have post previews? Perhaps that’s an MSIE-only thing.

Jonathan Baker-Bates — Wed, 07 Mar 2007 14:34:04 GMT

Nice summary of the field. I would point out though that in my own experience, anything to do with web analytics is a huge time sink – you need to be 100% clear about about what it is you are trying to find out, and exactly how you are going to do it. Far too often, I find people looking at almost useless aggregate traffic reports in the hope that they will reveal some insight. I think you’ve shown that it takes a lot more than that to be in supporting design directions.

By the way, I recall the presentation about CHI. Has anyone subsequently gone on to use that in any way?

rajat julka — Fri, 26 Jan 2007 14:53:40 GMT

Good Article!

Web Anlaytics can be very usefull set of information to test wether the Information Architect has done the job correctly or not.

Web Analytics can help us calculate the difference between “how designer completes the IA work ‘THINKING from user perspective ’ ” and “how the target audience behaves/interacts with the application ‘IN ORIGINAL’ ”.

Geoff Simon — Wed, 07 Mar 2007 14:33:41 GMT

Andrea, I enjoyed reading the first and second part of your article. I believe that the combination of usability testing and analytics can work together.

Of course, if you are on a budget you can get away with just using a sophisticated analytics service, if it includes click path and other valuable metrics along with someone who knows how to interperet the data.

Celia Fiennes — Fri, 26 Jan 2007 14:53:30 GMT

Thoughtful and clear. Thank you Andrea, I appreciate your rigor.