Comments on Metrics for Heuristics: Quantifying User Experience (Part 1 of 2)

Andrea Wiggins — Tue, 31 Jul 2007 13:25:15 GMT

Hi Tim -

It’s definitely true that a long visit may mean that people just couldn’t find what they are looking for. However, it’s been my experience (from looking at the long-term trend stats on dozens of sites) that in general, if visitors cannot find what they are looking for, they simply leave. For the most part, people just don’t hang around long if they don’t find what they want. Visit lengths will be short (particularly in number of pages) and there may be a lot of null search results or abandonment from a search results page.

To drill down into the question of whether visitors are really engaged, I would specifically look for navigation trends like pogosticking to identify whether users are getting lost in the site; it’s also possible that they’re viewing many pages because they’re going back and forth between a product listing and product details, for example. Like most web analytic measures, it all depends on the site; I would always look at a combination of several metrics to get an idea of whether or not the site is really engaging visitors, and would definitely recommend using visitor segmentation on the metrics as well.

Tim Yeo — Thu, 17 May 2007 05:13:23 GMT

Hi Andrea.

This statement seems rather contentious:
“The length of the average site visit, in both time and pages viewed, also provides verification of the brand experience….....Similarly, by considering content groups of similar pages, measuring the depth or breadth of visits to each content group can assess how engaging the users find the experience.”

It’s contentious because users who spend a long time on a site + going deep and across many content groups could simply be users who can’t find what they want. In this case, the site is not providing an engaging experience, but an annoying one.

Just using web analytics, do you think there are any other ways of looking at the data and being able to distinguish between users who are truly engaged and users who are simply lost?

What do you think?

Andrea Wiggins — Fri, 26 Jan 2007 14:53:48 GMT

Oliver—good points! I was trying to stay as general as possible and so used some of the specific example items that Rubinoff provided. I hope that anyone using web analytic evaluation for information architectures would choose the audit statements that best fit the specific site, because these are certainly very broad, and as you point out, may not fit with any one person’s conception of what measures a given site attribute.

In my view, brand, usability and content are parts of an interrelated continuum, but for practicality’s sake, we have to separate these characteristics of a site somehow if they are to be evaluated. Fortunately, everyone can choose or create the audit statements that best suit their specific purposes.

Oliver Reichenstein — Fri, 07 Jan 2011 19:39:58 GMT

Andrea,

Very interesting article. I fully agree with part 2 and I agree with most points of part 1. However, I have strong doubts on the way you define branding and separate content, usability and branding from each other:

“Branding:
The site provides visitors with an engaging and memorable experience.
Graphics, collaterals and multimedia add value to the experience.”

I think a website provides an engaging and memorable experience if it is
a) identifiable in its character purpose and value
b) pleasing
c) memorable
It is easily identifiable if it has a clear purpose that it communicates, if it is consistent in its appearance, and if follows the guidelines of corporate design if it belongs to a bigger corporation.
It is memorable it solved my problem quicker and better than expected.
Graphics and multimedia should follow the corporate guidelines and be highly relevant if used within contents. If they’re used just to please the user they create mistrust and annoyance.

“Usability
The site prevents errors and helps the user recover from them.
The site helps its visitors accomplish common goals and tasks.”

From my experience usability is a crucial element of the overall brand experience. If you go to a website that doesn’t work, you have a bad brand experience. First of all it needs to work. People hardly pay attention to design. They feel it, but they don’t really notice it. Design is important though, as it’s critical in the first few seconds, where the user decides to stay or go. Good design doesn’t mean the user stays. Again here usability is more important, but it helps building trust during the first judgment on the quality of the website.

Usability tests deliver good results within the test environments, they hardly assess one crucial moment of user experience: The first sight and the question: Should I stay or should I go. This moment is hard to test.

That’s maybe why usability consultants started developing interest for branding and design in general (see Norman). Coming from a big brand company it’s quite refreshing to see a usability expert get all excited about the matter:

http://video.google.com/videoplay?docid=56548785834474352…

“Content
Link density provides clarity and easy navigation.
Content is structured in a way that facilitates the achievement of user goals.
Content is appropriate to customer needs and business goals.”

Again here the separation and usability and content is quite fuzzy. Structuring content is not content, it’s IA, and as such rather a matter usability. Maybe I misread the separation, but I’d understand the three categories more like this:

Brand > Usability > Content – And yet that’s still not quite correct.

What’s my point? In as short as I can:
1. Contemporary branding = interface (of a company or product – depending on weather you look at corporate branding or product branding), as nowadays brand is defined as an overall experience one makes in connection with a product. Best, i.e. most memorable and marketable (inciting word of mouth) experiences are interactive ones.
2. The interface is first of all a matter of usability. Products need to work. In the end the Interfaces get skinned adapting corporate guidelines. But only bad corporate guidelines intrude on the product.
3. An interactive product doesn’t need too much skinning, its Interface shapes through the inert functionality of what it’s built for (see craigslist, delicious, ebay, reddit).
4. The Internet has changed and will continue to change our understanding of quality, brand and communication. And – at least form the consumer’s point of view – in a good way.
5. Brand experience IS measurable. With web stats and usability tests on one side and by crosschecking with the corporate design manual on the other side.
6. Brand is still widely misunderstood as a superficial discipline that starts with the logo, leads into an expensive brand manual that costs millions defining a couple of grids, colors and fonts and ends with an emotional ad on TV.
7. Brand on the Internet does not equal to special graphic effects. The most special effect a website can make is that it’s useful and easy to use. If in plus it looks good. That’s branding. Branding and usability are not contradictory. They used to be quarreling twins. But they have grown up and now they can’t be without each other anymore.

Well that was not so short… Anyway, I hope I’ve not been too intrusive with my little speech here. But I am excited about that matter for over 4 years now, and as I am currently writing a book on usability and branding, I think this articles and the comment section provide a wonderful opportunity to get more insights in a matter that is widely ignored.

Andrea Wiggins — Fri, 26 Jan 2007 14:53:31 GMT

Chris, you’re right on—the link is actually fine, but the period at the end of the sentence breaks it. I can’t edit the link to fix it, so I’ll retype it:

http://www.enlighten.com/pdfs/wp_analytics_08_06.pdf

Chris Collingridge — Fri, 26 Jan 2007 14:53:29 GMT

Andrea,

I think all you’d need to do to repair the link is remove the full stop (period) after .pdf

Chris.

Andrea Wiggins — Fri, 26 Jan 2007 14:53:24 GMT

Someone mentioned trouble with the direct link to the PDF; you can also just go to http://www.enlighten.com and there’s a link right from the home page.

Andrea Wiggins — Fri, 26 Jan 2007 14:53:24 GMT

I can’t agree more regarding the use of web analytic data to inform persona development. While I didn’t really get into that topic in this article, I did talk about personas and usability in a recent similarly-themed but differently-focused white paper, “Data Driven Design: Leveraging Analytics to Improve your Website Overhaul,” which is available at http://www.enlighten.com/pdfs/wp_analytics_08_06.pdf.

Michael Beavers — Fri, 26 Jan 2007 14:53:23 GMT

I agree with Fred—and by extension, Steve Mulder. Personas can be a highly accurate starting point for fully assessing tasks and content needs through on behavioral research. There really should be some effort to validate the data by comparing what research respondents are saying within the framework of issue categorization with Web analytics and analysis of behavior in a lab or other settings appropriate for contextual inquiry.

As an example, I’m in the middle of evaluating the impact of AJAX and social/community architectures on a client’s particular industry and marketing landscape. We have performed a preliminary round of research in three cities and have established enough data to suggest 2 solid personas, 1 emerging persona type, and suggestions or mention of about 4 others. Taking the first solid persona types, it was very easy to see them represented on YouTube and blogs as self-publishing persona types directly impacting my client’s brand. However, my client has been able to present little to no meaningful Web analytic data for their current site.

The good news is that we have some observable persona behavior for validation out there on the Web, but Web analytics data from my client’s site would have helped us paint a more complete picture. Without them, our heuristic analyses are based more on opinion than quantifiable activity—and I suspect would have taken us further in validating our persona types and helped to reduce our overall research timeframe.

Fred Beecher — Fri, 07 Jan 2011 19:39:40 GMT

Andrea,

At the ‘06 IA Summit, Steve Mulder gave a presentation about bringing more science to persona creation, in which he advocated (and this is a gross oversimplification here) that personas be developed and then validated/updated through surveys & research. Your insightful comments about using analytics data to validate a heuristic analysis got me to thinking that this same data could be used to either create or validate personas. I’m not sure if this is something you’ll be covering in Part 2, but it seems like another good use for this data.

- F.

Austin Govella — Fri, 26 Jan 2007 14:53:22 GMT

Masood,

How would you formulate some of the things you mention as heuristics we could apply for sites?

Andrea Wiggins — Fri, 26 Jan 2007 14:53:22 GMT

While I certainly agree that business goals are another key point of measurement and evaluation, I did not include them here because the focus of the user experience audit is the users’ goals. I think everyone can agree that business and user goals are strongly related; in my personal philosophy of design, if a site does not meet user needs, it doesn’t matter what the business goal behind it was because it cannot have succeeded at that goal. Regardless of how user-centric or business-centric your perspective, my experience has been that the same web analytic measures often define success from both perspectives.

You make an excellent point of other ways in which web analytics can be used to improve site management strategies, and I agree wholeheartedly that evaluation of multiple aspects of a site should be an up-front consideration. I look forward to more people adopting your perspective on the value of incorporating measurement in the entire iterative design process!

Masood Nasser — Sat, 22 May 2010 12:56:50 GMT

Nice article Andrea. Providing a context is invaluable while defining Heuristics in the first place. Though you have mentioned that Rubinoffs chart is customizable, there are Couple of things I would like to add. Business is left out in a gazillion of heuristics that I have seen to date. While some may argue that every one of the 4 given broad categories affect business, in my opinion it is not so.

The entire purpose of a customer facing site is to realize the business objectives, and that should reflect in the design. When I say business, it means not only the Dollars that you are expecting. It can also include future strategies, and time to market of those strategies. Is your IA designed in a way to facilitate new additions, what would be the time to launch new campaigns? This involves not only looking at the site, but also the entire design development process including the CMS. This is just to name a few.

If you look at the site from a Customer experience perspective, then each and every “touch point” should be accounted for and the experience should be checked against metrics. How you define those metrics is an interesting challenge, more difficult than the actual rating and evaluating process. Waiting for the next part of this article..