Boxes and Arrows: Comments by Andrea Wiggins

Andrea Wiggins — Fri, 26 Jan 2007 14:52:23 GMT

I recently wrote a research paper for a master’s course in information architecture entitled, “Metrics to Validate Heuristics: Proving the Information Architect’s Case;” it discusses web analytic measures to validate Rubinoff’s user experience audit statements. The instructor suggested that I should seek submission for it here, and it would seem that there is a demand so I’ll have to make some revisions and give it a shot.

Andrea Wiggins — Fri, 26 Jan 2007 14:53:22 GMT

While I certainly agree that business goals are another key point of measurement and evaluation, I did not include them here because the focus of the user experience audit is the users’ goals. I think everyone can agree that business and user goals are strongly related; in my personal philosophy of design, if a site does not meet user needs, it doesn’t matter what the business goal behind it was because it cannot have succeeded at that goal. Regardless of how user-centric or business-centric your perspective, my experience has been that the same web analytic measures often define success from both perspectives.

You make an excellent point of other ways in which web analytics can be used to improve site management strategies, and I agree wholeheartedly that evaluation of multiple aspects of a site should be an up-front consideration. I look forward to more people adopting your perspective on the value of incorporating measurement in the entire iterative design process!

Andrea Wiggins — Fri, 26 Jan 2007 14:53:24 GMT

I can’t agree more regarding the use of web analytic data to inform persona development. While I didn’t really get into that topic in this article, I did talk about personas and usability in a recent similarly-themed but differently-focused white paper, “Data Driven Design: Leveraging Analytics to Improve your Website Overhaul,” which is available at http://www.enlighten.com/pdfs/wp_analytics_08_06.pdf.

Andrea Wiggins — Fri, 26 Jan 2007 14:53:24 GMT

Someone mentioned trouble with the direct link to the PDF; you can also just go to http://www.enlighten.com and there’s a link right from the home page.

Andrea Wiggins — Fri, 26 Jan 2007 14:53:31 GMT

Chris, you’re right on—the link is actually fine, but the period at the end of the sentence breaks it. I can’t edit the link to fix it, so I’ll retype it:

http://www.enlighten.com/pdfs/wp_analytics_08_06.pdf

Andrea Wiggins — Fri, 26 Jan 2007 14:53:48 GMT

Oliver—good points! I was trying to stay as general as possible and so used some of the specific example items that Rubinoff provided. I hope that anyone using web analytic evaluation for information architectures would choose the audit statements that best fit the specific site, because these are certainly very broad, and as you point out, may not fit with any one person’s conception of what measures a given site attribute.

In my view, brand, usability and content are parts of an interrelated continuum, but for practicality’s sake, we have to separate these characteristics of a site somehow if they are to be evaluated. Fortunately, everyone can choose or create the audit statements that best suit their specific purposes.

Andrea Wiggins — Wed, 07 Mar 2007 14:34:39 GMT

Jonathan, you’re definitely right about the time-sink potential with web analytics. Not all statistics are useful, decision-making statistics. My favorite heuristic for deciding whether a statistic should be reported is whether it measures something that can be changed or affected by a decision of the people who would receive the report. If they can’t do anything about it, then there’s not necessarily much value in reporting on it; fortunately, we are often able to influence outcomes when we have a goal state in mind.

Instead, my approach is to evaluate the site goals (as established by the site stakeholders) and work out the measures that can be used to reflect those site goals based on the available data set. Sometimes you can get exquisitely precise figures that really tell a tale, and sometimes you can only get the gist of it; either way, the only tenable approach is to know what you’re looking for and how you will find it before you dive in. That’s just good old-fashioned research design. You can’t build a site architecture if you don’t know what the site is intended to do and for whom, and you can’t measure that site’s success if you don’t know how that site works and what constitutes success. The other big problem with mucking about blindly in data is that you may find something of “significance” that really has no face validity.

Andrea Wiggins — Thu, 29 Mar 2007 12:29:31 GMT

This does indeed sound interesting; the reason my articles do not have more cautionary detail along these lines was a matter of editorial necessity. I’m all for a good cautionary tale about using the wrong data to answer a reserach question; however, not all log data is useless nor subject to quite as much vagary as the prototypical server logfile. I cannot gauge your familiarity with the topic of web analytic data integrity, but do be cautious on that point if you take up the topic. You might also consider an analysis that shows both sides of the coin.

I believe that user behavior is the entire point of bothering to even consider the statistics, and I believe that any hard user data is better than none at all. I think there are a lot of people out there who agree with me and have practical experience that would indicate strong empirical support for the notion, but a well-researched dissenting opinion is worth hearing.

Andrea Wiggins — Sat, 07 Apr 2007 20:43:04 GMT

I would love to see this! I’ve worked for several nonprofits and been their accidental techie and webmaster; it was a major influence in my decision to pursue my MSI in Human-Computer Interaction. Not-for-profit organizations have unique needs, goals, resources and constraints. An article on this topic is much-needed by the nonprofit community as well; I would suggest cross-polination with nonprofit-specific tech resources like TechSoup. If I can help to contribute to such an article, please let me know!

(and btw, it was really great to meet you at the IA Summit, Olga!)

Andrea Wiggins — Tue, 31 Jul 2007 13:25:15 GMT

Hi Tim -

It’s definitely true that a long visit may mean that people just couldn’t find what they are looking for. However, it’s been my experience (from looking at the long-term trend stats on dozens of sites) that in general, if visitors cannot find what they are looking for, they simply leave. For the most part, people just don’t hang around long if they don’t find what they want. Visit lengths will be short (particularly in number of pages) and there may be a lot of null search results or abandonment from a search results page.

To drill down into the question of whether visitors are really engaged, I would specifically look for navigation trends like pogosticking to identify whether users are getting lost in the site; it’s also possible that they’re viewing many pages because they’re going back and forth between a product listing and product details, for example. Like most web analytic measures, it all depends on the site; I would always look at a combination of several metrics to get an idea of whether or not the site is really engaging visitors, and would definitely recommend using visitor segmentation on the metrics as well.

Andrea Wiggins — Tue, 31 Jul 2007 13:34:45 GMT

Masood,
Austin has it nailed. My agency work experience has involved the discovery of site goals as a primary part of the site development process. Even if the organization already has goals for their site, they often require adjustment for a new site structure and features. For example, if having 3 or more pageviews in an average visit was a goal metric, a change to a different site technology could interfere by invisibly adding another pageview to every visit, and the original goal metric would need to be adjusted accordingly. In my experience, setting site goals is often a “committee” activity, so I definitely agree that the UX practitioner shouldn’t be defining objectives independently.

Andrea Wiggins — Thu, 27 Sep 2007 01:36:37 GMT

Thanks for the feedback, folks – I can only hope the “real thing” lives up to the hype! My perspective on this practice is to give a jumping-off point for developing personas using available user data, as opposed to solely from assumption or imagination. For more and more people, Google Analytics is one of few sources of user data, and while it can only get you started, it can lend structure and credibility, much like Elizabeth’s “Evidence-Based IA”. I’d love to see Elizabeth write an article or two about this topic – it sounds great!

Andrea Wiggins — Sun, 02 Mar 2008 01:11:00 GMT

Speaking as a professional web traffic analyst (no longer practicing due to other ambitions), the use of server logs is definitely “deprecated” at this point – no web analytics professional today will recommend using server log data for much more than system function monitoring, unless it’s an extremely unusual situation. Client-side data collection significantly improves on ALL of these issues; appropriate vendor solutions make it relatively simple to sessionize even AOL visitors, identify returning visitors with reasonable accuracy over very long periods of time, and filter out bots, regardless of whether they can identify themselves as IE. And a whole lot more.

I appreciate the thorough analysis of why server logs stink. They do. That’s what the web analytics community has been saying for years, and why they are trying so hard to get better data collection methods and analysis standards adopted. Appropriately collected data and careful analysis (by professionals who know the weaknesses of the methods and tools) minimizes threats to validity and provides reliable insights. Not all usability insights require advance knowledge of user goals; many branches of social science have spent centuries inferring goals from behaviors, so this is not a sufficient reason to discount the potential usefulness of this particular behavioral data for usability.

I think that this is a sadly one-sided perspective which is ignorantly dismissive of client-side analytics, and implore every reader to investigate the vast improvements to web analytic technology that have come about in the last two years. Things have really changed a lot with respect to data quality; the attitude of this article is very frustrating because it may prevent people from exploring the full range of options with respect to user data. Usability tests are almost entirely contrived, and web analytic data is almost entirely naturalistic; I can see no reason not to use both in method triangulation, as each data source balances the weakness of the other.

Andrea Wiggins — Wed, 31 Oct 2007 00:43:05 GMT

This article is very well-informed; Avinash Kaushik is one of web analytics’ great “gurus” and almost undoubtedly the best source in the industry for this topic. I appreciate the discussion of the importance of properly representing insights to stakeholders – your SAM packs a one-two punch that would beat any SAR hands-down, and I’d love to see more people using more meaningful measures like these! There’s so much more value to be gained when you can use multiple sources of user data to paint a picture for the stakeholders.

Andrea Wiggins — Fri, 16 Nov 2007 16:19:39 GMT

Thanks for the comments, folks!

Jonathan, the technique doesn’t make assumptions about the rightness or wrongness of the site’s current instantiation. In fact, I would generally assume it’s not optimal! This technique is less about the behavioral end than most of us would like, and that’s largely because of the tool’s robustness. I would not recommend using Google Analytics to dig into user goals; you can really do some wonders toward that end with the more fully-featured vendor solutions, but for this article I was specifically constraining the technique to a widely accessible tool that pretty much everyone can use. I wouldn’t speculate on why people were viewing the “About Us” page because there are lots of possible reasons for that one – but with something like “Classes” or “Auditions,” you can make more reasonable assumptions about user interests, if not goals. The page popularity statistics are among the least solid to interpret for this purpose, which is why I emphasize taking advantage of more directly interpretable information like geographic locations, browser/OS, and new/returning visitor status. This is hard data about real users and I think there’s a lot of value to be had from taking advantage of every data source we can get.

Alec, I love the approach you describe! I have made use of web analytics to extract user behavior “in the wild” for similar uses, and I think it’s a very powerful approach. The main difference is the source of the user behavior data, and every data source has its advantages and disadvantages. Again, for digging into the behavioral aspects, I don’t think Google Analytics is a strong enough tool. To use the more sophisticated web analytics applications for this purpose, it’s important to work with a web analytics professional for validity reasons – they know the strengths and weaknesses of the data collection methods and analysis tools, which is important for interpreting the data.

And you’re right – this was specifically written for all the folks who don’t have access to the preferred methods of collecting user data. Even when you have all of those user research data streams available, I think there’s value in bringing in the web analytic data. User interviews and surveys provide great information but there are notorious problems with self-reporting, so it may be useful to verify self-reported behavior against actual usage. The research world is quite fond of triangulating data, but I haven’t seen much emphasis on this type of validation for UX design.

Alec, I think you’ve hit the nail on the head by saying that personas are communication tools. What the persona communicates will depend in part on what you put into it; my opinion is that a greater variety of sources for user data will help you generate a stronger persona. This technique is simply a way to use what you have, even when you haven’t got much to use, which is often the case – as you point out, it can be really hard to get support for qualitative data collection.