Boxes And Arrows : The Design Behind the Design

Would this be a riposte to the recent articles by Andrea Wiggins on “Metrics for Heuristics: Quantifying User Experience”? If so, I’d like to read it.

Sounds very interesting, the aforementioned article already generated a lively discussion in-house.

Jonathan – this would indeed be a response to the article you mentioned. More than that, it is a response to the entire sentiment that any data of value (for usability’s sake) can be gotten from log files. Typically, those who espouse such a practice misunderstand what is found in log files and how it gets there. Things like browser caching, network & proxy caching, and user behavior totally invalidate the statistics.

I would love to read this, but saying that log files are rendered worthless by usability study raises my eyebrows. It is an interesting hypothesis, but I don’t agree with it in the least.

Before you dive into this, I highly recommend reading “The User is Always Right” by Steve Mulder. While his book focuses primarily on persona development, he shows how balancing qualitative data and quantitative data (the universe from which log file reports come from) is a more solid way to develop personas. It is a fast, enjoyable read.

While there are challenges with setting up, processing, and cleaning log file data, a little trend information can really help validate other areas of research. This is not an either/or proposition between qualitative and quantitative.

I wholeheartedly believe in the wisdom gained from qualitative user research. However, I’ve yet to find a marketing department or corporate interactive team that isn’t more well-versed in quant than qual. The most feasible way to sell the value of qualitative research to these teams is to be open to supporting and fleshing-out your findings with quantitative research.

If you don’t, I promise you’ll be spitting in the wind with the vast majority of site owners—and probably with this article.

This does indeed sound interesting; the reason my articles do not have more cautionary detail along these lines was a matter of editorial necessity. I’m all for a good cautionary tale about using the wrong data to answer a reserach question; however, not all log data is useless nor subject to quite as much vagary as the prototypical server logfile. I cannot gauge your familiarity with the topic of web analytic data integrity, but do be cautious on that point if you take up the topic. You might also consider an analysis that shows both sides of the coin.

I believe that user behavior is the entire point of bothering to even consider the statistics, and I believe that any hard user data is better than none at all. I think there are a lot of people out there who agree with me and have practical experience that would indicate strong empirical support for the notion, but a well-researched dissenting opinion is worth hearing.