Comments on The Limitations of Server Log Files for Usability Analysis

Alistair Harper — Tue, 20 Nov 2007 22:38:51 GMT

I am a bit surprised by the focus of this article. Do many people still spend time analysing web server logs? The limitations of this form of analysis seem to have been well understood for sometime.

I agree that there are limitations in using web analytics tools in usability analysis. Fundamentally, these tools tell you WHAT a customer did on the site not WHY they did it or HOW frustrating the found the experience.

I still think that there is great value in web analytics, but not so much in providing answers, but in helping provide a focus. An example could be the analysis an extended application process. A web analytics tool can be used to identify the points at which users are giving up on completing the process, down to a specific form field on a specific page.

This can then be reviewed. Maybe that page needs to be redesigned, maybe the field asks for information that users need to go and look up, or maybe the field is asking for personal information that some users do not want to give out.

Analytics may not provide many answers – but it can help you understand exactly what questions need answering

Andrea Wiggins — Sun, 02 Mar 2008 01:11:00 GMT

Speaking as a professional web traffic analyst (no longer practicing due to other ambitions), the use of server logs is definitely “deprecated” at this point – no web analytics professional today will recommend using server log data for much more than system function monitoring, unless it’s an extremely unusual situation. Client-side data collection significantly improves on ALL of these issues; appropriate vendor solutions make it relatively simple to sessionize even AOL visitors, identify returning visitors with reasonable accuracy over very long periods of time, and filter out bots, regardless of whether they can identify themselves as IE. And a whole lot more.

I appreciate the thorough analysis of why server logs stink. They do. That’s what the web analytics community has been saying for years, and why they are trying so hard to get better data collection methods and analysis standards adopted. Appropriately collected data and careful analysis (by professionals who know the weaknesses of the methods and tools) minimizes threats to validity and provides reliable insights. Not all usability insights require advance knowledge of user goals; many branches of social science have spent centuries inferring goals from behaviors, so this is not a sufficient reason to discount the potential usefulness of this particular behavioral data for usability.

I think that this is a sadly one-sided perspective which is ignorantly dismissive of client-side analytics, and implore every reader to investigate the vast improvements to web analytic technology that have come about in the last two years. Things have really changed a lot with respect to data quality; the attitude of this article is very frustrating because it may prevent people from exploring the full range of options with respect to user data. Usability tests are almost entirely contrived, and web analytic data is almost entirely naturalistic; I can see no reason not to use both in method triangulation, as each data source balances the weakness of the other.

Melissa Robison — Fri, 26 Oct 2007 16:56:44 GMT

Thanks for the article Karl,
You provide good and specific examples outlining the limitations of using server logs to accurately analyze a site’s usability, success, and/or user flows.

I agree with Peter and others above about the importance of looking at all available tools objectively, understanding their individual limitations, and then moving forward to extract the best value from all tools at hand.

While I’m not personally a fan of using server logs to measure usability, I’ve had to defend my position and discuss the pros and cons of using server log data with clients and peers. Often, I’m also in the position of recommending the best approach for clients to extract accurate user data from existing sources prior to planning a site redesign. Sometimes, the server logs or poorly implemented analytic software are the only data sources we have to review. The examples in your article will enable people in similar situations to analyze the data in an informed manner.

I think it’s also important to keep in mind that less experienced User Experience “experts” are consulting with clients or working on sites every day. In my opinion, this discussion is a valuable one for them.

Peter Krantz — Fri, 26 Oct 2007 13:40:57 GMT

Karl wrote: “First, how do you know those requests are being performed by actual people and not bots? If you attempt to say that you filter out bots, I will show you one I’ve written which identifies itself successfully as IE.”

Well, analytics software typically relies on javascript to log a request when the page is viewed in the user’s browser. This means that bots can be filtered from the collected data and that caching between the user and the web server are irrelevant. It is unlikely that bots implement a javascript engine. Saying that you can construct a bot to mimic that is like saying that I can instruct a user to lie in a usability test situation.

I agree that analytics software do not replace testing with real users but I still believe that you can get valuable data from them that can be indicative of user behaviour. This may help you design test scenarios better.

Peter Meyers — Sat, 03 Nov 2007 07:40:37 GMT

You certainly make some valid points about the limitations of web logs, but I really think you’re throwing out the baby with the bathwater when it comes to analytics. Yes, analytics aren’t very useful for getting a window on any individual user, but they are useful as one more tool for understanding people in the aggregate. A lot of the caching issues do work themselves out in the wash, just as noise does in any system where you’re collecting large amounts of data. Personally, I have gleaned valuable usability insights by tracking long-term trends in exit pages, bounce rates, jumps in error-page activitiy, etc. In some of these cases, analytics helped me pinpoint a problem that I didn’t know was there, and that problem was completely verifiable (in other words, it wasn’t an illusion of bad data).

Look at the flip-side as well. Is testing a single user (or a couple of users) in a traditional usability test perfect, statistically speaking? Not remotely. Using a couple of users to generalize to thousands or tens of thousands is wildly unreliable, especially since those users may not even be representative. Does that mean the technique is useless? Of course not. In fact, using these techniques in tandem helps reduce noise even further: if you take the hypotheses from testing users individually and reference them against the clues you get from aggregate data, you’ll end up with information that is all the richer and more valuable.

My grad advisor used to say that there were three levels of understanding statistics: (1) knowing just enough to be dangerous, (2) knowing enough to play by the rules, and (3) and knowing when the rules don’t matter. I’ve met statisticians who can tear apart absolutely any real-world analysis someone could ever run, and they’re all level 2 people. The problem is, if you carry that too far, you can’t get anything done. I think it’s important to look at any tool objectively, know its faults, but then move forward and try to extract the best value you can. Web analytics aren’t a new, unproven tool, and while many people misuse them, I also know some flat-out brilliant people deriving real value from them for both clients and end-users.

Nick Besseling — Fri, 26 Oct 2007 14:21:34 GMT

Hi Karl/
I tend agree with Cennydd and Steve.

While your in-depth analysis does really well to point out the limitations of analytics I think the idea that one would replace analytics with a usability expert or vice-vera is a wrong. They both serve different purposes and both have their faults and I really can’t see how they are directly interchangeable.

I also think a lot of the analyitcs software around these days adds a lot more value and are ‘smarter’ than what it used to be.

I see analytics as another tool that adds weight to certain directions and provides valuable trend and user system data.

In the IA/Usability field I tend to feel there is far too many idelogical skews for and against certain approaches (I have been/am guilty of it myself) and feel that collecting all the information (both qualatative and quantative) possible that time/cost allows will be far more helpful in coming up with a suitable direction and solution.

Striking out the only available source of full scale system/user qualatative data (despite the limitations) seems a little strong.

Flexibility and a broad perspective is always the best approach and that nothing should be ruled out (unless it really is stretching the boundries of usefulness).

But depsite these points the article is a useful one as it does point out a log of specific analytic limitations I wasn’t fully aware of.

Cheers.

Steve Fleckenstein — Thu, 25 Oct 2007 20:24:54 GMT

P.S. And Karl, my apologies for the inflammatory opening line to the first comment—it deserved the negative rating it received. There’s lots of good analysis in the article and it was irresponsible of me to toss the baby out with the bathwater, which I… er… think I accused you of doing.

Steve Fleckenstein — Fri, 26 Oct 2007 12:49:34 GMT

Cennydd, well stated. Karl, thanks for the response to my earlier comment. I agree with your last statement—we shouldn’t make changes based solely on analytics. I like Cennydd’s statement that analytics provide a good starting off point for future investigation (especially if combined with search log analysis and other information). Bots, caching, etc. all complicate things but again that doesn’t mean I should ignore web analytics completely.

Cennydd Bowles — Tue, 20 Nov 2007 22:40:01 GMT

I agree with a lot of what you say but, to me, this article reads a little bit like one of those “BREAKING NEWS: Wikipedia may not be entirely accurate” stories. If you write a Masters’ dissertation based on a Wikipedia article you’re a fool. If you base any kind of big design decisions solely off log files you’re a fool. But both *can* be a useful guide, a starting off point for future investigation. IAs are pretty adept at gathering data from numerous sources and appreciating the strengths and weaknesses of each observation. I’d content that analytics can contribute to this. To write it off completely, as this article does, seems a little over-zealous.

Karl Groves — Tue, 20 Nov 2007 22:40:10 GMT

No, it would not be reasonable to do so. First, how do you know those requests are being performed by actual people and not bots? If you attempt to say that you filter out bots, I will show you one I’ve written which identifies itself successfully as IE. Second, you say this content page is being requested many thousands of times. To most proponents of analytics, this would seem to indicate success, would it not? After all, analytics tools only report on successful requests, right? So if it is successfully being requested, why move it? Last, and always, without knowing why the user is doing something, there’s no way to determine whether the request was successful or not successful. Exactly what brand of analytics tool, through client-side means or server-side, does it tell you the user traveled to that page and then realized it was or was not what they were looking for? Without that information, any decision to modify the site due to analytics is purely driven by conjecture.

Steve Fleckenstein — Thu, 25 Oct 2007 16:29:28 GMT

Here’s an example. I discover from analytics (regardless of method) that a certain detailed content page buried in my site is requested many thousands of times. I’m using the word “request” specifically here to indicate my agreement about the inherent limitation of analytics. I also know that a link to this page does not appear anywhere in the top 3 levels of the site. Even without user testing (which is important, I’m not trying to debate that) would it be reasonable to draw the conclusion that users would be better served if I surfaced this content to a higher level?

henrik persson — Thu, 25 Oct 2007 15:55:48 GMT

The article makes a great point and backs it up very well. I was hoping for a possible solution or alternative though.

Matt Probert — Thu, 25 Oct 2007 15:04:33 GMT

An excellent article. Rather wordy, but providing excellent supporting evidence for the points it makes. Which in short, are, http log files provide data ONLY about http server utilisation, which bears no resemblance to web site usage.

Karl Groves — Tue, 20 Nov 2007 22:40:26 GMT

As I say at the top of the article: “I realize that some analytics software gather data by other means than parsing log files and may in fact contain some features meant to overcome one or more of the criticisms I outline throughout this article. I do not discuss such programs, primarily because there is little consistency between them and ultimately they are just as poor at gathering real usability data as analytic tools which parse log files.”

Regardless of how the data is captured, it is unrealistic to expect to glean usability data from analytics. Without knowing what task a specific user is attempting to do, there’s no way of judging whether they’ve actually done it. Analytics tools only record requests. They can’t record whether that request did or did not satisfy a user’s need – only the user themselves can tell you that.

Steve Fleckenstein — Wed, 31 Oct 2007 00:44:47 GMT

The title of this article should be “Log files: 99% Bad” because, like Nielsen’s article of similar name this one seems to have been written in 2000. While you focus on log files—which I agree have a lot of limitations—your article completely ignores client side analytics tagging (which addresses many caching issues and which is used by thousands of sites). And while I agree that it is difficult to measure online success via analytics (but that too is possible to some degree with advanced features of some web analytics tools that can track a purchase funnel) I can still get a lot of great info out of web analytics tools that help me improve the site for users. A more balanced view would have been helpful—“log files aren’t all that helpful but client side tagging is a viable workaround.”