Comments on Report Review: Nielsen/Norman Group's Usability Return on Investment

Liam Friedland — Fri, 26 Jan 2007 14:51:36 GMT

A couple of things strike me about this piece:

1. Delving into the granularity of ROI for specific UE deliverables is a good idea. If you don’t measure it, you can’t improve it. On the other hand, it may not always be necessary or practical to get to the level of detail described here. Your company may not care about the minutae of ROI, or not care to support your drilling down that much, or may just want to build usable products because they’ve been burned enough in the past to know better. If this stuff is necessary for survival, then do it. If not, then screw it. Focus your efforts on building better product instead, or revamping the company website, or improving the corporate identity, or working on business strategy, or doing detailed observational studies to gain better insight into customer behavior, or benchmarking usability improvement metrics between releases, etc…

2. It’s often enough to discuss ROI anecdotally to make your point in business (how many other disciplines are guilty?). Of course having a set of detailed, real-world case studies on ROI would be great—but wouldn’t necessarily ensure anything. If your CEO doesn’t buy your approach, you’re screwed—regardless of how you spin it. This is true for all disciplines & players in the business arena.

3. IMO Messrs. Hirsch & Merholz are too focused on ROI here, and not enough on how UE can make a strategic impact by most effectively leveraging its outputs to drive cross organizational collaboration, and business strategy. I think it’s important to drive home the ROI measurements, but the bigger picture is that if done right, we are squarely impacting product definition, program management, and business strategy.

4. The authors missed a key piece of literature on the ROI for usability. There is a whole tome on the subject: “Cost-Justifying Usability (Bias & Mayhew, 1994). Actually, one would think a thorough literature search on the topic would be in order—particularly if one is going to throw stones. There are many other sources that deal with the ROI for IT projects which can be leveraged for our discipline.

5. Finally, when purchasing anything (“research” included), the old adage—caveat emptor—should always be applied .

Ron Zeno — Fri, 26 Jan 2007 14:51:36 GMT

Bravo Eric! Wish I had said all that.

I’m very disappointed at the direct and indirect ad hominem arguments against Peter and Scott. That’s not critical thinking, it’s just the opposite – ignoring the evidence and detracting the discussion to other directions.

I feel NNG can produce high-quality information, helpful to all. Sadly, it won’t happen unless more people like Peter and Scott professionally and critically assess reports and articles such as this. As long as low-quality reports and articles are tolerated, we’ll keep getting more of them and the entire field will be worse off because of it.

Eric Diamond — Fri, 26 Jan 2007 14:51:36 GMT

Elizabeth McLachlan brings up some interesting points. Of course we all should acknowledge that in this field, Mssrs. Nielsen and Norman are the 800 lb. gorillas of usability. Like Microsoft, and other high visibility “industry” leaders it is natural for us to pile on the criticism, as we are enthusiasts. It is natural enthusiast behavior. Nielsen and Norman are to be commended for moving the field forward and by providing the raw material for interesting and productive discussion. Still, it is somewhat irresistable to play David to their Goliath, and hopefully they take some comfort in the fact that if they weren’t so successful and influential, they would not be the target for so much critcism. But, like J-Lo and Ben, visibility invites scrutiny.

That being said, Elizabeth wrote, “The methodologies used by NNG and the statistics presented can of course be questioned and can also make a skeptic out of all of us.” Not only can they be questioned, they SHOULD be questioned. Repeatedly. Thoroughly. Deeply. To do anything less is to abdicate our responsibility as professionals. Boxes and Arrows is the closest thing I have seen to a peer-reviewed journal in this field, and while obviously the same peer-review rigor is not applied here, the fact that criticism is part of the mix is not only valid it is vital, especially for leaders in the practice. NN group, as professionals *profess* their point of view and it is up to us to either accept or challenge their assertions. It is the basis of critical thinking and the foundation of our practice.

Though it is important to examine ones own methods, especially to justify what is essentially a “soft” metric, reviewing and critiquing methods in a constructive wy can only help us in our quest to find the “Philosopher’s Stone” of User Experience practice: ROI. Research and assertions that cannot stand up to scrutiny help foster the idea that like perpetual motion, measuring ROI is an preposterous endeavor, one that will help to erode our credibility and pereceived value in the marketplace.

She goes on to write, “Then they charge people for it [the report]. I’m certainly not interested in sussing the credibilty of that data or in the methods used to not only gather it but display it.” I would respectfully disagree. The data is the support for their assertions. If it is not sound, it calls into question the validity of the assertions. And it is one thing to get faulty research for free, but NN group charges for it, and they charge a lot. It is not too much to expect that a $500 report contain valid data, sound methodology and valid conclusions.

Peter et al, have done both our profession and potential clients a service by taking a stand that simply having a reputation is not a license to publish questionable research. It also makes a statement to the world that we as a profession value our standards enough to ensure the products of our work stand up to scrutiny by peers. It ultimately earns us (and Boxes and Arrows) more credibility as a community of professionals, with standards and practices.

And that to me is a pretty good ROI.

David Dunkle — Fri, 26 Jan 2007 14:51:36 GMT

Analogy Time!

On one hand we have usability designer/practitioners, who come from a variety of backgrounds, who endorse a variety of yet-unproven methods, and who seem to gain notoriety by making bold, brash, sweeping pronouncements of a usability:ROI correlation that appears to explain maybe ~10% of the ROI variability.

“10%” ...hmm, where have I heard that before…

On the other hand, we have advertisers and marketeers, who invented the jewel, “Everybody knows that only 10% of advertising works, but nobody knows which 10%.”

Are these fields merging? Are there lessons to be learned from the dark side? The ad-men always seem to have the ROI spiel down pat.

elizabeth mclachlan — Fri, 26 Jan 2007 14:51:36 GMT

Interesting that this review appears in a place where the primary audience consists of UX/IA folks.

The methodologies used by NNG and the statistics presented can of course be questioned and can also make a skeptic out of all of us. NNG has taken a lof “data” and presented it in a way that exemplifies the points they want to make. Then they charge people for it. I’m certainly not interested in sussing the credibilty of that data or in the methods used to not only gather it but display it.

Obviously, the findings of the NNG report have somehow caused hurt to be felt among some of the UX crowd. But also quite obviously, the NNG report it isn’t the only thing there is to read about ROI.

Defending the profession of “user experience research” requires first that you take a look at your own practices, research methods and busines abilities to not “defend” but rather support your profession. Are you doing a good job of your job? Perhaps you will choose to get snarled up in the provability of your research, findings and recommendations; perhaps you will spend a lot time defending your recommendations to a client, and proving its worth. Or perhaps you will spend time “reviewing” an NNG report that was published 7 months ago. We pick our battles.

Peter says on his PeterMe blog that this review is “one of the best things I’ve ever (co-)written”. Why? Because you believe you were successful in debunking an NNG report to well-oiled audience? So, what’s the return on that investment?

Christopher Fahey — Fri, 26 Jan 2007 14:51:35 GMT

Good points, Heidi, I was wondering how to say what you said in your last paragraph.

IAs take it for granted that a positive ROI for usability actually exists, or that high-quality products are always a good business idea. But this assumption may not even be true. There are countless businesses in the world for whom deliberately providing low-quality products and services is a critical ingredient of their business model. I am thinking of cheap fast food, discount big-box stores, generic or knockoff consumer products, free web hosts, Kurt Russell movies, etc.

We IAs who are committed to building high-quality products have to recognize that companies who specialize in low-quality products may never have any justifiable use for a strong investment in our skills.

heidi adkisson — Fri, 26 Jan 2007 14:51:35 GMT

As an interactive architect with both academic (MBA) and practical (senior management) experience in business, I am leery about making a case for usability based on ROI-type calculations (benefits minus costs). Cost-accounting is a very complex endeavor—decisions made along the way on how to treat and allocate costs can create a lot of variation between how one accountant would do it vs. another. Attributing revenue increases to particular causation factors is also tricky business—it is too easy to mistake correlation for causation. I think you could spend a lot of time quantifying ROI a given set of site improvements and ultimately come up with an ROI figure that would not endure accounting scrutiny. I believe, by the way, that the cost-savings and revenue-enhancing benefits of usability engineering are real. I just think there are some inherent pitfalls in the quantification approach if your goal is to argue the value usability has to an organization. I would love to see a thorough business case study on usability ROI, one that details the accounting methodology. I’ve not seen anything like this. I believe that the criticisms this article directs at the nn/g study could also be applied to the figures presented in the Aaron Marcus whitepaper .

In answer to Christopher Fahey’s question about the abuse of ROI in other contexts, this also happens with vendors of Customer Management Systems (CMS). The ROI claims for these systems often don’t pass accounting scrutiny.

I suspect that the ROI argument for usability itself does not have a positive ROI (in terms of its ultimate effectiveness). In Seattle we are fortunate to have an artistically acclaimed opera company that is also consistently on-budget (even this year when many arts organization are struggling with deficits). Over the years, it has built a diverse audience base (including many folks in the 20’s and 30’s) in part through its education and outreach programs. Opera education programs are costly and have almost zero associated revenue. How does Seattle Opera justify a program with no short-term return? They have a clear, strategic vision about audience enjoyment and are willing to invest long-term in that vision. Maybe that’s the type of thinking that needs to be needs to be present in an organization before usability become institutionalized: that no amount of quantitative data will convince an organization of the value of usability if that organization isn’t strategically focused on providing a high-quality customer experience.

Brenda — Fri, 26 Jan 2007 14:51:35 GMT

Hey Joe-

If this is your way of offering to be a (volunteer) copyeditor on the (all-volunteer) staff of this (no-fee) publication, we’ll be glad to have the extra pair of hands. Just say the word and we’ll add you to the roster.

Joe Clark — Fri, 26 Jan 2007 14:51:35 GMT

Copy-editing: Every instance of “before and after” plus a noun has mangled hyphenation (never seen space-emdash used before). “the case infers that these purchases would have otherwise been made”: Humans infer; the case might suggest or imply.

There are others, of course.

Greg — Fri, 26 Jan 2007 14:51:35 GMT

Great article. I think this is one of the single biggest challenges facing the IA/UX field in the quest for broad acceptance and growth. As an MBA who has led several UX projects and has a good awareness of the field, I have been consistently concerned with the lack of effort given in trying to set ROI metrics for usability projects during the early stages of this field. As a lobbyist of good IA/UX within the business community, it is crucial to have hard numbers that can be delivered. If managers are given a true positive ROI proposition, they would be hard pressed to NOT approve the project. If IA/UX professionals can get to this point of justification, concerns about growth within the community will be a thing of the past.

Derick Sumrall — Fri, 26 Jan 2007 14:51:35 GMT

Based on Peter Merholz and Scott Hirsch comments, they seemed to have an expectation level for the report to strategically breakout of all the usability recommendation that led to ROI. This sounds great… but, to vividly breakdown and report the number of sites that were featured, it would be to cost prohibitive and a research nightmare to justify writing a report of that magnitude. If they were expecting that amount of information, they need to look for a book. —-Tell us when you find it… ;-)

I think the goal of Nielsen/Norman Group’s report was to make a general assessment that would inspire the reader to go and interact with the available sites. Merholz’s and Hirsch’s review reads as though they got caught up in having paid for such a light report. I had to ask myself, “Did they forget to actually go and interact with the available featured web sites?”

To be fair, I recognize other influencers were taken for granted. What about brand experience? Did usability recommendations add to brand perception? In some of the case studies, you can connect it. Having roots in software engineering, Nielson/Norman Group has a tendency of distancing itself from “how” functionality can heighten brand awareness (versus user experience). So, Yes… you could argue that brand experience was slightly discounted by the Nielson/Norman Group report.

Anyway, Usability is about user interaction/perception and it seems Merholz and Hirsch was reacting to some of the limitations of a flat report. To their credit, I agree with their points concerning the report’s case study selection and measurement process. Further, I think that a more tactical approach with fewer sites would have allowed more design context to be established. With this context, I think several of Merholz’s and Hirsch’s issues would have been addressed.

So, I have to ponder what both sets of arthur’s would answer: Tactically (2 to 3 focused sites) or Strategically (10 to 15 broad vision sites), How would you address a report on usability and ROI?

Derick Sumrall — Fri, 26 Jan 2007 14:51:35 GMT

So, I have to ponder what both sets of Arthur’s would answer: Tactically (2 to 3 focused sites) or Strategically (10 to 15 broad vision sites), How would you address a report on usability and ROI?

Christopher Fahey — Fri, 26 Jan 2007 14:51:34 GMT

Great article.

My impression from this review is not so much the (scandalous!) idea that NN/g practices slipshod methodology… what I have learned is that measuring usability ROI is quite often nearly impossible. In particular the “mitigating factors” section pulled the rug out from under almost every ROI calculation/model I’ve ever heard of (there are few usability changes that are not accompanied by simultaneous business, design, technological, and/or marketing changes). This article leaves me even more skeptical about *any* consultant that claims that usability can be statistically tied to revenue… wearing what Jesse James Garrett called “the lab coat”.

ROI for usability may be no easier to prove than, say, ROI for any other “production quality” choice in any other industry, from the strength of the threads used to sew a t-shirt together to the amount of sawdust added to a fast-food hamburger patty.

Is the fudging of ROI metrics really that uncommon in the usability world? Or is it, as I beleive, far and away the norm?

Or, for that matter, is crappy ROI methodology really that uncommon in other industries? ROI may be a business concept that is abused universally, not just by usability pundits.

My firm sells ourselves as ‘experienced and smart’, not necessarily as ‘impartial and scientific’. We are user experience designers, not research scientists. This article only confirms my suspicion that most of what we do in this industry is difficult to measure with a large degree of scientific rigor. There is always a huge space where innovative and clever people fill in the gaps.

I’m not saying that it is *always* impossible to measure usability ROI, but I think this article is suggesting that it is a immensely more difficult task than many of us imagine—and that it is, indeed, often totally impossible. ROI numbers analysis can *inform* but will never replace smart business and design sensibilities.

The article’s opening statements, which I think are meant to be facetious, end up seeming provide the best existing rationale for investment in usability:
“Of course you’ll sell more products if they’re more usable! Or you’ll decrease costs because of heightened productivity! Exactly how much will you profit? I don’t know, but don’t you want to build the best product you can anyway?”

(I had my doubts about this article before I read it, seeing that one of its authors is a partner at a firm that competes with NN/g. But after reading it these doubts were dispelled.)

(I also am chuckling at the idea that there may be tons and tons of similar “evidence” DISproving NN/g’s thesis: where a ton of money was spent on usability and yet sales remained flat, or even decreased dramatically.)

-Cf

Beth — Fri, 26 Jan 2007 14:51:34 GMT

My jaw was dropping when I heard Nielsen make some of these ROI claims at the DC world tour a year or so ago. I’m a math major, and although there weren’t written slides that showed his math, they just sounded bizarre. I’d hoped that I just misheard. Apparently not!

Ron Zeno — Tue, 28 Aug 2007 00:25:36 GMT

First, NNG hasn’t demonstrated much credibility in discussing roi: http://www.useit.com/alertbox/20021111.html

Second, as Peter more clearly states in his weblog, “The problem is, we never learn the cost necessary for achieving this improvement. And you can’t calculate a “return” if you don’t figure the initial investment. All we know is how much it improved, not what it cost to get there.”

Third, these roi arguments look more like marketing ploys rather than honest attempts at actually presenting a credible roi argument.

Fourth, given how difficult it is to actually create a credible roi argument, user experience practitioners are far better off working on improving their own skills and capabilities.

Fifth, I agree with Peter’s comments in his weblog: Reports such as NNG’s hurt the credibility of all user experience practitioners.