Comments on Usability Heuristics for Rich Internet Applications

Jim Kalbach — Fri, 26 Jan 2007 14:51:29 GMT

Good application of some old work, but the limitations of Mr Nielsen’s heuristics have not been addressed or expanded on.

1. These ten heuristics by and large ignore any emotional aspects of design and interaction. Instead, they are firmly rooted in a cognitive HCI perspective. Research in emotions over the last few decades, however, shows a tight connection between feeling and thought. Don Norman – the other half of “NNG” – has certainly picked up on this. Unfortunately, these heuristics do not address affective concerns of UI design.

2. They do not account for a larger cultural context and may not apply universally. They instead reflect Mr Nielsen’s western cultural perspective. Not everyone in the world values a minimalist design, for example. Or, high uncertainty avoidance cultures (a la Hofstede) may or may not require more control and freedom, but instead the opposite.

(Don’t forget the first two “w’s” in “www” stand for “World Wide.”)

In the end these heuristic are concerned with the syntax of UI design and not with the sematics of the application in question from a contextual user perspective.

Cheers,
Jim

Andrei Herasimchuk — Fri, 26 Jan 2007 14:51:29 GMT

I love it.

“The fact that animation sometimes works doesn’t negate the bigger point that applications are only engaging (or useful, or profitable, or whatever criteria you use to judge them) if users can successfully perform their task.”

Well… of course.

“Remember that we tested 46 Flash applications on three continents and found an average success rate of only 45%.”

Meaning? Poor design? Is Flash inhernetly the culprit? I could easily test any arbitrary mix of 46 desktop applications on three continents and get an average success rate of 45% as well.

The point?

“Violations of basic usability principles will prevent users from being able to use an application, particularly if it’s something people haven’t paid money to acquire.”

Well… yeah.

And then you post two links, one of which takes us to a page on your site where you want us to pay money to read a report that somehow tells us something less obvious than what you posted here.

I’m *REALLY* doing my damndest to hold back on saying something that I know will get me booted off these boards.

But you are making it real hard Jakob.

Andrei

Eric Diamond — Fri, 26 Jan 2007 14:51:29 GMT

Jess, et al:

It’s hard to disagree with your definition of “gratuitous” animation, but I think it is like saying, “don’t design anything bad.” Yes of course, gratuitious anything should be avoided if it does not contribute to the experience. It is why our friend “Skip Intro” is more reviled with each passing day. (poor Skip…). And I want to make it clear that my intention is not to pick on you or your article, which was pretty good, all things considered. But I’ve been thinking a lot about this subject ever since I asked Don Norman (at Living Surfaces, 2000) “Are you ready for the Usability Backlash?”

My problem is the problem of rules. Like guns, fission and other potentially dangerous ideas made manifest, the problem is not the rules in and of themselves. It is what happens when they fall into the hands of people who don’t understand the rules in context to what they represent and who then go on to use them in irresponsible ways.

We live in a society of lists. We think in Powerpoint bullets. Look at any magazine cover, and you’ll see what I mean. Lists and bullets become a substitute for thinking and rhetoric. At my workplace we used to do these Heuristic reviews where we would rate a web site based on criteria like usability, brand, design and so forth. The numbers would be averaged into a score and the client presented with that score. The number was worthless (IMHO) and though I did my duty and filled out the form, I always, always included a 3-5 page written critique along with it. I felt that my years of design experience would be more valuable to a client than a score based on how well they followed the rules.

Rules are great until they restrain our thinking, or worse, substitute for thinking. (see Terry Gilliam’s film Brazil for extreme yet entertaining examples of rules gone awry and what happens when design thinking is constrained by rules.) Though I understand the intent is that the rules become a jumping off point for discussion and serve some usefulness, we must be aware of how rules, once codified and released into the world can be misinterpreted and misused. My caution for the community as a whole is that we must be careful when we write them, and only do so when we absolutely need them, and when we write them we do it carefull and with posterity in mind.

Like I said, I don’t have much to complain about in the rules you did present. I am asking a bigger question as to whether we need rules at all?

For example, you write, “While RIAs can offer novel metaphors, novelty often slows usefulness and usability.”

I would suggest that novelty does not slow usefulness, rather confusion and lack of clarity hampers usability. Novelty is what makes us want to pick up something and play with it, examine it. For RIAs this is extremely important, because unlike a website I am directly manipulating data. As a designer I *want* to entice you to play with it. I want you to watch the interface build so you learn where things are and what they do, without me having to explain them with “documentation.” Is such animation gratuitous? I would say no, the animation is educating the user on how the interface is laid out and used.

Problems arise if I make assumptions about what has meaning for the user without investigating or testing, and my metaphor does not entice and engage, but instead confuses and alienates. We are humans and humans thrive on novelty. Novelty can be an essential contibution to usefulness. The trick of design is to balance novelty with familiarity. Perhaps rather than a rule we could come up with characteristics of good RIAs and say that good RIA design balances novelty with familiarity. Go too far in one direction and you alienate me. Go too far in the other and I will be bored and fatigued.

Now back to the rules. So some client gets hold of the Heuristics and says, “I don’t want novelty. Novelty will slow usefulness.” Does that really serve usability? Or does it just further constrain designers from creating somehing really useful? We must be careful about how we make rules, and what we do with them.

OK, one more thing—about the animation issue. I agree with you Jess, and in your reply to Andrei, you summed it up in a very elegant way: you wrote, “don’t prioritize animation over user goals and tasks.” THAT should be the rule. It is so much more elegant and empowering than “use animation sparingly.” Especially to a cranky designer like me. ;)

Eric

David Meeker — Fri, 26 Jan 2007 14:51:29 GMT

I think there needs to be a clear distinction between usability issues and / or design issues and the technology itself.

It is not a fair statement to say that RIAs have usability issues by nature. Usability issues are caused by design, and not by the technology used to create a design.

I agree that poorly trained flash developers, or designers that are unskilled have created work that is highly unusable.

The same case can be made for the use of HTML, C, Visual Basic, etc. Having made this point, RIAs are more than just a product of Macroemedia Flash. RIAs are the future, and it is the responsibility of the design community and the leaders in our industry to guide the use of RIAs. They are indeed the future of thin clients and design practices should adapt to the technology.

You MUST acknowledge that human computer interaction must be a fluid topic. As humans become more skilled in their use of computer systems, their interactions and ability to interact becomes much more sophisticated.

Best,

Dave

David Heller — Fri, 26 Jan 2007 14:51:29 GMT

What I was hoping w/ my previous article here in B&A is to form a community of people who are interested in doing RIAs, who need support in convincing their groups of their value, and sharing solutions that RIAs offer that other mediums don’t.

But I also want to not create yet another niche b/c of a buzz term. To me, whether it is mainframe, web, client/server, RIA it is all software and the while the technologies and methods change, the end-user’s goals and motivations are pretty much evaluated the same way and thus the experience directive is the same.

What is particular about RIA (regardless of technology) that is going to allow us to address the question of experience (utility, usability, and engagement).

Something that Nielsen’s criteria regarding Flash assumes is that Flash exists in the browser. What happens in the world of “Central”? Does that change expectations so considerably that we should re-evaluate the Heuristic criteria? Specifically the issue of “backward Navigation” is an interesting one. It is a pure web concept, though it basically translates to the thick client equiv of cancel.

Jakob Nielsen — Fri, 26 Jan 2007 14:51:29 GMT

Back in 1995, I wrote an article listing 7 ways in which animation can be good for user interfaces: http://www.useit.com/alertbox/9512.html

Our more recent project testing how children on two different continents used 27 different websites certainly confirmed that kids often (but not always) like animation: http://www.nngroup.com/reports/kids/

The fact that animation sometimes works doesn’t negate the bigger point that applications are only engaging (or useful, or profitable, or whatever criteria you use to judge them) if users can successfully perform their task.

Remember that we tested 46 Flash applications on three continents and found an average success rate of only 45%. Violations of basic usability principles will prevent users from being able to use an application, particularly if it’s something people haven’t paid money to acquire. (Microsoft Office can get away with more usability problems because of its higher price and lower level of competition; neither of which are typically true of Web-based applications.)

See:
http://www.macromedia.com/newsletters/edge/december2002/a…
and
http://www.nngroup.com/reports/flash/
for more details of this user research.

jess — Fri, 26 Jan 2007 14:51:29 GMT

>The article itself: There is NOTHING in this article that is about RIAs.

>Look at the work that MS is doing w/ their Sharepoint 2003 environment and you’ll see so much richer an environment than even many of what Macromedia calls an RIA.

Just to point out – RIAs don’t depend on Flash. We talked about Flash because it’s Grant’s expertise and it currently dominates discussions of RIAs. However, go to Forrester and read up on their X-Internet coverage (executable internet) and you’ll see a lot of alternatives.

However, Dave’s point that our article could apply to a wide number of situations is true…as could Keith Instone’s article about the heuristics applied to the web. I hope that the article does help as a springboard for more discussion about RIAs and user experience, and provide some simple starting points for folks outside the UX community who are developing rich apps.

David Heller — Fri, 26 Jan 2007 14:51:29 GMT

Ok, I can’t resist. I have to speak up. (I also invite people to take this conversation to my webGUI list which while stale has a good # of people subscribed who would find this discussion incredibly interesting: webgui.htmhell.com to subscribe, webgui@htmhell.com to post. I will reiterate my comments there.)

Ok, my points.
1. Animation: I think there are TONS of good examples out there that show that transitional animations are an important aspect in usability and probably one of the main reasons why RIA’s are better than both client/server and previous HTML thin-clients. The phrase is “gratuitous” a vague sentiment, so lets not get all bent out of shape about it.

2. The article itself: There is NOTHING in this article that is about RIAs. Sorry but its true. As someone who is a HUGE advocate of RIAs I was so excited that someone was continuig the discussion on B&A in this way, but I was duly disappointed in the content of the article in that there isn’t anything I can’t do in an IE based DOM2 system that you mentioned in a context of saying this is about Flash. Look at the work that MS is doing w/ their Sharepoint 2003 environment and you’ll see so much richer an environment than even many of what Macromedia calls an RIA (I bet they are loving that we are using this acronym as if it is real). I would say that if you took RIA out of this article EVERYONE should read it to be reminded of what they should be doing in all of their apps. Thick, then, RIA, mainframe, other.

3. Examples of good usable flash. The favorite demo I have seen of late is at www.laszlosystems.com. Their calendar demo rocks and shows the real power of a Flashbased RIA in terms of using transitional animations for usability AND in terms that Flash affords us the one thing that most people complain about, namely the ability to build your own pattern language instead of being stuck in either previously bad webForms or Windows/MacOS X based forms. Yes, it is NOT perfect, but it is a great example that takes the iHotlier model that is used as the centerpiece of RIAs by most articles writing about it and then pushes it a step further by increasing contextual cues and making a much more engaging environment.

4. Engagement. RIAs are about engagement more than anything IMHO. This is something that is so missing from Nielsen’s vocabulary it is ridiculous. Engagement is what makes an application’s user experience move to the next level. This is the stuff that can’t be tested for in the lab b/c engagement is about use over time. Is this application a fast-food restaurant or not? Word, Outlook, even the web browser are fast-food restaurants. They work, I go there often, but I don’t want to stay there. MacOS X is an attempt to create a space to live in, not just visit and to me this is the next area to explore and RIAs are one way to begin to do this, for apps that require ease of deployment, networked inter-communication, engagement and ease-of use.

Jed Wood — Fri, 26 Jan 2007 14:51:29 GMT

To take Jess and Grant up on their closing sentence: “We’re looking forward to great new things— if you’re already doing them, drop us a line in the comments.”

I gave a presentation at the UPA (Usability Professionals Association) conference last month highlighting some of the ways Flash can actually help increase usability. I’ve grown tired of hearing statements like “well, Flash has it’s place. Its good for things like games and advertisements.” Hats off to Grant for his work in pushing Flash into the realm of serious applications.

A meager site with some of my examples can be found at www.usableflash.com. If anyone has an example they want showcased, please drop me a line. The example that is probably of greatest interest to readers of BoxesAndArrows is our web-based card sorting application (www.websort.net). I consider a compliment when people don’t realize it’s done in Flash.

Finally, as per CHris’ comments regarding animations that are less than 1/4 of a second- I agree. I’ve been amazed at how a very fast fade-in can really help direct the users attention better than instantaneous appearance, which can sometimes be confusing.

-Jed

jess — Fri, 26 Jan 2007 14:51:28 GMT

Hey Andrei,

Maybe I am missing something, but I don’t disagree particularly with you or Eric. I think this is largely a semantic quibble about the intent of “sparingly”....next we could ask “what is IA”. Just kidding.

We’re not saying never use animation. Just don’t prioritize animation over user goals and tasks. The Aqua transitions (and Windows XP transitions) don’t get in the way of user tasks and provide subtle continuity and feedback. Another case where animation can *assist* the user would be in information visualization apps.

I’m a big fan of animation, and think it’s one of the advantages of the Flash platform. However, I still would argue that the *default* assumption about elements is that they are not animated. Then when the decision is made to use animation it’s informed by user goals and tasks, instead of a technology-centric “we can so we will” mentality.

Here’s an example of the kind of animation & transition I’m talking about (it also has hide & seek issues)
http://www.kcount.com/nutrition/index.asp

Where’s the global navigation? Hiding. Why? I’m not sure, but I suspect the reason is to display a cool animation.

Animation is like bold type. When animation is used indiscriminately throughout an application, it loses its impact. It’s a powerful tool that will benefit users most when applied (here it is again) sparingly ;-)

Instead of trying to reword that paragraph by moving “gratuitous” around, maybe it would be clearer if we reframed it as a suggestion when to use animation, rather than when not to use it…

How about:

At their best, animation and transitions can provide powerful information visualization, functional cues, and a sense of continuity through subtle feedback. However, animation’s impact and value can be significantly lessened through overuse. Use animation when it will support user goals and tasks. A focus on goals helps ensure that the decision to use animation improves the user experience, instead of interrupting it.

Thoughts?

CHris — Fri, 26 Jan 2007 14:51:28 GMT

I’d like to throw in my two cents on this whole animation debate. Animation can be very benificial to the user’s ability to comprehend the UI of an RIA. Is a collection of data is ‘moved’ through an interaction, some animation can help the user to ‘follow’ the location of that data. In OSX the animation of windows minimizing into the dock allows users to follow where the window is going when they click the yellow traffic light icon. This type of animation is ok, with limitations. Animations that are used to convey UI features or state changes should be quick so as not to delay the user from their next task. In the Home Automation UI I am developing all animations that relate to state change take less than a quarter of a second (and they are few in number). At that speed there is almost no delay in the use of the UI and the animation are just long enough to register with the user.

Now, animation for animation’s sake is different than using animation to address changes in the UI’s state. Animation for animation’s sake means elements like ‘flaming 3D company logos’ and the like. This type of animation has been shown to distracts the peripheral vision and draws the user’s eye away from the content. Of course, this type of animation is also fairly common in many Flash sites, but not Flash RIAs.

Andrei Herasimchuk — Fri, 26 Jan 2007 14:51:28 GMT

“The key in that sentence is “gratuitous” and “disrupting task flow”. That’s the context, and if animation are done just for animation’s sake, and interupts the user, then it isn’t a good thing.”

If you meant what you said, then the rule should state:

“Gratuitous animation and transitions should also be used sparingly. While animation and transition can make a great demo in the boardroom, gratuitous animation will provoke user frustration…”

The point Eric made, which I’m not clear you either got, or just simply disagree, is that animation should not neccessarily be used sparingly in the general context. The actual user context matters too much to make a broad sweeping rule like you did.

Changing the rule to how I stated would then force you to define what you mean by “gratiutous” rather than statiung that all animation should be used sparingly.

Here’s an example of how animation for animation’s sake might actually be fine by the way… In the new Mac OS X Jaguar, Apple added an ever so slight four frame or so transition to all dialogs that switch preferences. Like the System Preferences control panel. It happens fairly quickly, and is quite nice to look at actually.

It adds next to nothing however in overall usability, and there might be some who feel it adds a small amount to help people know the contents of the dialog switched. but the dialog changes shape and you just clikced on a button, so how much does it really add?

Andrei

jess — Fri, 26 Jan 2007 14:51:28 GMT

First, let me say thanks for the great comments so far…

The intent of the article (and of heuristics in general) is to help people think about what makes good design. It isn’t to decree a set of unbreakable rules, but to provide a framework and some guidance when you have limited opportunity to do user research.

However – we meant what we said when we said

“Animation and transitions should also be used sparingly. While animation and transition can make a great demo in the boardroom, gratuitous animation will provoke user frustration. The time to animate an element interrupts a user’s concentration and engagement in their task. Disrupting task flow significantly impacts usability and user satisfaction”

The key in that sentence is “gratuitous” and “disrupting task flow”. That’s the context, and if animation are done just for animation’s sake, and interupts the user, then it isn’t a good thing.

Also – a Teletubbies game is probably *not* an RIA by our definition (ie about data manipulation). My own 4 year old loves Flash games, but she’s not manipulating data.

Finally, I’ll reiterate – RIAs aren’t about getting attention, they’re about getting things done. That task focus means that animation should be used when it helps users accomplish their tasks. And that often means less is more.

cheers,

Jess

Eric Diamond — Fri, 26 Jan 2007 14:51:28 GMT

Where Usability rules are concerned I say to the community, PLEASE STOP THE MADNESS! As design professional who works on both sides of the user experience profession (as a creative director and as a practicing IA), I see our profession being reduced to that of being gatekeepers and rule-enforcers. I read Jakob Nielsen’s (expensive) treatise on Flash usability and what I came away with is the sense that Mr. Nielsen has no sense of imagination, nor design. He is like the color-blind curator of a modern art museum who only likes monochromatic impressionism.

This article is less egregoius than most on usability, as the authors tend to deal in broad general principles rather than in gratutious “rules.” But then they clearly step over the line with:

Without knowing who the user is, what the user’s task is, the context of the task, or anything else about the user, how can they make a statement like that? Teletubbies Flash applications on the BBC web site and other games are full of gratuitious animation, yet they are completely appropriate for my delighted 3 year old. Having a quick and timely exit would be inappropriate for a user who is only learning to transition from activity to activity. Context, folks, context.

Rules of thumb are good. Heuristics are good. they allow us to share our experiences as UE professionals and discuss ideas and principles. (and I know this is what the authors intended) But the dark side of heuristics is that we rely on “the rules” where we should be spending the time and resources on understanding the user. If I had a dollar for every time an engineer waved the blue and green book at me and told me that navigation MUST go down the left side….

The design process is a structured inquiry that solves problems. As UE professionals we are part of the design process. We should provide flexible frameworks for designers to work within, not blunt rules to follow. Those IA’s who wish to make a name for themselves in developing better Flash apps should work in broad principles. Let’s seek to understand what goes into a successful experience. Let’s not start an arms race to apply HTML-style “usability heuristics” to what should always be a disciplined inquiry.

Flash is a new world. RIA’s marry the flexibility of desktop software with the currency and dynamism of the internet. the limitations imposed by HTML are gone and with it, many of the rules we’ve followed to make what was a document retrieval language work for online banking will be gone too. We should consider this to be a wake-up call and an opportunity to redefine our profession.

I have little to quibble with in what the authors have presented here, other than point 8. My problem is with the rush to define rules, before we’ve fully understood the medium.

Andrew — Fri, 26 Jan 2007 14:51:28 GMT

Nice article and some good takes on the old heuristic chestnuts. It’s interesting to watch the Flash world via the bookshelves: there are fewer and fewer Flash books coming out these days, and a higher percentage are really good, such as Hall and Wans OOP in Flash, or some of the stuff around the Flash Communication Server.

Also worth looking at is the efforts Macromedia’s making around “Flash Central”, a centralized OS-like environment for hosting RIAs on the users machine which forces some consistency and UI standards for RIAs.