Comments on On A Scale of 1 to 5

Alex Kirtland — Mon, 11 Aug 2008 19:18:18 GMT

Thanks for all these great comments, observations, questions, and recommendations for reading. We most certainly didn’t cover all aspects of rating and reputation systems, nor went as deep as we would have liked. There is the whole psychological aspect of rating that we didn’t address that’s very important, particularly that people are influenced by ratings they’ve just made (if I rate something a 5 I’m more likely to rate the next thing I see a 5), and how others rate that thing.

The story of eBay’s rating system doesn’t seem to have been documented very well anywhere, but it pretty fascinating, especially in trying to solve the practice of defensive rating.

Rating in a social environment is also something that deserves more attention. A good place to start might be here: http://www.slideshare.net/soldierant/designing-your-reput…

One thing that we avoided (on purpose) was how the information design of the rating system should look (5 stars, grades, etc etc etc). It’s a good discussion, but we wanted to focus only on the risk side of rating and reputation systems. Perhaps that’s another article ….

~alex

Will Evans — Wed, 06 Aug 2008 12:11:55 GMT

Very interesting, but I think your article, while the title didn’t imply it, focused mostly on the use of rating systems in mitigating risk for e-commerce sites, for instance, but not for social media sites. I can understand, and it seems rather easy – when anonymous, or even named, users, are rating on something relatively objective – a movie, or an artist, or a song or a book. What about user-generated content, when authors/users trade reputation and ratings for friendship, thus skewing effective use of ratings at all – if I publish an article, and all my friends rate it a 10 on a scale of 1-10, and every other article gets a 10, 10 has no meaning.
I notice a number of IAs out there designing rating systems for social media systems, social networks, and user generated media, and are either completely ignorant of the psycho-social aspects, or willfully ignoring the sociology and psychology of ‘members’ using social media tools to mediate conversations and create content.

Muhammad Faizan Ali — Wed, 06 Aug 2008 12:11:15 GMT

Great writing

Zephyr Zephyr — Wed, 06 Aug 2008 12:11:22 GMT

@David Shen: I wonder if too many 4s and 5s is really a problem… anything on this end of the scale is high-quality and worthy of users’ attention… they can further decide based on other indicators. Now, if the aggregated rating is the only way to determine appropriateness, that would be a problem. I don’t think it’s powerful & relevant enough as a stand-alone feature, but it helps. As mentioned in the article, ratings become more powerful when *combined* with written reviews (for example).

Zephyr Zephyr — Wed, 09 Jul 2008 18:17:30 GMT

Nice overview.

I’m curious about “defensive rating” and how eBay solved it… is rating still limited to registered, signed-in members, but they can now rate anonymously? Or does rating no longer require sign-in? Thanks.

Sean McBride — Wed, 06 Aug 2008 12:10:31 GMT

I disagree with your assessment of Amazon’s “Most helpful positive and negative reviews” feature. It’s not important that the two reviewers aren’t judging the product by the same metrics. In fact, I’d say it’s desirable. The questions that this answers are “What things did a trustworthy person like?” and “What things did a trustworthy person not like?” Then, the potential buyer can compare the priorities of those two reviewers to their own and decide which factors they think are most important. This mirrors real life. I ask multiple people their opinions about an important decision so that I can see how different people think about the same problem.

Colleen Jones — Wed, 06 Aug 2008 12:10:06 GMT

Thanks for a useful article. Risk management is an interesting lens through which to examine rating systems. I think it largely works. In addition, I think persuasion is a helpful lens because rating systems are trying to persuade users that the risk is worth taking or there is little risk. In persuasion, the credibility of the speaker / author is a significant factor in his or her ability to influence. The question is what determines credibility in the eyes of the user / customer? Looking at that from the perspective of different persuasion theories would be interesting. Also, ratings systems have much persuasive potential because they include a mix of rational and emotional appeals. The rating itself, especially an aggregate rating, comes across as objective and appeals to our rational side. The comments are more subjective and appeal more to our emotional side, as well as help us determine the speaker’s credibility.

John Donohoe — Wed, 06 Aug 2008 12:08:52 GMT

Thank you for a great article. Rating systems have definitely helped put some of the power into the hands of the people but like you said, they can be flawed and implemented poorly. I appreciate your effort to help offer some thoughts about best practices.

A couple of things occurred to me while I read this article:

“Real Names increases likelihood of Responsible Content”
I was struck by your comment about Amazon’s use of “Real Names” to promote social responsibility in the rating system. This assumes that the old saw that anonymity on the web fosters poor behavior. I believed the same thing until I saw a few studies that have shown that ‘real names’ does not do much to improve online social graces. I guess the take away is if others perceive that a rating by a user with a ‘real name’ is more reliable, so much the better but in terms of data collection, it may not make much difference.

One research report – http://www.guardian.co.uk/technology/2007/jul/12/guardian…

“Finding an Appropriate Rating Instrument”
Your point about appropriate rating systems is important. Everyone has “5 stars” on the brain – from iTunes to Amazon to Netflix and beyond. It’s great for a simple interface design but often it’s woefully inappropriate for the context it’s being used for. Some things are just too complicated to reduce to a single representative number.

I recently had a similar conversation concerning Likert scales on political surveys. Many of the hot-topic issues are so nuanced that a 5 point scale doesn’t really allow me to express my thoughts accurately ( Do I “somewhat support” abortion? What does that even mean?).

Creating an appropriate measuring instrument is just as important as making the decision to ask for rating feedback in the first place.

Christina Wodtke — Wed, 06 Aug 2008 12:08:57 GMT

Great article, I also recommend Chris Allen’s awesome series on rating systems http://www.lifewithalacrity.com/2005/12/collective_choi.h…
In particular, he looks at the problem with bianary distribution or ratings (too many 1’s and 5’s) and how to use weighting to help create a more accurate and fine-toothed rating (poor B&A could use some of this thinking rolled into our system).

Also related is the brilliant Bernardo Huberman’s research, in particular his work with voting sites
http://www.hpl.hp.com/research/idl/papers/opinion_express…
His Parc Forum talk is very enlightening as well
http://www.parc.com/cms/get_article.php?id=703

Tim Sheiner — Wed, 06 Aug 2008 12:09:03 GMT

Really interesting article. Thanks for giving me the concept of understanding the user value of ratings as a risk management tool! That is a simple and powerful insight.

However, to follow on to David Shen’s comment, I wish you’d included a bit more discussion on how the risk management perspective can instruct the design of the ratings display. Some sites are using histograms to provide a bit more richness for interpreting the rating. But to David’s point, even if I can see the distribution, how do I know if I am more like the group who gave 4 stars, or the group who gave 5? And how do I relate the difference between 4 and 5 to my purchase risk?

David Shen — Wed, 06 Aug 2008 12:09:12 GMT

Great article! But can you or someone talk about what to do in the case where a ratings system generates too many 4s or 5s, resulting in a problem for users to determine the minute difference between the two? Thanks!

Chris Butler — Wed, 06 Aug 2008 12:09:18 GMT

Alex and Aaron,

Thanks for a thorough look at this subject. I agree that the system used for rating should be created with the particular product or service in mind, as there are variables appropriate to one that will not be appropriate to others. This is one reason why the effort of the reviewer, especially whether he or she has taken the time to write clear and comprehensive comments in addition to any numeric rating system that is available, is so important to the person viewing the rating later. As you noted, one customer may give a product a low rating due to a particular expectation he or she had for it, while another may give the same product a high rating based upon a completely different expectation. Having access to comments or an explanation helps the prospective customer discern the importance of certain reviews to his or her decision.

One “genre” of rating systems for which this idea is applicable is the dining-related ratings site (like yelp.com or foodbuzz.com). I’ve often noted that I might read a disparaging review that gives low marks for having a limited menu and think, “that sounds like my kind of place- why would someone consider a limited menu a bad thing?” Interestingly, many users only need one not-so-good dining experience to prompt them to submit a negative review, whereas positive reviews might come from long-time loyal customers. Given that any restaurant’s service can be affected by many unpredictable factors, a measured reviewer should take into consideration whether it is their first visit, whether their order was a specialty of the restaurant, the experience of the waiter, etc. Thankfully, most websites dedicated to rating particular services would have considered specific attributes (i.e. wait time, menu scope, wait staff, etc. for restaurants, as apposed to format, page count, dimensions, etc. for books) in order to make sure that the overall presentation of any given business/product is, in general, accurate. As you mentioned, this can become a problem on sites like ebay.com, where the user is in full control over the portrayal of what is being sold. Perhaps Amazon.com has found a good fusion in this regard in the way it allows users to sell used versions of books already in their inventory?

Chris