Boxes and Arrows: Comments on stories by Alex Kirtland

Marcel Britsch — Thu, 25 Jun 2009 14:08:16 GMT

great article. thanks for looking at this indetail!
it’s not that closely related, but in this articel about voting system (focusing on political elections) the authors also present some interesting findings about rating methods:
http://www.newscientist.com/article/mg19826511.600-why-fi…
cheers m

Nick Trendov — Sun, 24 May 2009 13:55:16 GMT

It’s the Content not the Reputation

Currently valuing reputation depends entirely on the perspective of the judges who value the contribution whether it is one person or thousands and the problem here is that a single judges perspective will be different than every reader and more judges will hold perspective that may appear close to but will not fit any reader, past current or future.

Going back to first principals for the value behind reputation systems may help. Reputation confirms content value.

While reputation systems appear to help focus on content from a trusted source, one with a good reputation, it is important to note the command and control nature of the system. The judge rules, context is fixed.

Take any story with a low reputation value and you will find any number of readers who rank it very high. Why?
It fits their context and objectives, which were apparently unimportant to the judge at time of rank pronouncement.

Are reputation systems helpful? Maybe.

On the other hand it appears to me that they are another filter and should either work with common filters such as search engines or context engines rather thant being held to high esteem.

Would you place more importance on a reputation system than a google ranking?

Cheers,
Nick
www.neuropersona.com

Nick Trendov — Sat, 23 May 2009 15:10:22 GMT

I concur with this post…” It’s not important that the two reviewers aren’t judging the product by the same metrics. In fact, I’d say it’s desirable. “

The issue with the ratings is the perspective and in my experience it can be accomodated with a little effort.
Consider a datacube used in analytics where the perspective can be changed at will.
The way I approach it is by using stories and Metics with 5 dimensions—Story, Process, Software, Brand and Metrics.
Either google Story Lens and Neuropersona or visit my site if measuring from multiple perspectives would help.

Cheers,
Nick
www.neuropersona.com

Higher Ground — Thu, 30 Apr 2009 15:12:11 GMT

I really useful article. I think you have discussed a very important issue and it’s something E-Commerce sites need to look into more.

Aaron Hemmelgarn — Fri, 23 Jan 2009 09:58:44 GMT

I’ve found in my dealing with some of my clients that the User Ratings are only as good as the people who write them. While some or be that, most are good, it really only takes a few bad Apples to spoil the whole bunch and that whole bunch being new and prospecting clients who may be looking for what you’re offering.

So it really depends on what you’re using the rating for, what’s it’s purpose, what’s the end goal you’re looking to achieve by using a rating system?

All in all, this was a great Article and I would recommend it to anyone interested in understanding User Generated rating systems.

Check out some of our work at www.fiswebdesign.com a Dallas Web Design company..

Chas Linn — Thu, 22 Jan 2009 00:06:07 GMT

Alex, I found this article a great starting point in thinking about the mechanics of rating systems which I regularly use (benefit from) but generally don’t engage in. Just recently I have been asked to think about the design of a reputation engine. Now I know that your article is mostly aimed at buyers and sellers in a transaction but the point about risk does have some extension into social media too. However I would be interested in Christina’s view about the recommendation number 4 – ‘Avoid negative reputations’.

Do you mean by this that we should avoid reputations going negative as in ‘-3’, or we should avoid rating and reputation systems which allow negative comment full stop.

I notice that this site allows negative reputations (and seeing as I am currently on 0 at the point of writing this there is a danger I might go there too). But also it allows the attributed rating of 1/5 or ‘Yuck!’ which is negative in all but numerical value. This site then encourages negative comment and negative reputation where it is warranted. This obviously runs counter to your advice. What do you think about this?

Richard Johnson — Tue, 13 Jan 2009 17:48:31 GMT

In my experience I have found that Amazon’s approach, which must cover a wide range of scenarios for products does a good job of giving an the overall picture of a product. Reading great and poor reviews side-by-side helps me to decide whether a product may be what I am looking for or has problems I did not know about, or lack features that were in a different product. The things that do throw off ratings are when users rant about things that are not related to the product, but the service or even the delivery guy.

Alex Kirtland — Tue, 13 Jan 2009 17:48:43 GMT

Thanks for all these great comments, observations, questions, and recommendations for reading. We most certainly didn’t cover all aspects of rating and reputation systems, nor went as deep as we would have liked. There is the whole psychological aspect of rating that we didn’t address that’s very important, particularly that people are influenced by ratings they’ve just made (if I rate something a 5 I’m more likely to rate the next thing I see a 5), and how others rate that thing.

The story of eBay’s rating system doesn’t seem to have been documented very well anywhere, but it pretty fascinating, especially in trying to solve the practice of defensive rating.

Rating in a social environment is also something that deserves more attention. A good place to start might be here: http://www.slideshare.net/soldierant/designing-your-reput…

One thing that we avoided (on purpose) was how the information design of the rating system should look (5 stars, grades, etc etc etc). It’s a good discussion, but we wanted to focus only on the risk side of rating and reputation systems. Perhaps that’s another article ….

~alex

Will Evans — Wed, 06 Aug 2008 12:11:55 GMT

Very interesting, but I think your article, while the title didn’t imply it, focused mostly on the use of rating systems in mitigating risk for e-commerce sites, for instance, but not for social media sites. I can understand, and it seems rather easy – when anonymous, or even named, users, are rating on something relatively objective – a movie, or an artist, or a song or a book. What about user-generated content, when authors/users trade reputation and ratings for friendship, thus skewing effective use of ratings at all – if I publish an article, and all my friends rate it a 10 on a scale of 1-10, and every other article gets a 10, 10 has no meaning.
I notice a number of IAs out there designing rating systems for social media systems, social networks, and user generated media, and are either completely ignorant of the psycho-social aspects, or willfully ignoring the sociology and psychology of ‘members’ using social media tools to mediate conversations and create content.

Muhammad Faizan Ali — Tue, 13 Jan 2009 17:39:49 GMT

Great writing

Zephyr Zephyr — Wed, 06 Aug 2008 12:11:22 GMT

@David Shen: I wonder if too many 4s and 5s is really a problem… anything on this end of the scale is high-quality and worthy of users’ attention… they can further decide based on other indicators. Now, if the aggregated rating is the only way to determine appropriateness, that would be a problem. I don’t think it’s powerful & relevant enough as a stand-alone feature, but it helps. As mentioned in the article, ratings become more powerful when *combined* with written reviews (for example).

Zephyr Zephyr — Sun, 24 May 2009 13:58:26 GMT

Nice overview.

I’m curious about “defensive rating” and how eBay solved it… is rating still limited to registered, signed-in members, but they can now rate anonymously? Or does rating no longer require sign-in? Thanks.

Sean McBride — Sun, 24 May 2009 13:58:46 GMT

I disagree with your assessment of Amazon’s “Most helpful positive and negative reviews” feature. It’s not important that the two reviewers aren’t judging the product by the same metrics. In fact, I’d say it’s desirable. The questions that this answers are “What things did a trustworthy person like?” and “What things did a trustworthy person not like?” Then, the potential buyer can compare the priorities of those two reviewers to their own and decide which factors they think are most important. This mirrors real life. I ask multiple people their opinions about an important decision so that I can see how different people think about the same problem.

Colleen Jones — Wed, 06 Aug 2008 12:10:06 GMT

Thanks for a useful article. Risk management is an interesting lens through which to examine rating systems. I think it largely works. In addition, I think persuasion is a helpful lens because rating systems are trying to persuade users that the risk is worth taking or there is little risk. In persuasion, the credibility of the speaker / author is a significant factor in his or her ability to influence. The question is what determines credibility in the eyes of the user / customer? Looking at that from the perspective of different persuasion theories would be interesting. Also, ratings systems have much persuasive potential because they include a mix of rational and emotional appeals. The rating itself, especially an aggregate rating, comes across as objective and appeals to our rational side. The comments are more subjective and appeal more to our emotional side, as well as help us determine the speaker’s credibility.

John Donohoe — Wed, 06 Aug 2008 12:08:52 GMT

Thank you for a great article. Rating systems have definitely helped put some of the power into the hands of the people but like you said, they can be flawed and implemented poorly. I appreciate your effort to help offer some thoughts about best practices.

A couple of things occurred to me while I read this article:

“Real Names increases likelihood of Responsible Content”
I was struck by your comment about Amazon’s use of “Real Names” to promote social responsibility in the rating system. This assumes that the old saw that anonymity on the web fosters poor behavior. I believed the same thing until I saw a few studies that have shown that ‘real names’ does not do much to improve online social graces. I guess the take away is if others perceive that a rating by a user with a ‘real name’ is more reliable, so much the better but in terms of data collection, it may not make much difference.

One research report – http://www.guardian.co.uk/technology/2007/jul/12/guardian…

“Finding an Appropriate Rating Instrument”
Your point about appropriate rating systems is important. Everyone has “5 stars” on the brain – from iTunes to Amazon to Netflix and beyond. It’s great for a simple interface design but often it’s woefully inappropriate for the context it’s being used for. Some things are just too complicated to reduce to a single representative number.

I recently had a similar conversation concerning Likert scales on political surveys. Many of the hot-topic issues are so nuanced that a 5 point scale doesn’t really allow me to express my thoughts accurately ( Do I “somewhat support” abortion? What does that even mean?).

Creating an appropriate measuring instrument is just as important as making the decision to ask for rating feedback in the first place.