Benchmarking Rating Systems
One of the things that I always do when designing or re-designing a system is to perform research and competitive analysis on similar systems that are available. I've already talked at length at how Mapraider's rating system is slow, confusing and is hardly used by the members of the system, so clearly it is a prime candidate for this analytic step.
Not surprisingly, my first observation was that "5-point" rating systems are widely entrenched on the web for everything from books, movies and other product ratings to website ratings themselves. Almost exclusively, the 5-points system is displayed in terms of stars: five points/stars being the best, 3 average and a single star the worst.
It's not hard to understand why the 5-point rating system is so popular:
- It's simple to understand, even for young or less savvy visitors
- An item's rating and the maximum score can be gleamed with only a glance
- It's practically standardized even in the off-line world (e.g. 5-star hotel, 5-star restaurant)
- It provides opportunities for clean, efficient interface design that uses screen real estate effectively and encourages people to participate
And there are many well-known web sites that use the 5-star rating system:
- Amazon.com
- ConsumerReports.com
- Froogle.com
- IMDB.com
- Netflix.com
- PriceGrabber.com
- ResellerRatings.com
Froogle.com appears to even go so far as to force-fit ratings from other sites into its 5-point systems for consistency if they aren't already. By far though, Netflix.com's 5-star rating system is superior from all others I've seen. It's deceivingly simple, but a look under the hood reveals that it uses some pretty fancy CSS and AJAX-based communication with the server to enable you to RAPIDLY rate multiple movies on one page without a single page refresh.
Netflix.com's amazingly robust rating system
For comparison, Amazon.com and IMDB.com first require you to go to a new web page to rate each item, select a numeric rating from a drop-down and finally click the submit button. The end result is displayed as 5- and 10-point stars respectively.
Amazon.com's rating pulldown
The IMDB.com user rating pulldown
Also note how the numeric order for these sites is reversed, forcing the user to adapt for each site individually. Amazon.com lists rating values in descending order, while IMDB's are in ascending. It'd be intersting to see if this subtle difference had a subconcious effect on user ratings.
Competitive Analysis
When I started looking at gaming sites that host user-rated maps I was a bit surprised. What I didn't realize is that although these sites had confusing review schema's (which had their own, long page of explanation usually), they only provided the means for users to rate maps with a single score!
NaliCity's single-score user rating system
Insite's single-score user rating system
I probably shouldn't be all that surprised. Rating a map by using someone else's review schema can be difficult. A single score system provides a simple interface and can also lessen the number of round-trips to the server between page refreshes. All that said, what are the apparent strengths and weaknesses of the other various map rating sites?
Sunspire Codex
A unique and interesting idea, but body language with the thumb is offensive in some cultures, making this scoring scheme a bit touchy from the get-go. In addition, the "middle thumb" seems to have the potential for confusion. In the American culture, thumbs-up = good and thumbs-down = bad, but thumbs-sideways usually means something like "it's out back" or "pull my finger" :)
Insite
One of the more straight-forward review sites. A simple, 0-10 scale... as long as you don't read the review schema. Users can pick a single rating for each map. The only real issue here isn't the rating system itself, but rather the placement of the rating on the page — it's buried, and especially when there are user comments below the review and score.
Levels4You
The lesser-known Levels4You site employs a simple 1-5 scale. It's really ugly in my opinion, but it works and is fairly well displayed. What's interesting here is that because they didn't use a star-based system that shows the total maximum score, they felt the need to display the total possible score in each rating, i.e. 1/5 instead of 5 stars with only one highlighted. Not a major problem, but with the site's current design it is conceivably possible to misread the "1/5" as a "15" or "115" while quickly scanning for good maps.
NaliCity
I've watched NaliCity change over the course of many years and I have to say, it has one of the most extremely confusing review schemas I've seen on any site anywhere. It's based on 0-3 scores for three very oddly named categories: Awe, Build and Cast. This gives a maximum score of 9 by the reviewer, but the users can also influence the reviewer's score with a single +1, making the final score range a possible 0-10. Despite the confusing review schema, users are only provided the ability to give a single rating for the map. Personally, I don't think I've ever paid any attention to the 3 category scores, instead focusing on only the overall score and individual user scores.
UnrealPlayground
UnrealPlayground's system is based on custom categories, but each category and the final overall score is based on a 0-100 rating scale. Map ratings are also reserved for staff members, with no ability for users to rate the maps. In any case, the high precision of scores is probably a bit unnecessary. What's the difference between a map that is rated 80 for "Enjoyment" and one that is rated 81 or 85 in the same category?
Conclusion
So if we were to look at Mapraider's current rating system, we'd see a combination of custom weighted categories all based on a 0-10 scale. Compared to the other mapping sites I've discussed here, it is almost as complicated as NaliCity's rating system, with the major difference being that users are burdened with learning the review schema instead of just staff members writing reviews. Out of all the rating systems, it is also the slowest and most time-consuming system. Obviously Mapraider's rating system needs to be modified if ratings are to ever become a more active part of the system. I'm not going to be too specific about the changes at this point, but you can rest assured that it will:
- Adopt to some degree the rating system "standards" that have been well established and time tested by popular sites
- Be more simple and efficient, most likely requiring the user to only provide one score for each map
- Have a better user interface, allowing for fast and easy rating without page refreshes
Stay tuned.