Simplifying the Rating System
The current rating system, although not overly complicated, seems to confuse many people on the site.
- People tend to take moderations personally
- Members think that administrators are the ones making the moderations
- Some individuals rate maps in a way to gain karma instead of rating the map appropriately
Add to that the fact that the current category structure doesn't really work for all maps (specifically the A.I. category which is used to rate how well the computer plays against you) and that there are a few bugs in the rating system and I can understand why someone might be confused.
So the challenge for Mapraider 2.0 was to come up with revisions to the rating system that would alleviate these issues while keeping the back-end bug free. I even asked the members for their opinions to help me make some decisions that I've been mulling over in my head for a long time now.
The usability guru in me wanted to make the rating system significantly more simple by having only a single 0-10 rating score for each map, completely removing categories. I still think this is the best long-term approach, but the opinion of the audience has clearly stated that it's not acceptable.
I'd be lying if I said I wasn't disappointed. There are a number of huge advantages to simplifying it to a single rating, including everything from an easier user-interface and less database usage to less dynamic score calculation, which also has the added benefits of less bugs and a faster system.
The most-missed advantages of a single-rating system in my mind will be: A single rating system would also leave the methodology of arriving at a score completely up to the user instead of pigeon-holing them into a group of categories. And one rating per map is also flexible enough for any gaming system currently available or not yet released.
I should mention that it has been suggested to implement a system where users could have their own categories or be allowed to adjust the weighting of categories on an individual basis. An interesting idea to be sure, but the database design and system requirements for having individual rating schemes for THOUSANDS of users seems more than a little extreme and against my quest in simplifying things for me and the users.
I've also tossed around the idea of having a default simple rating scheme with an optional category-based variation but again, this overly complicates things (not to mention would mean building two rating systems instead of one).
So the compromise has become to simply merge the A.I. with the game play category. After all, A.I. is all about how well the computer PLAYS the GAME against you and probably shouldn't have been broken out separately to begin with. This change effectively allows the site to take an average of the two category scores (since game play and construction are equally as important in map design in my opinion) rather than having a weighted-system that involves a series of percentage calculations for each category.
With the category dilemma solved, it was time to move onto addressing the moderation system. Like it or hate it, the current moderation system is extremely effective at keeping the map scores at a very accurate representation of each map. I've seen numerous examples of clan maps where clan members have rated extremely poor maps (i.e. a single, large, one-room, no-texture) a perfect 10 score but that score was never applied to the map because of the moderation system. Conversely, I've seen members give an overly poor score to an otherwise average map but have it moderated out.
There are some exceptions where the moderation system hasn't worked, but these are only because of a design flaw in the system. Specifically, ratings that sit unmoderated for a long period of time are automatically counted as fair and applied to the map score. The reason for this feature was meant to eliminate a bunch of pending ratings in the system since not all maps receive the same amount of traffic. In hindsight, this was a poor decision as the maps that receive less traffic tend to be the ones where users are more apt to rate them inappropriately.
A number of individuals have commented that they feel the moderation system forces people to vote "in line with the current score," and to a limited degree I do see that as a potential reality. However, the same could be argued for any rating system where the current score is visible to the user making the rating. In the case of Mapraider though, rating the map similarly to the current score is a good way to gain karma since you are almost guaranteed to earn some for a fair rating.
As I previously mentioned, many members also take the moderations very personally, getting outwardly upset about getting an unfair moderation on one of their scores. I won't comment about how I feel about this, but I do realize it's something that needs to be addressed — I don't want anyone to feel put out for participating in the community :)
Figuring out how to eliminate these two behaviors is a simple matter of completely removing karma and never showing how a rating was moderated or the current tally for fair vs unfair ratings. The user is left to assume that his rating is always included in the current score.
You might be a bit surprised to learn that karma is going away. Don't be. Mapraider 1.0 was a good test of a totally new way of cataloging and rating maps while avoiding some of the pitfalls of other community-driven sites. There were bound to be some things that proved unnecessary or imperfect.
You may not realize it, but Karma has a large impact on a variety of functions throughout the site (or at least, it did). There are a number of reasons it is being removed, but I'll talk more about them in my next post.