Mapraider 2.0

Virgo47 popped into the Mapraider IRC chat the other night and proposed an interesting solution to Mapraider's somewhat bumpy moderation system. As I've mentioned before, there are instances where ratings are applied to a map's score when it doesn't get moderated enough times or at all. This is problematic because In the current system, once a rating is moderated fair or unfair, it's permanent (unless an administrator puts it back into the moderation queue).

His solution was quite elegant and has been dubbed the "OMS" — Open Moderation System. The principle is this:

Ratings are never permanently moderated. Instead, their moderation can change over time, fluctuating as more people rate each map and moderate the ratings.

In other words, the moderation of each ratings will change as more people contribute their thoughts, rather than being fixed once a minimum number of moderations have been reached. The map scores will also be more dynamic and better representations of a group's ratings.

I really like this approach as it gets rid of the issue of weird ratings indefinitely applied to a map score and also allows ratings to be instantly applied to the maps rather than waiting for a specific number of moderations. There are, however a few sticking points to be flushed out:

• The number of moderations in the database will increase exponentially with ratings, using up valuable database storage.
  
  To get around this, my initial thought was to stop moderations for a rating once the difference between fair/unfair moderation count reaches a specific value. This value would either be a ratio or a percentage of the total number of ratings for the map. For example, 2:1 or 40%, which would make the gap larger with a higher number of ratings. I'd also probably specify a minimum number regardless.
  
  I'm not sure if this makes the new system too much like the old, but in any case, I really feel the need to have some way to avoid using up database storage for moderations.
• An issue arises when there is a tie between moderations — fair or unfair?
  
  In this case, it seems to make sense that a tie is considered fair.
• Obviously unfair ratings will always start out as fair
  
  I'm not sure if this is really an issue. On one hand, this means that perfect 10 scores on crappy maps may happen more often. But on the other, because they no longer "time out" and become permanently fair, they can be moderated out more easily and the trade off may make sense.

I'm excited about this change. It feels like a natural evolution to the system that may actually help to increase rating/moderation participation in the community. The real test will be when I have something actually working.

Categories:

... or at least, that's what the plan was.

It turns out that karma is a cool idea, but prone to a few "social flaws." In community sites, there is always going to be an element of undesirable behavior. I've seen it in every web site and online service since the dial-up BBS.

In the case of Mapraider, it was foreseen that undesirable behavior would include two primary issues that exist on nearly every other map catalog web site:

1. Poor ratings, whether through manipulation or just crappy rating
2. Poor etiquette in the forums

At the time, there weren't many well-known self-moderating systems on the web. The best example at the time was probably Slashdot.org which used a system of peer moderation based on "karma points" that members could earn with good behavior.

While not perfect, Slashdot's system for comment posting works well. Members read articles and post comments about them. Members gain karma points by posting messages that other members think are a good contribution to the discussion, and only those members with good karma actually get to decide which posts are good contributions to begin with.

A simple idea, and it seemed like it could be adapted as a method of promoting good map ratings — gaining karma points when your map ratings are moderated fair by your peers. Lose enough karma and you temporarily lose the ability to rate maps or post in the forums until your karma regenerates after a certain time period.

At first, it sounded like a perfect solution to the problem of people rating poor maps perfect 10 because their friend made it and good maps a perfect 0 because they disliked the map author. However, It became evident that taking away privileges like rating and posting was a bad idea, and eventually even the need of having a karma point system was deemed unnecessary.

The straw that broke the camel's back in my opinion was a well-known Slashdot symptom: "Karma Whoring", or the act of performing an action with the sole intent of gaining karma. On Slashdot, karma whoring takes the form of re-posting an article's text in case the original site goes down, or taking someone else's buried but insightful post and re-posting it.

On Mapraider however, karma whoring has a very detrimental effect in that these members rate maps based on the current score of the map rather than what they really think is an appropriate score — almost guaranteeing them a fair rating and thus karma points.

Watching the rise of an enormous number of community-driven web sites since the launch of Mapraider, I can say that I have yet to see a single, flawless, self-moderated community model that works — all of them appear to be prone to at least a few social flaws. Mapraider suffers from a few of the same, but my hope is that a few subtle changes in the system such as removing karma will hopefully make these issues much less prevalent or even non-existent in version 2.0 of the site.

Edit: I realize it must seem like the blog thus far has been a postmortem of Mapraider 1.0 rather than details on change in version 2. The reason for this is to provide some insight to why things are changing before I hit you over the head with all the new goodness. Please bear with me, it'll be worth it. :) If you have any comments or feedback to add in the meantime, feel free to click the comment link and add your thoughts.

Categories:

Apparently, the jury isn't quite out on the single-rating per map system planned for Mapraider 2.0. I've been doing some in-depth analysis on map searches performed on the site over the past year (prompted by a recent user review that gave an absurdly high construction rating to a poorly made map) and am quickly discovering that the standard usability anomaly between what users say they do and what they actually do is alive and well on the site :)

Take March 2006 for example. In that month there were 323,688 map searches on Mapraider. Out of that, only 20,298 were advanced search queries. Of the advanced queries, only 355 of them actually used the category score filters — or 0.10967% of all searches and 1.75% of advanced searches.

As a comparison, November of 2005 had 270,855 searches, of which 14,065 where advanced searches and only 122 of those were filtered by category — or 0.045% of all searches and 0.867% of advanced searches.

Essentially, what this illustrates is that although members have said they value the individual category scores, only an extremely small (and practically non-existent) portion of searches are actually performed with category score filtering.

Almost 94% of searches performed on Mapraider in March and 95% in November were done using the Quick Search, which can filter against the overall map score.

Unfortunately, I have no easy way to determine if those searches are being performed by members or by anonymous guests, but based on my observation of online users in the admin tool, I'd wager that at least half of those category filtered advanced searches were performed by non-registered visitors. I also haven't performed formal usability testing to determine if the advanced search is simply not used as often because it's not heavily promoted on the page, but with 20,298 advanced searches in March, it's evident that a significant number of users are finding it. Based on my examinations of what users are searching for though (typically searching by game system, game type and overall score), I'd say that the advanced search is just simply "too much search" for most users.

Now combine this information with the fact that out of over 3646 current members, only 200 have rated more than 10 maps and only 43 of those 200 have rated at least 100 maps. Clearly, many members are never rating maps, probably because the rating system suffers from over complication and sluggishness.

The benefits to keeping the multiple category scoring system are quickly being surpassed by the disadvantages.

For now, I'm making the decision to move forward on a single-rating based system. Once the site is at a point where I'm ready for testing and feedback, we'll see what happens.

Categories:

The current rating system, although not overly complicated, seems to confuse many people on the site.

• People tend to take moderations personally
• Members think that administrators are the ones making the moderations
• Some individuals rate maps in a way to gain karma instead of rating the map appropriately

Add to that the fact that the current category structure doesn't really work for all maps (specifically the A.I. category which is used to rate how well the computer plays against you) and that there are a few bugs in the rating system and I can understand why someone might be confused.

So the challenge for Mapraider 2.0 was to come up with revisions to the rating system that would alleviate these issues while keeping the back-end bug free. I even asked the members for their opinions to help me make some decisions that I've been mulling over in my head for a long time now.

The usability guru in me wanted to make the rating system significantly more simple by having only a single 0-10 rating score for each map, completely removing categories. I still think this is the best long-term approach, but the opinion of the audience has clearly stated that it's not acceptable.

I'd be lying if I said I wasn't disappointed. There are a number of huge advantages to simplifying it to a single rating, including everything from an easier user-interface and less database usage to less dynamic score calculation, which also has the added benefits of less bugs and a faster system.

The most-missed advantages of a single-rating system in my mind will be: A single rating system would also leave the methodology of arriving at a score completely up to the user instead of pigeon-holing them into a group of categories. And one rating per map is also flexible enough for any gaming system currently available or not yet released.

I should mention that it has been suggested to implement a system where users could have their own categories or be allowed to adjust the weighting of categories on an individual basis. An interesting idea to be sure, but the database design and system requirements for having individual rating schemes for THOUSANDS of users seems more than a little extreme and against my quest in simplifying things for me and the users.

I've also tossed around the idea of having a default simple rating scheme with an optional category-based variation but again, this overly complicates things (not to mention would mean building two rating systems instead of one).

So the compromise has become to simply merge the A.I. with the game play category. After all, A.I. is all about how well the computer PLAYS the GAME against you and probably shouldn't have been broken out separately to begin with. This change effectively allows the site to take an average of the two category scores (since game play and construction are equally as important in map design in my opinion) rather than having a weighted-system that involves a series of percentage calculations for each category.

With the category dilemma solved, it was time to move onto addressing the moderation system. Like it or hate it, the current moderation system is extremely effective at keeping the map scores at a very accurate representation of each map. I've seen numerous examples of clan maps where clan members have rated extremely poor maps (i.e. a single, large, one-room, no-texture) a perfect 10 score but that score was never applied to the map because of the moderation system. Conversely, I've seen members give an overly poor score to an otherwise average map but have it moderated out.

There are some exceptions where the moderation system hasn't worked, but these are only because of a design flaw in the system. Specifically, ratings that sit unmoderated for a long period of time are automatically counted as fair and applied to the map score. The reason for this feature was meant to eliminate a bunch of pending ratings in the system since not all maps receive the same amount of traffic. In hindsight, this was a poor decision as the maps that receive less traffic tend to be the ones where users are more apt to rate them inappropriately.

A number of individuals have commented that they feel the moderation system forces people to vote "in line with the current score," and to a limited degree I do see that as a potential reality. However, the same could be argued for any rating system where the current score is visible to the user making the rating. In the case of Mapraider though, rating the map similarly to the current score is a good way to gain karma since you are almost guaranteed to earn some for a fair rating.

As I previously mentioned, many members also take the moderations very personally, getting outwardly upset about getting an unfair moderation on one of their scores. I won't comment about how I feel about this, but I do realize it's something that needs to be addressed — I don't want anyone to feel put out for participating in the community :)

Figuring out how to eliminate these two behaviors is a simple matter of completely removing karma and never showing how a rating was moderated or the current tally for fair vs unfair ratings. The user is left to assume that his rating is always included in the current score.

You might be a bit surprised to learn that karma is going away. Don't be. Mapraider 1.0 was a good test of a totally new way of cataloging and rating maps while avoiding some of the pitfalls of other community-driven sites. There were bound to be some things that proved unnecessary or imperfect.

You may not realize it, but Karma has a large impact on a variety of functions throughout the site (or at least, it did). There are a number of reasons it is being removed, but I'll talk more about them in my next post.

Categories:

I needed a bit of a break from the technical details of the site and played around with some design ideas for the site today. Aside from the back-end and UI enhancements, there's going to be a dramatic shift in how the site "feels," and if any part of the site is in dire need of updating, it's the visual design.

Some individuals who are aware of the work-in-progress of Mapraider 2.0 have stressed concern over changing the site too much. Rest assured I'm keeping that in mind and am not taking any of these changes lightly — it wasn't too long ago when eBay redesigned their site and the backlash from users forced them to revert :)

My primary goals with the visual re-design are:

• Update the look and feel so that newer technologies that enhance the experience, such as AJAX (think Google Maps), can be applied easily
• Clean up and optimize the page markup so download times are significantly reduced using primarily CSS for layout
• Increase usability by adjusting page content (or even removing in some cases) so that pages are more clear and concise
• Get the damn Mapraider logo on the site (why it got left out originally, I can't remember)

Expect a site design that harkens back to the original design but is easier to read, better organized and strengthens the focus on the maps even more — after all, content is king and my thoughts of web UI design as of late has been to "keep it simple and out of the way."

The architecture of the site (organization of information) on the other hand isn't going to be changing much. Maps are still organized under a "Maps" item and the forums are also still in the same place so getting around will feel very familiar. But it wasn't always going to be that similar; I've gone through many iterations of how the site could be restructured prior to arriving at the final architecture. But I'll talk more about the site's organization once I've got the site map more complete.

Categories:

There's a strange satisfaction that occurs when you re-visit an old application's design through the process of re-factoring code or re-working a database design. In the case of the transition of Mapraider 1.0 to version 2, there were many opportunities for improvement in the database design. Particularly in cases where quick-fixes had been applied to the original structure during the past few years.

This Entity Relationship Diagram (basically the structure of the site's database) may look like a jumbled mess, but in reality it's elegant and far superior to the existing database:

One of the biggest challenges in the database for Mapraider 2.0 is how to add support for Map Packs — a collection of 2 or more individual maps — in a way that was scaleable, fit into the UI seamlessly and made sense without duplicating a lot of information. Analyzing the logic/rules behind map packs seems simple enough, but when you start to factor in particular functional requirements, it can get hairy very quickly.

Map packs are treated as individual maps on the current version of Mapraider as single maps. It works, but it's a somewhat of a duct-tape fix since only one author can actually be assigned to the map and the others mentioned in the description. If all authors are members of the site, this makes it impossible for them to each get reporting on the map, or be added to a user's favorite authors list.

It is also not uncommon for pack authors to release their individual maps separately once the pack has been out for a period of time. In this instance, we end up with each map in the database twice, but completely unrelated on the current version of the site: once mentioned on the map pack page and once on the map's own detail page.

One of the requirements for 2.0 is to have a link between packs and individual releases if one or the other exists. The debate then becomes whether to store each map in the pack as stand-alone map entries (leaving download links empty until it is released individually) that are combined into a "pack" on the fly, or store the pack as a summary that links to individual releases map detail pages only if they exist.

There are a number of similar challenges when dealing with map packs, but I believe I've come up with an elegant solution (the details of which I won't reveal just yet :). In any case, the database design for Mapraider 2.0 is quickly arriving at a point where it will solve the map pack conundrum and facilitate many other new features and fixes.

Categories: