Why the rating system isn't working

Post your comments, suggestions, questions, etc. about IndyDDR.com here. Also use this forum to report issues or get help with the site.

Moderator: Moderators

User avatar
Gatekeeper
Standard
Standard
Posts: 50
Joined: Mon Feb 07, 2005 10:16 am
Location: Animated avatars are the devil
Contact:

Why the rating system isn't working

Post by Gatekeeper » Tue Mar 15, 2005 10:07 am

So IndyDDR is back, and fresh, and a lot of things are better than they were before. Some things aren’t. Sure there are still a lot of petty people out there, but there have always been petty people in the world and there always will be. I’m optimistic about the future of IndyDDR and am confidant it will remain a fun community to associate myself with.

One of the new features we’ve seen on the forums here is the new rating system. If I were to critique Brian and Brandon, which I am hesitant to do, I would have to say they may have made a small mistake when implementing the rating system. Don’t get me wrong guys, I am very grateful for all the hard work you have both put into making this forum worth participating in. And I don’t feel that a ‘user moderated’ forum is a bad idea.

Your mistake, if I may be so bold, was in the lack of guidance. You gave us a great rating system, but I think an instruction booklet was in order. You told us how to use it, but not why to use it. People are downvoting posts they disagree with rather than posts that are poorly written.

If it were my place to tell people how to use the voting system (and it certainly isn’t) I would tell them to vote based on how clearly the poster communicated their message, and whether it contributes anything to the discussion. In other words, stop voting based on the content, and ask yourself one question: Does this post enlighten and inform me?

Obviously flames don’t enlighten and inform, nor do personal attacks. But neither do poorly written messages. Poorly written can mean just lazy typing, filled with poor grammar and spelling mistakes, or it can mean unsubstantiated claims and biased remarks. Someone jumping up and down screaming that their arcade is the best without giving any reasons why probably deserves to be downvoted. That same person can enumerate the reasons their arcade is superior, and no matter how much I might disagree, as long as they are making sense I will upvote them.

But I’m doubtful if this will ever happen on IndyDDR. It has been demonstrated time and time again that enough people lack the maturity to separate themselves from their emotional attachment to the subject, so how can we expect them to objectively rate posts with which they disagree? These immature people will continue to cause trouble for the rest of us.

Don’t get me wrong, I like the voting system. I wish it could work. But given the atmosphere we are dealing with, I am very sceptical of this, and I don’t think it will last. Even some of the people on here who I consider to be nice, good people resort to flames, and I will downvote those posts. And they will come to me and say, Ian, I thought you were my friend, why did you downvote me?

I don’t downvote people, I downvote posts. Whether I like you or not, agree with you or not, I’ll vote your posts based on how clearly they are expressed and on whether they provide accurate, useful information to the community. Even the worst trolls occasionally write posts that deserve Greats, even if you don’t like them. Even your best friend can write a stupid post while he’s drunk, and he deserves to have that post downvoted.
Image

User avatar
Ho
Site Admin
Site Admin
Posts: 5645
Joined: Fri Dec 31, 2004 10:26 am
Location: The Ho-House

Post by Ho » Tue Mar 15, 2005 10:51 am

Good points. However, I actually specifically provided little guideance on purpose. I also made mention that the ratings affect ranks, but I was not specific as to how. This was also intentional.

The idea behind the rating system is that it gives the community a voice to express its opinion on the posts within the forums. In that sense, you are dead on. It is a rating of individual posts--not of threads and not of individual users. The rated posts are then taken collectively to determine ratings/rankings of threads and users.

But what is key here is that it is a community voice. The community is made up of many individuals. These individuals will all have varying ideas on what makes for a good post and a bad post. You have expressed your views on the subject and I find them to be quite valid. However, I can also understand and have seen demonstrated other criteria used to rate posts. I have found many of these to be valid as well--even if it wasn't something that I would have conceived of myself.

I've been watching how the system has been used. In many cases, it is working just as I had intended. In some cases, it's has worked out quite different: sometimes good, sometimes not. But in all cases, it represents the collective views of the community it serves. In that regard, I consider it an uncomprimised success!

We may not all agree with every rating, but it still represents the opinion of at least one member of this community. And that's what the system is all about. It helps us all quantify what kinds of things are liked and disliked here and can guide us and help us to make this a community we can all enjoy more.

From an administrative perspective, Brandon and I have put a lot of thought and effort into balancing the system so that ratings and rankings respond to consistent trends and not to outlying votes or isolated attacks of someone with a grudge. As a result, you'll notice that there have been very few rank changes as a result of the ratings.

You call out immaturity as a cause of downvoting based on personal disagreement. I am happy to report that there have been far more positive votes cast than negative ones. Only a few people have had negative ratings drag them down into the Danger rank, and those people either have an antagonistic history here anyway or were involved in discussion or flaming that raised the ire of multiple people. And in most cases, these people were automatically elevated in a reasonable amount of time due to their positive participation being recognized in other areas. On the other side of the spectrum, no one has yet attained a Heavy rank...although a few people are pretty close.

I thank you for your opinion on these matters and for opening this discussion. However, some of what you see as weaknesses are actually the system working just as it was intended.
Last edited by Ho on Tue Mar 15, 2005 11:01 am, edited 1 time in total.
Image

User avatar
God Of Rock
Standard
Standard
Posts: 875
Joined: Wed Feb 02, 2005 1:40 pm
Contact:

Post by God Of Rock » Tue Mar 15, 2005 11:00 am

All very valid points, and I admit I was a bit malicious with
this thing at the start. I voted on posts based on what was
being said, rather than how it was presented. I don't go through
the post to look for perfect grammar, spelling, or sentence
structure. I read them and vote accordingly. If it's a good thought,
picture, or otherwise I usually give a perfect or great. If it's
a derrogatory statement directed at someone I usually give
a boo.

Paragraph break. I know I might not be using it to the exact
purpose it was intended to be, but I doubt many people are on
the boards. It probably didn't come with instructions because we
were meant to learn how and when to use it on our own. But
thats just my 2 gil.

Also, how do we get to heavy anyways?

User avatar
MonMotha
Site Code Monkey
Site Code Monkey
Posts: 2505
Joined: Sun Jan 23, 2005 9:18 pm

Post by MonMotha » Tue Mar 15, 2005 11:10 am

Getting to heavy involves getting a certain number of ratings that average out to a certain level or higher. There's also a time requirement (number of days since registration) that some people are just starting to reach. We're keeping the actual numbers a secret for now, but there's actually one user who would be Heavy right now except for the time requirement.

We've intentionally kept the usage of the ratings system rather open. Obviously there are some minor guidelines that got posted in the original topic regarding the launch of the system, but overall it's pretty freeform. I will say that the stats on the ratings system indicate an almost 2:1 bias towards positive ratings, with the majority of those being perfect. That's not a bad thing.
A normality test:
+++ATH
If you are no longer connected to the internet, you need to apply more wax to your modem: it'll make it go faster.
If you find this funny, you're a nerd.
If neither of the above apply, you are normal. Congratulations.

User avatar
Grubb
Standard
Standard
Posts: 773
Joined: Thu Feb 03, 2005 1:01 am
Location: Drugachusetts

Post by Grubb » Tue Mar 15, 2005 11:45 am

Only a few people have had negative ratings drag them down into the Danger rank, and those people either have an antagonistic history here anyway or were involved in discussion or flaming that raised the ire of multiple people.
Well, Ho, I'm one of those being whose been "dragged down" into the danger rank. Mostly sertain people giving posts I make a "Boo" for no other reason than because they dislike me. On the flip side, some people have even gotten a "AA" based on the "Quality" of the insult.

How is it fair that I go into the "Danger" area, because one person didn't agree with the way I voted in God of War's popularity contest? Mind you, my vote was the only one that got rated.

Another example would be my cousin's introduction thread. For some stupid reason, people desided that he and I were the same person, and flamed him out, throwing "boos" at both him and me.

On the other side of things, a good way for people to get a positive rating, is by randomly insulting me. There are atleast 5 people who will give a "perfect" rating to anyone who insults me and the more innapropreate, the better.

I appriciate the attempt, but frankly, this ranking system just doesn't work. It's simply another form of flaming.
Image

User avatar
Ho
Site Admin
Site Admin
Posts: 5645
Joined: Fri Dec 31, 2004 10:26 am
Location: The Ho-House

Post by Ho » Tue Mar 15, 2005 11:55 am

Yes, it's true. But you also notice that you are not there now. So the balancing is working. You also demonstrate my other point which was that in the instances where you were ranked down to Danger it was a result of a situation that generated multiple negative votes, not just a single person. So I disagree, the system is working just fine. You may not agree with it...and that's fine, but it's working.
Image

User avatar
Gatekeeper
Standard
Standard
Posts: 50
Joined: Mon Feb 07, 2005 10:16 am
Location: Animated avatars are the devil
Contact:

Post by Gatekeeper » Tue Mar 15, 2005 2:30 pm

You make some good points, but I still have concerns. What if a person wants to express an opinion that they know is very unpopular? Should they be afraid people will downvote them just because their opinion is in the minority?

I feel this squelches creativity and discussion, when a person knows their post is likely to be rated based on the number of people agreeing with it rather than its quality or relevance.

For example I posted an unpopular viewpoint on the Cheating... thread (link). I thought I did a good job of conveying my opinion clearly and politely, and yet I was downvoted because I didn't follow the groupthink. Should I be discouraged from posting messages which are contrary to the thought-trend here?

What if all my messages conveyed unpopular viewpoints but were well written and politely expressed? Would I be in 'danger' for daring to think so differently?
Image

User avatar
O 16581 72452 5
Standard
Standard
Posts: 145
Joined: Wed Feb 02, 2005 3:45 pm
Location: Terre Haute, IN.
Contact:

Post by O 16581 72452 5 » Tue Mar 15, 2005 2:52 pm

Yo,

I dislike that ratings aren't annonymous.

User avatar
Ho
Site Admin
Site Admin
Posts: 5645
Joined: Fri Dec 31, 2004 10:26 am
Location: The Ho-House

Post by Ho » Tue Mar 15, 2005 3:03 pm

I understand your concern. I would hope that you would not choose to withhold your views solely on the fear of reprisal through the rating system.

I've seen posts rated using both kinds of criteria you mention. There will always be people who agree and disagree on topics--including agreeing and disagreeing on how they should be rated. ;)

I would direct your attention once again to the vote balancing. In your particular example, you were rated down by only one vote. This is hardly anything to worry about. You've already more than balanced that out with the ratings you've received as a result of starting this thread.

Like I said, it's all based on trends. You have proven yourself to be a quality poster (on whatever criteria may be in use) on more occasions than a few random negative votes would counter. Thus the system works as intended.
Image

User avatar
Pheadra
Standard
Standard
Posts: 735
Joined: Sat Feb 05, 2005 8:51 am
Location: Indianapolis
Contact:

Post by Pheadra » Tue Mar 15, 2005 3:09 pm

When I rate a post I usually rate it on the quality that the post gives to the thread. Although I love to read posts that are grammatically correct, I usually dont rate it based on grammar errors only because that reflects the person's writing style. I also rate on 'Off Topic' posts that are posted to flame or 'lash back.'

I also dont like the rating system when someone goes to vote a 'boo' because I voted a 'boo' for a valid reason. As if we dont know what Im talking about. When you call my bf an asshole without reason, this is a cause to 'boo' you, but then coming back and 'boo'ing me with a post that was completly valid to the thread is uncalled for.

EDIT: Also, how do we know wether we are getting to be 'dragged' down or not?
Last edited by Pheadra on Tue Mar 15, 2005 3:12 pm, edited 1 time in total.
Image
ImageThe secret to life is, oh look! Something Shiny!Image
Image

User avatar
Ho
Site Admin
Site Admin
Posts: 5645
Joined: Fri Dec 31, 2004 10:26 am
Location: The Ho-House

Post by Ho » Tue Mar 15, 2005 3:11 pm

O 16581 72452 5 wrote:Yo,

I dislike that ratings aren't annonymous.
I can certainly understand that sentiment. I considered allowing anonymous ratings, but my feeling was that the inherent accountability of non-anonymous ratings would help to further balance the system by preventing people from attempting to sway ratings one way or the other in secret. Instead, this type of action is identifiable by all and my hope was that people would vote more responsibly as a result.

I realize that this might cause some people to choose not to vote because they do not want their names attached to their votes. It is certainly your right to simply not participate in this case.
Last edited by Ho on Tue Mar 15, 2005 3:17 pm, edited 1 time in total.
Image

User avatar
Grubb
Standard
Standard
Posts: 773
Joined: Thu Feb 03, 2005 1:01 am
Location: Drugachusetts

Post by Grubb » Tue Mar 15, 2005 3:12 pm

Pheadra wrote:When you call my bf an asshole without reason, this is a cause to 'boo' you, but then coming back and 'boo'ing me with a post that was completly valid to the thread is uncalled for.
Whatever, I have plenty of reasons to call your bf an asshole, and so do a lot of others.
Image

User avatar
Pheadra
Standard
Standard
Posts: 735
Joined: Sat Feb 05, 2005 8:51 am
Location: Indianapolis
Contact:

Post by Pheadra » Tue Mar 15, 2005 3:15 pm

Grubb wrote:
Pheadra wrote:When you call my bf an asshole without reason, this is a cause to 'boo' you, but then coming back and 'boo'ing me with a post that was completly valid to the thread is uncalled for.
Whatever, I have plenty of reasons to call your bf an asshole.
Not in that thread. Stop talking about it....we do have a PM system use it. You have some personal vendetta against him, and as stated before I dont care. Either way dont respond to this post in here, you want to talk to me more I have aim and/or PM me.
Image
ImageThe secret to life is, oh look! Something Shiny!Image
Image

User avatar
Grubb
Standard
Standard
Posts: 773
Joined: Thu Feb 03, 2005 1:01 am
Location: Drugachusetts

Post by Grubb » Tue Mar 15, 2005 3:17 pm

Pheadra wrote:
Grubb wrote:
Pheadra wrote:When you call my bf an asshole without reason, this is a cause to 'boo' you, but then coming back and 'boo'ing me with a post that was completly valid to the thread is uncalled for.
Whatever, I have plenty of reasons to call your bf an asshole.
Not in that thread. Stop talking about it....we do have a PM system use it. You have some personal vendetta against him, and as stated before I dont care. Either way dont respond to this post in here, you want to talk to me more I have aim and/or PM me.
I thought we already did that. Your the one who brought it up agian.

If you don't like me, just leave me alone. I don't have time to sit here and bitch with you all day.
Image

User avatar
God Of Rock
Standard
Standard
Posts: 875
Joined: Wed Feb 02, 2005 1:40 pm
Contact:

Post by God Of Rock » Tue Mar 15, 2005 3:18 pm

Grubb wrote:
Pheadra wrote:
Grubb wrote: Whatever, I have plenty of reasons to call your bf an asshole.
Not in that thread. Stop talking about it....we do have a PM system use it. You have some personal vendetta against him, and as stated before I dont care. Either way dont respond to this post in here, you want to talk to me more I have aim and/or PM me.
I thought we already did that. Your the one who brought it up agian.

If you don't like me, just leave me alone. I don't have time to sit here and bitch with you all day.
Okay, both of you: KNOCK IT OFF.

Locked