Normalised comment rating

NOTE HXA7241 2011-01-02T09:55Z

News-‘forum’ sites, like Hacker News, seem to rate comments just by counting votes: is that sophisticated enough as a measure?

The ‘party's ended’ problem

European users of Hacker News might have noticed this. You open HN, see an interesting post and have a good idea that must be expressed. But on looking you see it was posted 18 hours ago, gone to the top of the page and back down again, gathered 100 comments, replies and replies to them . . . – everyone has seen it, commented, and gone home. You can make the best comment possible, but it won't get upvoted.

In terms of metrics: upvotes depend on time and duration.

The ‘grade inflation’ problem

Early users of Reddit, if they return, will have noticed this. In the old days, an upvote in double figures was pretty respectable. Now, comments routinely get hundreds of upvotes. People are getting more comment karma for a single comment than you got for all your comments combined.

In terms of metrics: upvotes depend on visitor numbers.

The general problem

The size of upvote appears to simply show comparative quality, but actually also measures duration and visitor numbers. For any level of quality, the longer the comment exists and the more visitors, the larger the score will be. Compounding this is also ‘competition’: upvotes depend on being noticed amongst others. The earlier a comment exists, the less competition it has.

Possible solutions

One approach is to control upvoting. Open the post for commenting for some period, then close it and open it for voting for some number of visitors. It is something to ponder, but much easier to implement would be a kind of opposite of that: leave upvoting free, and control the ‘interpretation’. Resultant scores are not counts, but rates of voting per time duration, per competition, and per visitor. The scores are normalised for those aspects.