Puzzler #2: Building on Bias

Our team of volunteers is grappling with some tricky questions. Should a team member rate another team member's review of a company's social / environmental behavior? Should we encourage our community to invite peer ratings from friends?

First, the context. We're building a platform for crowdsourced ratings of corporate behavior. In just a few weeks anyone will be able to write a short review and give a 1-5 star rating for any company's performance on a variety of social and environmental issues. And anyone else can provide a "peer rating" by clicking a "thumbs up" or "thumbs down" to indicate whether or not they find the original review and rating of the company to be persuasive. (Our system is similar to reviews and peer ratings in Yelp or Amazon.)

Eventually we expect all of our company ratings and peer ratings to be supplied by our community. But to get the ball rolling our team is contributing our own ratings of companies. We'll invite our community to rate the quality of the information contributed by our team. Conversely, we'll rate the quality of information contributed by our community.

Here's where it gets interesting - and tricky. Should our team members peer-rate other team members' reviews?

On one hand, there are some good reasons to answer "yes" and enable anyone to rate any review:
• This would be the simplest solution to implement and manage. That's a critical factor for our volunteer team.
• We'll be generating more content (peer ratings) and modeling behavior for our community.
• I like the idea of our team engaging in candid public feedback of each others' reviews. This approach could help mobilize our own volunteer review-writers and build our internal team rapport. This approach could also help establish our culture and build a sense of authenticity with our community.

On the other hand, there are good reasons to refrain from team ratings of team reviews:
• Crowdsourced ratings work best when derived from diverse, independent participants (Surowiecki, The Wisdom of Crowds). We can't escape the fact that our team is necessarily biased to favor each others' reviews.
• We'd probably discourage any other group from doing this on our site, such as an advocacy organization having staff write reviews and then rating each other.

Hmm. This touches on a broader tension. Community building requires sharing and inter-connectivity. But crowdsourcing theory calls for diversity and independence.

When a community member reviews and rates a company, presumably we'd encourage them to share that content via e-mail, Facebook, Twitter, or other social media. We'd want that community member to invite their friends to rate their review and thereby get connected with Citizens Market. Experts tell us that this 'sharing' feature is critical for building social websites (Porter, Designing For The Social Web). But we run into the same challenge: your friends are biased to favor your reviews!

Perhaps in the interest of community building we should encourage our team to rate each other and encourage users to invite peer ratings from friends. One day we hope to have a large enough crowd that these biases would have negligible impact.

Have we missed something here? What approach would you like to see us take?

- Stephane