Should crowdsourced ratings websites publish their algorithms?

The last issue of Inc. magazine featured a fascinating, in-depth article about Yelp, the crowdsourced website for restaurant reviews. This is a great article for anyone interested in crowdsourcing: http://www.inc.com/magazine/20100201/youve-been-yelped.html

The article focuses on the controversy around some small business owners who are frustrated with their treatment by Yelp's crowdsourced community. I was particularly interested in the fact that Yelp does not disclose the algorithms they use for developing the restaurant ratings, and this decision seems to be fueling the controversy. It makes me wonder: is it a good idea for Yelp to keep its algorithms a secret?

Before going further, I should say that I'm personally a big fan of Yelp. As a consumer, it continues to be my first source of information when trying to find a good place to eat. And I'm not alone. According to the Inc. article, "Yelp is by some measures the most popular reviews website in the world, with more than 26 million monthly readers and a library of user-generated content that is probably matched only by Wikipedia. There are some eight million Yelp reviews, covering service businesses in most major American metropolitan areas, along with Ireland, Canada, and the United Kingdom." Wow!

I'm also a professional admirer of Yelp. Our crowdsourced model at Citizens Market shares some similarities to Yelp's model, so in informal settings I often explain Citizens Market to people by saying that it's a bit like Yelp except that instead of reviewing a restaurant, our community reviews a company's social and environmental behavior. [Thanks to Dan Heath, co-author of Made to Stick, who taught me to describe a complex new idea (Citizens Market) by using an 'anchor' (Yelp) and a 'twist' to explain how we're different. I use that trick all the time!]

So my questions here about Yelp are posed with love.

What puzzles me is why Yelp continues to keep its algorithms a secret, long after its launch and the establishment of a large, robust community. I can understand why a start-up company with a crowdsourced model would want to keep its algorithms a secret during pre-launch stealth mode, to avoid having its intellectual property stolen by someone else. But once you have established a huge community and all the social capital that goes with it, I wonder if the costs of secrecy begin to outweigh the benefits.

The costs of secrecy are clear: users and stakeholders are always wondering if the rating system is fair. Here's an interesting excerpt from the Inc. article:

"Yelp lets anyone critique any business and grade it, with ratings from one star to five stars. Yelp then uses a closely guarded algorithm -- the company won't discuss even the basics of how it works -- to determine which reviews are displayed prominently, which are buried, and which are removed from the site...

... Some business owners have reported seeing their Yelp ratings fall after they declined to buy advertising. The rumblings came to the surface in a 2009 article that appeared in the East Bay Express, a weekly newspaper in Oakland, California. The article, "Yelp and the Business of Extortion 2.0," suggested that Yelp salespeople, like Mafia foot soldiers, were threatening businesses with bad reviews if they did not buy a sponsorship package. [Yelp co-founder and CEO Jeremy] Stoppelman denies the charges.

But the suspicion and anger are symptomatic of a larger problem, namely that Yelp's algorithm is a mystery to nearly everyone outside the company. Stoppelman says this is necessary to prevent business owners from hiring shill reviewers, but nearly every business owner I spoke with in reporting this story complained of being caught in the crossfire. "We've had some positive reviews suddenly disappear," says Laurie Lavy, the owner of an upscale home furnishings store in Phoenix. "They say it's the algorithm. But the whole thing is weird." "

I haven't been able to find anything on Yelp's site, FAQ or blog that lays out the rationale for keeping the algorithms secret. However, Yelp does elaborate on their "review filter" and efforts to prevent gaming their system on their blog. Yelp's experience confirms that gaming is a real problem, if only by a small fraction of companies. So the assumption seems to be that keeping the algorithms secret will help prevent companies from gaming the system.

But is that really true? Business owners can hire shill reviewers regardless of whether the algorithms are public. Even without access to the algorithms, anyone would naturally presume that writing positive reviews of a company and perhaps giving positive "compliments" on those reviews would tend to push a company's score higher. Hiding the algorithms may have some marginal benefit in preventing gaming - but does that outweigh the substantial costs of secrecy?

Perhaps Yelp can't tell us why their secret sauce needs to be kept secret because just talking about it would give away the secret. I suppose we'll never know.

And that leaves me wondering. At Citizens Market, we're planning to post our algorithms when we launch our beta website. Aside from the obvious boost in our credibility, we think that opening our algorithms to suggestions from our community will, on balance, strengthen rather than weaken our ability to prevent gaming.

What do you think? We'd love to hear your thoughts on this.

- Stephane

PS - Earlier in 2009, Yelp's Stoppelman responded to the unfolding controversy by blogging about "nine myths" about Yelp. It's an interesting read. For example, I've often assumed that consumer reviews will tend to be negative and so I was surprised - and heartened - to learn that 85% of Yelp! reviews are 3+ stars.