- MTurk is a fascinating phenomenon, especially if you're a crowdsourcing geek like me. People participate on all sorts of projects for all sorts of reasons - income, boredom, fun.
- MTurk could be valuable to Citizens Market in a variety of ways, such as researching information for our company profiles, testing our website via MTurk-driven tools such as Feedback Army, or handling a host of administrative chores that plague our little start-up.
- MTurk has attracted substantial criticism on minimum wage issues, namely that US-based employers are paying MTurk workers ("turkers") pennies per task, and that such workers are increasingly sourced from developing countries like India. Apparently the model is legal, but it does seem to be in a moral gray area that makes me uncomfortable.
- Personal ethics aside, Citizens Market could be exposed to substantial brand risk by using MTurk, as one of the corporate behavioral issues we cover on our website is "Employee Wages and Benefits."
Thursday, May 27, 2010
Mechanical Turk: A Brilliant Model... for Exploitation?
Thursday, May 13, 2010
Puzzler #2: Building on Bias
First, the context. We're building a platform for crowdsourced ratings of corporate behavior. In just a few weeks anyone will be able to write a short review and give a 1-5 star rating for any company's performance on a variety of social and environmental issues. And anyone else can provide a "peer rating" by clicking a "thumbs up" or "thumbs down" to indicate whether or not they find the original review and rating of the company to be persuasive. (Our system is similar to reviews and peer ratings in Yelp or Amazon.)
Eventually we expect all of our company ratings and peer ratings to be supplied by our community. But to get the ball rolling our team is contributing our own ratings of companies. We'll invite our community to rate the quality of the information contributed by our team. Conversely, we'll rate the quality of information contributed by our community.
Here's where it gets interesting - and tricky. Should our team members peer-rate other team members' reviews?
On one hand, there are some good reasons to answer "yes" and enable anyone to rate any review:
• This would be the simplest solution to implement and manage. That's a critical factor for our volunteer team.
• We'll be generating more content (peer ratings) and modeling behavior for our community.
• I like the idea of our team engaging in candid public feedback of each others' reviews. This approach could help mobilize our own volunteer review-writers and build our internal team rapport. This approach could also help establish our culture and build a sense of authenticity with our community.
On the other hand, there are good reasons to refrain from team ratings of team reviews:
• Crowdsourced ratings work best when derived from diverse, independent participants (Surowiecki, The Wisdom of Crowds). We can't escape the fact that our team is necessarily biased to favor each others' reviews.
• We'd probably discourage any other group from doing this on our site, such as an advocacy organization having staff write reviews and then rating each other.
Hmm. This touches on a broader tension. Community building requires sharing and inter-connectivity. But crowdsourcing theory calls for diversity and independence.
When a community member reviews and rates a company, presumably we'd encourage them to share that content via e-mail, Facebook, Twitter, or other social media. We'd want that community member to invite their friends to rate their review and thereby get connected with Citizens Market. Experts tell us that this 'sharing' feature is critical for building social websites (Porter, Designing For The Social Web). But we run into the same challenge: your friends are biased to favor your reviews!
Perhaps in the interest of community building we should encourage our team to rate each other and encourage users to invite peer ratings from friends. One day we hope to have a large enough crowd that these biases would have negligible impact.
Have we missed something here? What approach would you like to see us take?
- Stephane
Monday, April 5, 2010
New Citizens Market survey
Monday, March 1, 2010
Customer Service Mania
It's like hot potato. Except I take it seriously. When the phone rings, I literally jump over the desks just so I can get to the phone before anyone else. I love talking to customers, even angry ones. I learn a lot from them about how to make the site easier to use. When the call's over, I'll say, 'If you have any follow-up questions, my name is Paul English; I'm the co-founder of the company.' I'll give out my personal cell-phone number. Only one out of 20 people might actually call, but they're blown away when I do that."
Friday, February 19, 2010
Should crowdsourced ratings websites publish their algorithms?
The last issue of Inc. magazine featured a fascinating, in-depth article about Yelp, the crowdsourced website for restaurant reviews. This is a great article for anyone interested in crowdsourcing: http://www.inc.com/magazine/20100201/youve-been-yelped.html
The article focuses on the controversy around some small business owners who are frustrated with their treatment by Yelp's crowdsourced community. I was particularly interested in the fact that Yelp does not disclose the algorithms they use for developing the restaurant ratings, and this decision seems to be fueling the controversy. It makes me wonder: is it a good idea for Yelp to keep its algorithms a secret?
Before going further, I should say that I'm personally a big fan of Yelp. As a consumer, it continues to be my first source of information when trying to find a good place to eat. And I'm not alone. According to the Inc. article, "Yelp is by some measures the most popular reviews website in the world, with more than 26 million monthly readers and a library of user-generated content that is probably matched only by Wikipedia. There are some eight million Yelp reviews, covering service businesses in most major American metropolitan areas, along with
I'm also a professional admirer of Yelp. Our crowdsourced model at Citizens Market shares some similarities to Yelp's model, so in informal settings I often explain Citizens Market to people by saying that it's a bit like Yelp except that instead of reviewing a restaurant, our community reviews a company's social and environmental behavior. [Thanks to Dan Heath, co-author of Made to Stick, who taught me to describe a complex new idea (Citizens Market) by using an 'anchor' (Yelp) and a 'twist' to explain how we're different. I use that trick all the time!]
So my questions here about Yelp are posed with love.
What puzzles me is why Yelp continues to keep its algorithms a secret, long after its launch and the establishment of a large, robust community. I can understand why a start-up company with a crowdsourced model would want to keep its algorithms a secret during pre-launch stealth mode, to avoid having its intellectual property stolen by someone else. But once you have established a huge community and all the social capital that goes with it, I wonder if the costs of secrecy begin to outweigh the benefits.
The costs of secrecy are clear: users and stakeholders are always wondering if the rating system is fair. Here's an interesting excerpt from the Inc. article:
"Yelp lets anyone critique any business and grade it, with ratings from one star to five stars. Yelp then uses a closely guarded algorithm -- the company won't discuss even the basics of how it works -- to determine which reviews are displayed prominently, which are buried, and which are removed from the site...
... Some business owners have reported seeing their Yelp ratings fall after they declined to buy advertising. The rumblings came to the surface in a 2009 article that appeared in the East Bay Express, a weekly newspaper in
But the suspicion and anger are symptomatic of a larger problem, namely that Yelp's algorithm is a mystery to nearly everyone outside the company. Stoppelman says this is necessary to prevent business owners from hiring shill reviewers, but nearly every business owner I spoke with in reporting this story complained of being caught in the crossfire. "We've had some positive reviews suddenly disappear," says Laurie Lavy, the owner of an upscale home furnishings store in
I haven't been able to find anything on Yelp's site, FAQ or blog that lays out the rationale for keeping the algorithms secret. However, Yelp does elaborate on their "review filter" and efforts to prevent gaming their system on their blog. Yelp's experience confirms that gaming is a real problem, if only by a small fraction of companies. So the assumption seems to be that keeping the algorithms secret will help prevent companies from gaming the system.
But is that really true? Business owners can hire shill reviewers regardless of whether the algorithms are public. Even without access to the algorithms, anyone would naturally presume that writing positive reviews of a company and perhaps giving positive "compliments" on those reviews would tend to push a company's score higher. Hiding the algorithms may have some marginal benefit in preventing gaming - but does that outweigh the substantial costs of secrecy?
Perhaps Yelp can't tell us why their secret sauce needs to be kept secret because just talking about it would give away the secret. I suppose we'll never know.
And that leaves me wondering. At Citizens Market, we're planning to post our algorithms when we launch our beta website. Aside from the obvious boost in our credibility, we think that opening our algorithms to suggestions from our community will, on balance, strengthen rather than weaken our ability to prevent gaming.
What do you think? We'd love to hear your thoughts on this.
- Stephane
PS - Earlier in 2009, Yelp's Stoppelman responded to the unfolding controversy by blogging about "nine myths" about Yelp. It's an interesting read. For example, I've often assumed that consumer reviews will tend to be negative and so I was surprised - and heartened - to learn that 85% of Yelp! reviews are 3+ stars.
Thursday, October 22, 2009
I Spy A Lie?
Monday, October 19, 2009
Axe & Dove
Should we care about the CSR performance of Unilever's brand Axe when considering Unilever's brand Dove?
A while ago my friend Isaac posted a great review about Unilever, an Anglo-Dutch consumer goods conglomerate. He was inspired by an interesting YouTube video about Unilever's brands Dove and Axe, and claimed that the video demonstrates the "cynical paradigm of Unilever and other corporations that use positive social messages only when it's profitable."
Isaac summed up the video nicely:
"The gist of it is that Unilever owns both Dove and Axe. The recent Dove advertising campaign seems to be at least making steps towards a less harmful ideal of the feminine physique (e.g. the "Dove Evolution" commercial, also on YouTube). But this is undermined by Axe advertisements, which are impressively demeaning towards women. (Even if you find them entertaining, it's probably not the worldview you'd want to bequeath to your daughter.) And Unilever sells more products inasmuch as consumers believe the people behind Dove actually care about addressing body image issues in our society...
... there are worse things than being a hypocrite. Maybe we shouldn't expect corporations not to play both sides when there's money to be made. But there is real damage done here: it gives cause for cynicism and suspicion whenever anyone looks like they're doing a good thing."
Isaac's review illustrates a tricky question about how our team at Citizens Market should structure the scores for the social and environmental performance of brands and companies in our database. We'd love to hear what you think.
One option is to post a separate score for each brand, even if multiple brands are owned by a single parent company. A search for "Axe" would take the user to a profile page for Axe with a unique score for Axe. A search for "Dove" would lead to the Dove profile page with a different score for Dove. Both the Axe and Dove pages would notify the user that the brand is owned by Unilever, which would have yet another score combining Axe, Dove and all its other brands. Let's call this the "brand score" option.
Another option is to post only one score for the parent company, where all brands owned by a company inherit its score. A search for "Axe" would take the user to the Unilever profile page with Unilever's score, with an explanation that Axe is owned by Unilever. Same thing for Dove. Let's call this the "parent company score" option.
As a potential user of our scores, what would you prefer? Some folks like the nuance of the "brand score" option, reasoning that this will empower users with more information and create more incentive for individual brand teams to improve their social and environmental performance. Other folks want to stay focused on the overall performance of the parent company, arguing that laggard brands like Axe will receive more pressure from peer brands and corporate headquarters, and that parent company scores are the best way to battle the hypocrisy evident in the Axe-Dove case.
From a pragmatic perspective, it is certainly easier for us to pursue the "parent company score" option because there are fewer unique profiles and scores involved, and fewer interactive levels between scores. That means a lot less work for our crowdsourced information contributors and our in-house web developers. Therefore, we'll start with the "parent company score" option as we develop our website and community. But with time it may make sense to get more granular and turn to the "brand score" option. And in that case we'd want to start planning now.
We'd love to hear what you think and why. Should we stick with just one score for the parent company and all its brands? Or should we delve into different scores for different brands within a company?
- Stephane