Mechanical Turk: A Brilliant Model... for Exploitation?

In recent months several folks have recommended that we use Amazon's Mechanical Turk (MTurk), a popular business tool that crowdsources simple tasks for very small fees per task.  MTurk has gained a reputation for getting grunt work done quickly, effectively and cheaply.  

I poked around a bit and here are my conclusions so far:

1. MTurk is a fascinating phenomenon, especially if you're a crowdsourcing geek like me.  People participate on all sorts of projects for all sorts of reasons - income, boredom, fun.
2. MTurk could be valuable to Citizens Market in a variety of ways, such as researching information for our company profiles, testing our website via MTurk-driven tools such as Feedback Army, or handling a host of administrative chores that plague our little start-up.  
3. MTurk has attracted substantial criticism on minimum wage issues, namely that US-based employers are paying MTurk workers ("turkers") pennies per task, and that such workers are increasingly sourced from developing countries like India.  Apparently the model is legal, but it does seem to be in a moral gray area that makes me uncomfortable.  
4. Personal ethics aside, Citizens Market could be exposed to substantial brand risk by using MTurk, as one of the corporate behavioral issues we cover on our website is "Employee Wages and Benefits."

For these reasons I can understand why some others are using MTurk but I don't think we should use it here at Citizens Market.  Thoughts?

Puzzler #2: Building on Bias

Our team of volunteers is grappling with some tricky questions. Should a team member rate another team member's review of a company's social / environmental behavior? Should we encourage our community to invite peer ratings from friends?

First, the context. We're building a platform for crowdsourced ratings of corporate behavior. In just a few weeks anyone will be able to write a short review and give a 1-5 star rating for any company's performance on a variety of social and environmental issues. And anyone else can provide a "peer rating" by clicking a "thumbs up" or "thumbs down" to indicate whether or not they find the original review and rating of the company to be persuasive. (Our system is similar to reviews and peer ratings in Yelp or Amazon.)

Eventually we expect all of our company ratings and peer ratings to be supplied by our community. But to get the ball rolling our team is contributing our own ratings of companies. We'll invite our community to rate the quality of the information contributed by our team. Conversely, we'll rate the quality of information contributed by our community.

Here's where it gets interesting - and tricky. Should our team members peer-rate other team members' reviews?

On one hand, there are some good reasons to answer "yes" and enable anyone to rate any review:
• This would be the simplest solution to implement and manage. That's a critical factor for our volunteer team.
• We'll be generating more content (peer ratings) and modeling behavior for our community.
• I like the idea of our team engaging in candid public feedback of each others' reviews. This approach could help mobilize our own volunteer review-writers and build our internal team rapport. This approach could also help establish our culture and build a sense of authenticity with our community.

On the other hand, there are good reasons to refrain from team ratings of team reviews:
• Crowdsourced ratings work best when derived from diverse, independent participants (Surowiecki, The Wisdom of Crowds). We can't escape the fact that our team is necessarily biased to favor each others' reviews.
• We'd probably discourage any other group from doing this on our site, such as an advocacy organization having staff write reviews and then rating each other.

Hmm. This touches on a broader tension. Community building requires sharing and inter-connectivity. But crowdsourcing theory calls for diversity and independence.

When a community member reviews and rates a company, presumably we'd encourage them to share that content via e-mail, Facebook, Twitter, or other social media. We'd want that community member to invite their friends to rate their review and thereby get connected with Citizens Market. Experts tell us that this 'sharing' feature is critical for building social websites (Porter, Designing For The Social Web). But we run into the same challenge: your friends are biased to favor your reviews!

Perhaps in the interest of community building we should encourage our team to rate each other and encourage users to invite peer ratings from friends. One day we hope to have a large enough crowd that these biases would have negligible impact.

Have we missed something here? What approach would you like to see us take?

- Stephane

New Citizens Market survey

Help us out by taking a short survey and you'll be eligible to win a $100 gift card!

Citizens Market is exploring outreach strategies to attract folks to rate companies on our website. We're lucky to have a great team of MBA students from Boston University's "Marketing Social Change" class who have conducted research and prepared a survey. Take just a few minutes to complete the survey and help us understand which social and environmental issues matter to consumers and what will motivate people to share content on our site. Your survey responses will help us in our mission to empower consumers with tools to shop their values.

Here's the survey link: http://tinyurl.com/citizensmarket

Thanks!

-Stephane

Customer Service Mania

Customer service should be a high priority for any business. But is it possible to go overboard?

One of my favorite aspects of Inc. magazine is their monthly "The Way I Work" series, which zooms in on the daily life of a successful entrepreneur. Last month Inc. covered Kayak co-founder Paul English, highlighting his focus on customer service:

http://www.inc.com/magazine/20100201/the-way-i-work-paul-english-of-kayak.html

These two paragraphs stood out for me:

"About a year ago, I bought a red telephone with a really loud ringer for the office. Whenever a customer calls the help number on our website, that phone rings. The engineers initially complained about it. They said, 'That's so friggin' annoying!' And I'd say, 'There's a really simple solution: Answer the friggin' phone and do whatever it takes to make that customer happy. Then hang up, unplug the phone, walk it down to the other end of the office, and plug it in down there.'

It's like hot potato. Except I take it seriously. When the phone rings, I literally jump over the desks just so I can get to the phone before anyone else. I love talking to customers, even angry ones. I learn a lot from them about how to make the site easier to use. When the call's over, I'll say, 'If you have any follow-up questions, my name is Paul English; I'm the co-founder of the company.' I'll give out my personal cell-phone number. Only one out of 20 people might actually call, but they're blown away when I do that."

I find myself wondering if this scenario would read differently if the employees were interviewed. Those two paragraphs are full of "I" and not a single "we." What would it be like to work in that environment? Does an obsession with customer service foster accountability and creativity... or distraction?

Perhaps this is related to a broader battle about the costs and benefits of distraction. Some productivity gurus warn against the hidden costs of interruption, mainly due to the time required to re-focus on the task at hand. A few months ago I learned to disable the pop-up e-mails from my e-mail client, and I'm thankful for that. On the other hand, just this morning I read an article in Wired about the creative value of allowing oneself to get periodically distracted with a few Tweets or Facebook updates. Perhaps the customer service interruptions for English's team are similarly rejuvenating.

Of course, I cherry-picked the excerpt above from a heavily packaged article so there is undoubtedly more to English's side of it. Besides, he has founded four companies and sold one of them to Intuit. So he clearly has some credibility on how teams work.

That's why these "The Way I Work" articles are so great. You read the article and think, "This person is nuts." And then you remember, "This person is wildly successful." And then you start to wonder, "What else am I taking for granted?"

- Stephane

PS - This is my second blog about Inc. content in as many weeks. I know. I sleep with Inc. under my pillow now.

Should crowdsourced ratings websites publish their algorithms?

The last issue of Inc. magazine featured a fascinating, in-depth article about Yelp, the crowdsourced website for restaurant reviews. This is a great article for anyone interested in crowdsourcing: http://www.inc.com/magazine/20100201/youve-been-yelped.html

The article focuses on the controversy around some small business owners who are frustrated with their treatment by Yelp's crowdsourced community. I was particularly interested in the fact that Yelp does not disclose the algorithms they use for developing the restaurant ratings, and this decision seems to be fueling the controversy. It makes me wonder: is it a good idea for Yelp to keep its algorithms a secret?

Before going further, I should say that I'm personally a big fan of Yelp. As a consumer, it continues to be my first source of information when trying to find a good place to eat. And I'm not alone. According to the Inc. article, "Yelp is by some measures the most popular reviews website in the world, with more than 26 million monthly readers and a library of user-generated content that is probably matched only by Wikipedia. There are some eight million Yelp reviews, covering service businesses in most major American metropolitan areas, along with Ireland, Canada, and the United Kingdom." Wow!

I'm also a professional admirer of Yelp. Our crowdsourced model at Citizens Market shares some similarities to Yelp's model, so in informal settings I often explain Citizens Market to people by saying that it's a bit like Yelp except that instead of reviewing a restaurant, our community reviews a company's social and environmental behavior. [Thanks to Dan Heath, co-author of Made to Stick, who taught me to describe a complex new idea (Citizens Market) by using an 'anchor' (Yelp) and a 'twist' to explain how we're different. I use that trick all the time!]

So my questions here about Yelp are posed with love.

What puzzles me is why Yelp continues to keep its algorithms a secret, long after its launch and the establishment of a large, robust community. I can understand why a start-up company with a crowdsourced model would want to keep its algorithms a secret during pre-launch stealth mode, to avoid having its intellectual property stolen by someone else. But once you have established a huge community and all the social capital that goes with it, I wonder if the costs of secrecy begin to outweigh the benefits.

The costs of secrecy are clear: users and stakeholders are always wondering if the rating system is fair. Here's an interesting excerpt from the Inc. article:

"Yelp lets anyone critique any business and grade it, with ratings from one star to five stars. Yelp then uses a closely guarded algorithm -- the company won't discuss even the basics of how it works -- to determine which reviews are displayed prominently, which are buried, and which are removed from the site...

... Some business owners have reported seeing their Yelp ratings fall after they declined to buy advertising. The rumblings came to the surface in a 2009 article that appeared in the East Bay Express, a weekly newspaper in Oakland, California. The article, "Yelp and the Business of Extortion 2.0," suggested that Yelp salespeople, like Mafia foot soldiers, were threatening businesses with bad reviews if they did not buy a sponsorship package. [Yelp co-founder and CEO Jeremy] Stoppelman denies the charges.

But the suspicion and anger are symptomatic of a larger problem, namely that Yelp's algorithm is a mystery to nearly everyone outside the company. Stoppelman says this is necessary to prevent business owners from hiring shill reviewers, but nearly every business owner I spoke with in reporting this story complained of being caught in the crossfire. "We've had some positive reviews suddenly disappear," says Laurie Lavy, the owner of an upscale home furnishings store in Phoenix. "They say it's the algorithm. But the whole thing is weird." "

I haven't been able to find anything on Yelp's site, FAQ or blog that lays out the rationale for keeping the algorithms secret. However, Yelp does elaborate on their "review filter" and efforts to prevent gaming their system on their blog. Yelp's experience confirms that gaming is a real problem, if only by a small fraction of companies. So the assumption seems to be that keeping the algorithms secret will help prevent companies from gaming the system.

But is that really true? Business owners can hire shill reviewers regardless of whether the algorithms are public. Even without access to the algorithms, anyone would naturally presume that writing positive reviews of a company and perhaps giving positive "compliments" on those reviews would tend to push a company's score higher. Hiding the algorithms may have some marginal benefit in preventing gaming - but does that outweigh the substantial costs of secrecy?

Perhaps Yelp can't tell us why their secret sauce needs to be kept secret because just talking about it would give away the secret. I suppose we'll never know.

And that leaves me wondering. At Citizens Market, we're planning to post our algorithms when we launch our beta website. Aside from the obvious boost in our credibility, we think that opening our algorithms to suggestions from our community will, on balance, strengthen rather than weaken our ability to prevent gaming.

What do you think? We'd love to hear your thoughts on this.

- Stephane

PS - Earlier in 2009, Yelp's Stoppelman responded to the unfolding controversy by blogging about "nine myths" about Yelp. It's an interesting read. For example, I've often assumed that consumer reviews will tend to be negative and so I was surprised - and heartened - to learn that 85% of Yelp! reviews are 3+ stars.

I Spy A Lie?

Skimming Cone's new 2009 Consumer New Media Study, I'm intrigued that 47% of us "think companies are transparent and honest when it comes to talking about their corporate responsibility efforts using new media."

Is the glass half-empty or half-full? One the one hand, this only confirms a widespread understanding that the public is suspicious of the private sector. But I would have expected much worse; I was pleasantly surprised that the trust level is as high as it is. The web has created a trend towards transparency in corporate communications, and I think the public is catching on.

Our team at Citizens Market often fields questions from folks concerned that corporations will "game" our crowdsourced information on corporate behavior by spreading lies about their company on our website. Certainly "gaming" is a real risk for which our community must be vigilant. Our main defense will be our community's ability to sniff out truth from falsehood as they apply peer ratings to reviews of corporate behavior. We'll also be exploring technical solutions, such as tracking the connection between our users' IP addresses and corporate IP addresses. (A neat example of this is Wikiscanner. Let us know if you have other ideas!)

But if gaming is a risk for us at Citizens Market, it's an even bigger risk for any company that tries it. False public statements by corporations can have tremendous negative consequences for their brand and legal liability. In a landmark case in 1998, Kasky v Nike, California's Supreme Court ruled that Nike had engaged in false advertising by publicly denying the media's allegations of abominable working conditions for Nike's overseas suppliers. Nike ended up settling for $1.5 million.

It would be a categorically dumb move for a company to deliberately spread lies within our system or any other social media outlet. Any minor potential gains would be massively outweighed by certain negative media and potential legal consequences if such villainy comes to light. And the web is making it easier and easier to nab such a villain. Whistleblowers, muckrakers and hackers, oh my!

Of course, companies are run by people and sometimes people make very dumb moves. E.g., Enron. More often, we grapple with the blurred edges between dubious information and an outright lie.

What do you think? Over time are you more or less trusting of what companies say about themselves online?

- Stephane

Axe & Dove

Should we care about the CSR performance of Unilever's brand Axe when considering Unilever's brand Dove?

A while ago my friend Isaac posted a great review about Unilever, an Anglo-Dutch consumer goods conglomerate. He was inspired by an interesting YouTube video about Unilever's brands Dove and Axe, and claimed that the video demonstrates the "cynical paradigm of Unilever and other corporations that use positive social messages only when it's profitable."

Isaac summed up the video nicely:

"The gist of it is that Unilever owns both Dove and Axe. The recent Dove advertising campaign seems to be at least making steps towards a less harmful ideal of the feminine physique (e.g. the "Dove Evolution" commercial, also on YouTube). But this is undermined by Axe advertisements, which are impressively demeaning towards women. (Even if you find them entertaining, it's probably not the worldview you'd want to bequeath to your daughter.) And Unilever sells more products inasmuch as consumers believe the people behind Dove actually care about addressing body image issues in our society...

... there are worse things than being a hypocrite. Maybe we shouldn't expect corporations not to play both sides when there's money to be made. But there is real damage done here: it gives cause for cynicism and suspicion whenever anyone looks like they're doing a good thing."

Isaac's review illustrates a tricky question about how our team at Citizens Market should structure the scores for the social and environmental performance of brands and companies in our database. We'd love to hear what you think.

One option is to post a separate score for each brand, even if multiple brands are owned by a single parent company. A search for "Axe" would take the user to a profile page for Axe with a unique score for Axe. A search for "Dove" would lead to the Dove profile page with a different score for Dove. Both the Axe and Dove pages would notify the user that the brand is owned by Unilever, which would have yet another score combining Axe, Dove and all its other brands. Let's call this the "brand score" option.

Another option is to post only one score for the parent company, where all brands owned by a company inherit its score. A search for "Axe" would take the user to the Unilever profile page with Unilever's score, with an explanation that Axe is owned by Unilever. Same thing for Dove. Let's call this the "parent company score" option.

As a potential user of our scores, what would you prefer? Some folks like the nuance of the "brand score" option, reasoning that this will empower users with more information and create more incentive for individual brand teams to improve their social and environmental performance. Other folks want to stay focused on the overall performance of the parent company, arguing that laggard brands like Axe will receive more pressure from peer brands and corporate headquarters, and that parent company scores are the best way to battle the hypocrisy evident in the Axe-Dove case.

From a pragmatic perspective, it is certainly easier for us to pursue the "parent company score" option because there are fewer unique profiles and scores involved, and fewer interactive levels between scores. That means a lot less work for our crowdsourced information contributors and our in-house web developers. Therefore, we'll start with the "parent company score" option as we develop our website and community. But with time it may make sense to get more granular and turn to the "brand score" option. And in that case we'd want to start planning now.

We'd love to hear what you think and why. Should we stick with just one score for the parent company and all its brands? Or should we delve into different scores for different brands within a company?

- Stephane