LP Login

Think Big. Move Fast.

On April 15th, 2007 an excellent article in the NY Times magazine asked “Is Justin Timberlake the product of cumulative advantage?“. It describes an experiment in which a group of unknown songs was exposed to different sets of people who were able to listen, rate and download them. One group of people had no exposure to what songs others were listening to, rating and downloading. The other eight groups were exposed to feedback on what others in their group (and only their group) were doing. Here is the conclusion:

… if people know what they like regardless of what … other people like, the most successful songs should draw about the same … market share in both the independent and social-influence conditions … And … the “best” ones … should become hits in all social-influence worlds.

… we found… exactly the opposite. In all the social-influence worlds, the most popular songs were much more popular … than in the independent condition. At the same time, however, the particular songs that became hits were different in different worlds… Introducing social influence into human decision making, in other words, didn’t just make the hits bigger; it also made them more unpredictable.

… intrinsic “quality,” which we measured in terms of a song’s popularity in the independent condition, did help to explain success in the social-influence condition. … But the impact of a listener’s own reactions is easily overwhelmed by his or her reactions to others.

… Because the long-run success of a song depends so sensitively on the decisions of a few early-arriving individuals, whose choices are subsequently amplified and eventually locked in by the cumulative-advantage process, and because the particular individuals who play this important role are chosen randomly and may make different decisions from one moment to the next, the resulting unpredictability is inherent to the nature of the market.

(This is heavily edited – read the whole thing. )

This is fascinating stuff.

Jordan Schwartz and Tim O’Reilly point to this as an example of the “Hive Mind”:

You can see these same processes at work in social bookmarking and ranking services like Digg and Del.icio.us. Individual users rate a site as being interesting, causing other users to visit it and, in turn, assess whether the site is interesting enough to rate it as interesting. Over time, certain sites gain momentum and rise to the top of the heap, even though most individuals only ever see a small fraction of the options.

There is a bit of an implication that the BEST sites rise to the top due to these social processes – a sort of “Wisdom of the Crowds” effect.

Josh Porter at Bodarko draws the opposite conclusion, that

This result has huge implications for all social web sites, especially those that show aggregate data. Digg, for example, shows aggregate data everywhere on the site. This experiment, in addition to several other issues that I wrote about in Digg’s Design Dilemma, suggest that the results there are socially influenced to such an extent that it would be hard indeed to know where the quality lies…

Others have posted in the past about why the Wisdom of Crowds fails on Digg.

Its worth going back to Surowiecki’s book to understand why. Summarizing Kottke:

In order for a crowd to be smart … it needs to satisfy four conditions:

1. Diversity. A group with many different points of view will make better decisions than one where everyone knows the same information.

2. Independence. “People’s opinions are not determined by those around them.”

3. Decentralization. “Power does not fully reside in one central location, and many of the important decisions are made by individuals based on their own local and specific knowledge rather than by an omniscient or farseeing planner.”

4. Aggregation. You need some way of determining the group’s answer from the individual (i.e. unbiased) responses of its members..

Digg fails on all four of these criteria; it is not a completely diverse group of users, Digg intrinsically influences its users’ opinions, there are certain Digger’s who have more power to push a story to the front page than others, and Digging does not occur in a vacuum.

Harvard Business review has another explanation for why crowds don’t always identify the best solutions by way of the Marquis de Condorcet, a 17th Century mathematician:

Groups will do better than individuals in choosing a correct answer, and big groups better than little ones, as long as two conditions are met: the majority response “wins,” and each person is more likely than not to be correct.

(Italics mine)

In a large enough sample set of options, the likelihood that a person can choose the “best” article or “best” song rapidly drops below 50%, so this second criteria fails.

Ultimately, there are a couple of key takeaways for people who run social media sites:

1. If you’re trying to iterate towards a “best answer” then keep feedback loops to a minimum, at least before users “vote” on their own. (e.g. Hotornot, espgame)
2. If you’re trying to create “hits” out of some of your content (and don’t care if it’s the “most worthy” content – you only care that they are hits), then display feedback and popularity constantly, as this will effect user behavior and exacerbate the size of the hits (e.g. Youtube, Digg, American Idol?
3. If you want to “guide” user behavior in a certain direction, provide feedback that validates or shows the popularity of that behavior. This is consistent with my prior post on game mechanics applied to social media: keeping score.

Can you think of more examples of these three best practices? If so, please add them in comments.

  • http://gogelmogel.typepad.com Uri L.

    In a way, Last.fm radio player implements practice 1 – as u can simply mark if u like a song or not, without seeing any community stats related to it. So it’s kinda leaned into an unbiased voting process…

    I wonder what about all the answers services…which lead the mission to provide the most accurate answer utilizing wisdom of the crowds. Looking at some of them, I see that they don’t follow practice 1 that much.

  • wrwebb

    Your timing with this post is ridiculously good. Digg is complete chaos right now.

    I put a post on my blog giving some background:

    http://www.blog.robwebb2k.com/2007/05/digg-implosion.html

    or you can just go straight there to see the mayhem. Unclear how this will impact Digg in the long run. Definitely having a huge impact in the short run:

    http://www.digg.com/

  • http://www.500hats.com/ Dave McClure

    interesting post jeremy. good stuff :)

  • http://www.feedback20.com/en stephane lee

    Very interesting post, Jeremy.

    I think what practice you choose depends on the “filtering” power that you want the crowd to have.

    Do you want the crowd to produce a note (8/10,…) or do you want them to filter things, sort them out to get a concensus, even if it’s not trustworthy ?

    Both ways are valuable, and I think that combining the two would make sense :
    Phase 1/ Let a restricted community share and sort content
    Phase 2/ Ask individual people to rate the top results from Phase 1

    What do you think ?

    PS : As for Digg, I think the ratio – number of diggs / number of clicks – could be a balancing act. Maybe they already do ?

  • http://www.my-currency.com/ karim tahwi

    Great post Jeremy.

    I think you have hit this on the nose. The issue is herding versus crowd wisdom. Digg starts as wisdom of the crowds but crosses over to herding when the number of independents gets ambushed by the dependents. Ultimately, Digg is a herding platform because most people are consuming selection (or manipulating selection) rather than making it.

    We (www.my-currency.com) are using prediction markets and other social apps to aggregate future values of properties. We do not close information loops but actually encourage sharing so as to improve the aggregate knowledge in the community. Although we risk diluting independence and diversity, we feel markets manage this efficiently by allowing people to vote a problem up or down to varying degrees. We also measure performance over time and assign reputation scoring to individuals so their market impact is more substantial as they are proving to have knowledge. So our model gets smarter – as long as we continue to sponge new knowledge participants.

  • Pingback: More on using crowds to your advantage « Lightspeed Venture Partners Blog()

  • http://vanhecke.info Pascal Van Hecke

    Great post, as always – but the first “or” should be “of” in your title :-)

  • Pingback: Viral marketing, randomness and the difficulty of controlling growth in social media « Lightspeed Venture Partners Blog()

  • http://www.triggit.com embed

    Good stuff Jeremy. I continue to be impressed.

  • Pingback: Website Hits Are Shoddy Data, Now More Than Ever «()

  • Pingback: Facebook selling digital gifts at a $35m run rate « Lightspeed Venture Partners Blog()

  • Pingback: How to use popularity lists to influence your users behavior « Lightspeed Venture Partners Blog()

  • Pingback: How to use popularity lists to influence your users behavior | Bookmarks()