9-607-088
REV: AUGUST 20, 2007

PETER A. COLES KARIM R. LAKHANI ANDREW P. MCAFEE

Prediction Markets at Google
In late March of 2007, Bo Cowgill, Ilya Kirnos, Doug Banks, Patri Friedman, and Piaw Na sat down to lunch at Google’s headquarters in Mountain View, California, and reviewed the most recent results from the company’s internal prediction markets. The five Googlers (as Google employees referred to themselves) had launched the company’s prediction markets, built the information systems that supported them, and overseen them during the previous seven quarters, all while working at their “normal” jobs. The markets had grown in popularity and demonstrated their accuracy during that time, and the team was proud of its accomplishments. Prediction markets were very much like stock markets. They contained securities, each of which had a price. People used the market to trade with one another by buying and selling these securities. Because traders had differing beliefs about what the securities were worth, and because events occurred over time that altered these beliefs, the prices of securities varied over time. In a stock market like the New York Stock Exchange the securities being traded were shares in companies, the prices of which reflected beliefs about the value of the companies. In a prediction market, in contrast, the securities being traded were related to future events such as an American presidential election. In this case, the market could be designed so that each security was linked to a candidate, and its price was the same as the estimated probability that the candidate would win, according to the market’s traders. Prediction markets on the Internet had proved to be remarkably accurate at predicting the results of political elections and other events, and the Googlers had wanted to see if they could also be productively used within companies to forecast events of interest such as the launch date of a product or whether a competitor would take a specific action. The experiences of the previous seven quarters had shown that Google Prediction Markets (GPM) were in fact quite good at predicting such events. Googlers put none of their own money at risk when they traded within GPM; instead, they bought and sold securities within GPM using “Goobles,” an artificial currency. Over lunch, the team members talked about next steps for GPM. “We’ve got a huge amount of market and trading data and we’ve hardly begun to analyze it, but all of our work so far shows that our markets continue to be accurate and decisive. I was a believer from the start, but even I didn’t think they’d work this well,” said Cowgill.
________________________________________________________________________________________________________________
Professors Peter A. Coles, Karim R. Lakhani, and Andrew P. McAfee prepared this case. HBS cases are developed solely as the basis for class discussion. Cases are not intended to serve as endorsements, sources of primary data, or illustrations of effective or ineffective management. Copyright © 2007 President and Fellows of Harvard College. To order copies or request permission to reproduce materials, call 1-800-545-7685, write Harvard Business School Publishing, Boston, MA 02163, or go to http://www.hbsp.harvard.edu. No part of this publication may be reproduced, stored in a retrieval system, used in a spreadsheet, or transmitted in any form or by any means—electronic, mechanical, photocopying, recording, or otherwise—without the permission of Harvard Business School.

607-088

Prediction Markets at Google

“But do we know that they always work well?” Banks asked. “We should dig in to our data to find out if some kinds of markets work better than others.” “Well, one thing we know even without doing much analysis is that all markets work better as they get more traders,” Friedman replied. “In fact, I’m amazed that some of our markets worked as well as they did—they didn’t attract a lot of people or a lot of trades.” “I think that’s the biggest challenge we face right now,” Kirnos agreed. “We have a group of active traders every quarter, but it’s a pretty small group. We get a lot more people who sign up, then only make one or two trades. Why aren’t they doing more, and how do we encourage them to be more active?” Cowgill had a ready answer. “You all know what I think—we should make the trading more social and more personal, so that if you want to reveal your trades or positions, GPM will let you. Right now everything’s anonymous, and I think that works against us.” “We already identify the best and most active traders every quarter and reward plenty of other people with T-shirts. It already feels pretty social to me,” Kirnos responded. “Hold on,” said Friedman. “I think our biggest challenge isn’t getting more people to participate in the markets, it’s figuring out how to use their results. Is GPM a cool, fun curiosity, or is it actually a useful tool for running this company?” “Good question,” the others replied, almost in unison.

Google
Google was founded in 1998 by Larry Page and Sergey Brin, two Ph.D. students at Stanford. At that time, many Internet search engines ranked pages based on how many times they contained a certain word or phrase. If, for example, a user typed “Boston Red Sox” into the search box of one of these engines, its first result might be the page that contained the phrase “Boston Red Sox” the most times. Website operators soon learned how these search engines worked and found it easy to take advantage of them. They would, for example, build pages that had nothing to do with the Sox but still contained “Boston Red Sox” hundreds or even thousands of times in order to attract traffic. Such pages quickly proliferated and made Internet searching difficult. The Google founders’ insight was that the “best” page about the Boston Red Sox was not the one that used that phrase most often but instead the one that was most linked to using that phrase. Web pages typically contained many links to other pages, and this link structure provided a huge amount of information. The page that the most other pages linked to on a given topic was likely to be the best page on that topic and the one that a searcher was looking for. Google also assessed the “reputation” of each page on the Web; if it was a page that a lot of other pages linked to, its own links were given more credibility. The algorithm that took all this into account, called PageRank, proved to be enormously powerful and popular. Google soon became a leader in Internet searching, and the company also found a way to generate revenue by selling advertising space on its search-results page. By March of 2007, “to Google” had become a verb meaning “to search for information on the Internet,” and the company had a market capitalization of over $140 billion. Google employed over 10,000 people and in addition to its main campus in Mountain View (called The Googleplex) had over a dozen engineering centers
2

Prediction Markets at Google

607-088

around the world. The company described its mission as “to organize the world’s information and make it universally accessible and useful.”

The Birth of GPM
Bo Cowgill joined Google in 2003 after finishing his undergraduate degree in public policy at Stanford and worked at first in the company’s support organization. As an undergraduate, he had been fascinated by prediction markets for elections and impressed with their accuracy. In June of 2004 he went on vacation and took along James Surowiecki’s book The Wisdom of Crowds. The book’s theme was that it was often possible to harness the “collective intelligence” of a group of people, yielding better or more accurate information than any individual within the group possessed. Many readers found this a powerful and novel message. They were accustomed to thinking that groups usually yielded the lowest common denominator of their members’ contributions or, even worse, that groups could turn into mobs that were actually less intelligent than any of their members. The Wisdom of Crowds provided many examples of collective intelligence and discussed prediction markets including the Iowa Electronic Markets (IEM), which had interested Cowgill at Stanford. The IEM, which was founded in 1988, established markets to predict the winners of political elections in the United States and elsewhere. In these markets, security prices could be interpreted as the percentage of the vote each candidate was predicted to get, according to traders. Participants in the IEM traded using their own money and could set up accounts with $5–$500.1 In general, IEM results were quite accurate and compared favorably with other ways of predicting events. Across 12 national elections in five countries, for example, the average margin of error of the last large-scale voter polls taken before the election was 1.93%. The average margin of error of the final IEM markets prices was 1.49%.2 Surowiecki wrote: . . . the most mystifying thing about [prediction] markets is how little interest corporate America has shown in them. Corporate strategy is all about collecting information from many different sources, evaluating the probabilities of potential outcomes, and making decisions in the face of an uncertain future. These are tasks for which [prediction] markets are tailor-made. Yet companies have remained, for the most part, indifferent to this source of potentially excellent information, and have been surprisingly unwilling to improve their decision making by tapping into the collective wisdom of their employees. Cowgill became intrigued by the idea of starting a prediction market at Google. He knew he would need colleagues, particularly ones with programming expertise, to help build one, and he also knew how to find them. When he returned to work he wrote the following note on an internal online message board where employees could post their new ideas:

1 The IEM’s website states, “The IEM is an experimental market operated for academic research and teaching purposes. The

IEM is not regulated by, nor are its operators registered with, the Commodity Futures Trading Commission or any other regulatory authority. “The IEM has received two no-action letters from the Division of Trading and Markets of the Commodity Futures Trading Commission. Without explicitly asserting jurisdiction over the IEM or any of its submarkets, these letters, dated February 5, 1992, and June 18, 1993, extended no-action relief to the IEM’s Political and Economic Indicator Markets.” http://www.biz.uiowa.edu/iem/faq.html#Regulated, accessed February 27, 2007.
2 J. Berg, R. Forsythe, F. Nelson, and T. Rietz, “Results from a Dozen Years of Election Futures Markets Research,” Handbook of

Experimental Economics Results, 2001, available at http://www.biz.uiowa.edu/iem/archive/BFNR_2000.pdf.

3

607-088

Prediction Markets at Google

By aggregating the number and nature of incoming links to a webpage, Google already uses the collective genius of crowds to rank search results. “Democracy on the web works,” is part of our corporate culture. But PageRank isn’t the only way to harness the collective intelligence of large groups. The Iowa Electronic Markets, the Policy Analysis Market, the Hollywood Stock Exchange as well as numerous academic studies have shown that large, diverse crowds of independent thinking people are better at predicting the future of solving a problem than the brightest experts among them. This is especially true when the individuals in the crowd have a personal financial stake in getting it right. Google has exactly what such a market needs to perform well: A large, diverse user base and the ability to give financial incentives and lower barriers to entry. To some extent, Google can even ensure that our crowd thinks independently. So, I propose creating Google Decision Markets. (See Exhibit 1 for the remainder of Cowgill’s post.) All Google engineers had “20% time,” or the equivalent of one day a week during which they were free to pursue projects of interest within the company that were not directly related to their jobs. Cowgill was hoping to convince some Googlers to devote their 20% time to building a prediction market. He was also looking for quick feedback about the idea, and the message board let people rate posted ideas. Ilya Kirnos posted a reply less than 10 hours after Cowgill submitted his idea: “Hey Bo I had a similar idea and have written some code in that direction. I agree that markets have a lot of predictive power, much more so than surveys or polls for most things. . . . Kirnos was a computer science and applied math major from Princeton who had joined the advertising systems group at Google in 2004 after working at Oracle. Earlier in his career he had participated in a project that he and many other engineers knew would not succeed but that continued to get support and funding. He wondered why it seemed so hard to communicate accurate information within companies in some circumstances, and he thought that it might be possible to use technology to address this problem. Kirnos built a simple application called “itoldyouso” that allowed people to offer and accept nonmonetary bets and kept track of them over time. Employees could use it, for example, to essentially say to their colleagues, “I’ll bet you this project won’t be finished on time; any takers?” When they won a bet, the system helped them say, “I told you so!” Kirnos saw that Cowgill’s proposed prediction market could be used to accomplish many of the same objectives as his betting system and volunteered to help with the project. Na and Banks also replied to Cowgill’s post and became part of the GPM team (Freidman joined the team a few months later after hearing about the project). Another respondent to Cowgill's post was an associate of Hal Varian, a well-known Berkeley economist who consulted at Google. The respondent told Cowgill that Varian had an interest in prediction markets and had written columns about them in his New York Times economics column. Cowgill contacted Varian to solicit his help, and Varian began attending the group's regular meetings. He later offered crucial advice about how to design the markets, how to implement them, and how to sell them to the larger company.
4

Prediction Markets at Google

607-088

Cowgill’s idea yielded the following ratings (Figure A): Figure A
12

Histogram of Responses to Bo Cowgill’s Post on a Google Prediction Market

10 # of responses (normalized)

8

6

4

2

0
Great idea! Make it so Very good idea Definitely has potential May not be worth the effort Probably not a Dangerous or good idea harmful if implemented

Source:

Google.

Market Design Considerations
The team felt that for GPM to be effective, the markets would have to be intuitive and easy for participants to use. Cowgill knew that many people found the idea of trading stocks to be daunting, so he wanted everything about Google’s prediction markets to be as simple and straightforward as possible. Early on, a debate ensued over the best market design. There were two leading candidates: the IEM model, and a design that let traders engage in short selling. Short selling meant betting against a future event, such as the New York Yankees beating the Boston Red Sox in a seven-game playoff series.3

3 As an example of short selling, suppose a security called “Yankees-to-win” pays out one dollar if the Yankees win and zero

otherwise. To bet on the Yankees, a trader would simply buy this security at the going ask price. To bet against the Yankees, the trader would sell short. This means that the trader would receive the current bid price for the security on the spot, in exchange for an obligation to pay a dollar should the Yankees win. Because the trader in this case is “on the hook” for a dollar, a dollar of funds will be frozen in her account until the game is over. Unlike short selling of stocks, short selling of contracts in

5

607-088

Prediction Markets at Google

Some members of the team felt that a short-selling feature would be complicated enough to deter some potential GPM traders. The sticky issue was that short selling was not intuitive to many people because it involved selling something that a trader did not own. Others argued that Web-based prediction markets that allowed short selling, such as the Ireland-based company Tradesports, showed this was not the case. After all, they reasoned, Tradesports was a successful profit-seeking company, so it must be using the model that would attract the greatest number of participants. After consultation with Varian, the team ultimately decided on the IEM model. The IEM model relied on the use of “baskets”—mutually exclusive securities that covered all possible outcomes for an event. For example, suppose the team wished to run a market on users for application A as of the end of Q4 2007. A basket might consist of the following securities: Security A1 pays off 1 if users < 1m 0 otherwise Security A2 pays off 1 if users ≥ 1m and