This is a guest post by Ben Golden/@BenGoldn who is an Engineer at Cultivate Labs. I added a few comments on the bottom!

When I first heard about play-money prediction markets, I was skeptical that they could be as interesting, useful, or accurate as real-money equivalents.  I'd played enough poker with friends to know that without a financial incentive, hardly anyone took the game seriously.  Would it be different with prediction markets?  I was interested enough in the underlying technology that I wanted to explore further.

It turns out this exact question has been studied empirically.  In 2004, researchers compared play- and real-money forecasts on 208 NFL games and found that the play-money markets were slightly more accurate.  A 2006 analysis found that the performance of play- and real-money markets varied based on different topics.  A 2010 study found that play-money markets are slightly more accurate on the whole, but that in direct comparison when trading volume is equal, real-money markets are more accurate. 

Altogether, it's not clear whether tying money to prediction markets makes them more accurate.  While I initially found this result surprising, I've come to recognize a number of advantages held by play-money, which can partially or completely offset the lack of a financial incentive.

1) Larger potential user pool

Real-money prediction markets present forecasters with legal/regulatory, psychological, financial and user experience challenges that play-money markets largely avoid.  Betfair doesn't allow US residents to participate in most of its markets.  PredictIt does, but has strict limits on how users can participate.  Further, these sites are unappealing to people who don't want, or can't afford to lose money–a 2013 UK study found that only 3 percent of respondents had placed an online bet on an event or sport.  Finally, real-money markets generally employ a double-auction market mechanism that is appealing to those familiar with financial markets, but hard for newcomers to adopt.  By contrast, play-money markets can provide a simpler game-like interface, which is better suited to a wider audience.

The restrictions presented by real-money markets can additionally create bias, as only certain types of person are participating in the market.  By taking real money out of the equation, free prediction markets allow for a larger and more diverse user base, which is known to lead to better forecasts.

2) Less vulnerable to manipulation

One potential concern with real-money prediction markets is the ability for a single well-financed user to manipulate markets to tell a particular story, which reportedly happened during the 2012 presidential election.  It’s much harder to do this in play money markets because individuals can’t control markets by depositing vast sums of money; to grow in influence, they need to establish a track record of successful forecasting.  Real-money markets can combat manipulation by placing limits on how much participants can wager, but in so doing, they also reduce their accuracy advantage, since forecasters’ “skin in the game” becomes limited.

3) Competing attitudes toward risk

When cash is at stake, people who are averse to financial risk are often reluctant to forecast questions where they hold weak opinions based on hunch, instead focusing only on questions where they’ve conducted in-depth analyses.  I've written previously about the importance of capturing both rigorous analysis and users’ intuition and hunches.  Free-money markets, by removing the risk of financial loss, make it easier for forecasters to express their opinions, whereas they might decline to act when real money is in play.

The flip side to this argument is that without financial risk, forecasters may become too insensitive to risk, expressing their hopes (or fears), rather than objective assessments of likely outcomes.  So in different situations, each approach may prove more effective.

Conclusion

While real-money markets have a stronger incentive for users to forecast accurately, play-money markets can foster increased user participation, capturing insight that otherwise would be lost.  Further study is required to determine conclusively which type of market is more accurate, but both approaches have their advantages. 

Further, the competition between real- and play-money markets helps them improve.  Since most prediction markets are public, forecasters can use results from one market to inform their forecasts in another.  Forecasters on Inkling — a free site — can look at Betfair, PredictIt, and iPredict, while forecasters on Betfair can use public resources like Inkling, FiveThirtyEight, or Sports Club Stats to guide their decisions.  Since these sources capture different information from different groups users, they produce unique forecasts, which in turn helps both forecasters and observers to better understand and predict the future.

Ben Golden/@BenGoldn is an Engineer at Cultivate Labs.

Comments by David:

When I was in graduate school for economics I would pose questions to colleagues about why markets worked and they would inevitably focus on the money. Incentive compatibility they would shout! But, over time, working with expectation polling, learning about selection bias and opt-in polling, and experimenting with non-monetary incentive methods, I started to think more about the other elements that make prediction markets work. They get the right people, they ask the right questions, and they have the right aggregation methods. Markets can work without money and Ben makes a nice case here for potential advantages of play-money markets.

I pose that real-money markets versus play-money markets is not a case of right and wrong, but one of differences. Larger pools could be good (more dispersed information) or bad (more biased information from less informed people). It is both harder to manipulate play-money (because of equal wealth), but it can also be easier (infinite accounts). And, the shift in risk profile can move from risk-adverse traders in real-money to risk-loving … it is hard to get people to be risk-neutral in the real world! Just like moving from in-person, to telephone, to internet sampling shifts polling, the shift between real and play-money is one of different results, but not necessarily wrong or right as class.

I thank Ben for sending in this piece! I look forward to a robust discussion on this and related topics moving forward. If you have a piece to submit to PredictWise, please email me at David@ResearchDMR.com.