A response to David Leonhardt's "When the Crowd Isn't Wise"

David Leonhardt wrote an excellent piece in the New York Times on both the effectiveness and limitations of prediction markets.

Mr. Leonhardt was inspired to write his column after prediction markets, notably Intrade, predicted that there was a 70 percent likelihood that the Supreme Court would overturn the Individual Mandate, which ultimately did not happen. This led to an explosion of criticism of Intrade on Twitter and elsewhere, with most commenters concluding that Intrade being "wrong" in this sample size of one proves that it isn't a useful tool for prediction.

We have a few points to add to this discussion, both in agreement with Mr. Leonhardt and in defense of markets.

First, we wish that Mr. Leonhardt had mentioned one of the primary advantages that prediction markets have over experts: while experts generally forecast a binary outcome for an event ("will happen" or "will not happen"), markets provide a probability of each outcome. Forecasting with probabilities versus binary outcomes provides a lot more information, but it is nuanced. For any given prediction, the only way that a market (or expert) can be easily understood as right or wrong is if it predicts with 100% certainty that an event will happen and then it does, or with 0% certainty that an event will happen and then it doesn't. This binary framework is easier for us to grasp, but it is not reflective of reality.

For those willing to consider the nuances, we can more accurately judge a probability forecast on two different rubrics: its calibration (i.e., percent that come true versus probability) and its precision (i.e., how far towards 0 or 100 percent it goes, while maintaining its calibration).

Calibration can be a tricky concept to grasp, but here's a simple example: If Intrade has markets for 10 different events and predicts a 90% likelihood for each of them to happen, 9 out of the 10 events should happen. Importantly, 1 out of the 10 should NOT happen. If all 10 happen, the markets were mis-calibrated, just as they were mis-calibrated if only 8 happen.

As for precision, as Mr. Leonhardt correctly pointed out, in times of low information the market should be less precise. This is certainly the case when it comes to a Supreme Court decision. To some extent, the market itself recognized that this was a difficult event to predict, and responded accordingly. That is why the markets stalled at 70 percent likelihood on the Supreme Court ruling rather than going up to 80 or 90 percent. The ex-ante information (and, to an extent, the ex-post information backs up the ex-ante) pointed to a likely overturn of the mandate, but the market was not willing to move into 'certain' territory. Further, the market also revealed a key indicator of low information on top of the low precision forecast. When the 'spread' of a contract – the difference between what people are willing to pay and what people are willing to sell for – moves past 5 percentage points, we consider the market to be illiquid. It did that prior to the Supreme Court ruling.

For markets with more information (such as the upcoming presidential election), we can expect far more precision, and as the sample size grows (such as the numerous markets for state-by-state Electoral College, senatorial, and gubenatorial elections) we can expect far better, and easier-to-understand, calibration.

We also wish that Mr. Leonhardt mentioned one other advantage: prediction markets update in real-time while experts update at will. Markets are always the most recent aggregation, and are available whenever you need the data. Providing easier access to this real-time data was one of the primary motivations for us to create PredictWise.

Mr. Leonhardt's conclusion, if we may paraphrase it, was dead on: prediction markets aren't perfect, especially for certain types of events, but they are still a very useful – perhaps the most useful – tool for prognostication. We agree with his suggestion that the future of predictions is to find innovative ways to combine the data gleaned from multiple sources (markets, polls, experts, etc.) to create the best possible predictions. That is our long-term goal at PredictWise.

In the coming weeks we will start providing state-by-state forecasts of the Electoral College, senatorial and gubernatorial elections and these predictions will reflect much more than just raw prediction market prices. The predictions will first debias prediction market prices and then combine them with forecasts derived from debiased-voter intention polling data and fundamental data. In theory, prediction markets should be perfectly efficient and should include all of the polling and fundamental data (i.e., they should not need any corrections or combine with other available data to form more accurate predictions). But, empirically, with extensive academic investigations, we know that is not the case.

Stay tuned.
-David & Andrew