Election forecasters are in the midst of a spirited discussion about their models, debating both the proper level of certainty and the proper amount of volatility that a good forecast should have. I am very grateful of all the interesting and important work that both academics and data journalists have done in pushing the science of forecasting elections forward into interesting, meaningful, and dynamic spaces.

To move this conversation forward in a more constructive way, I have created, with the guidance of my colleague Glen Weyl, a creative and clear way to compare different forecasts and their dynamics. Imagine if a forecaster came to PredictWise every night at 11:59 PM ET and could wager me at my prevailing probability as much as they wanted. Would they be up or down against me at this point in the cycle? This question gets to the heart of which forecast is leading which forecast. Or, more important, if you were a decision maker using either of these forecasts to allocate resources (e.g., money), which one is better?

Assumptions: (1) The forecaster starts with $1,000,000. (2) PredictWise is buying and selling as much of two products at the forecaster wants: Democratic Nominee to Win and Republican Nominee to Win contracts. These contracts are worth $1 if the nominee wins the election and $0 if the nominee loses. (2) The forecaster uses his/her forecast at 11:59 PM ET to decide to buy or sell any given amount. Obviously, if the forecaster’s Democratic forecast is higher than the price of PredictWise, the forecaster will go long Democratic and if it is lower, the forecaster will go long Republican. (3) There are no transaction costs in any direction and all transactions occur at the marginal price (i.e., whatever the PredictWise probability is at that time). (4) The forecaster is risk neutral and uses the Kelly Scoring Rule to decide how long or short they want to be. This is a standard theory on maximizing return that takes the level of money you have and the difference between your forecast and the price to determine how much to wager each day. My colleague David Pennock has an awesome widget for you to explore how this works.

This works as comparison for forecasts, because if one forecast is leading the other, the leading forecast (the more predictive forecast) will take all of the trailing forecast’s money. Imagine if there are four days: t1, t2, t3, t4 and the Democratic probabilities for Forecaster are 80, 82, 60, 77 and the probabilities at PredictWise are 70, 68, 69, 77. So, on t1 Forecaster is going to buy a lot of Democratic contracts from PredictWise at $0.70 because her forecast is 10 pp higher than the price. On t2 Forecaster is going to buy a lot more contracts from PredictWise at $0.68 because her forecast is now much higher than PredictWise. On t3 Forecaster is going to sell all of the contracts of the Democrat to Win at $0.69 and buy some Republican to Win at $0.31 to PredictWise, because her forecast is now lower. Finally, on day t4, she sells all her Republican to Win contracts at $0.23, because they are now the same and Forecaster does not want to be up or down shares. Forecaster bought a bunch of shares at $0.70 and $0.68, and sold them at $0.69, basically netting nothing, but then bought a bunch of shares at $0.31 and sold them at $0.23 for a steep loss. This happened because PredictWise was ahead of Forecaster. PredictWise went down t1-t2 and up t2-t3, while Forecast went down t2-t3 and up t3-t4. She was trailing PredictWise by a period and it cost her.

While I hope to add every major player at some point, since I have the data for FiveThirtyEight, I am going to start there. FiveThirtyEight’s default forecast is the “Polls-Only” and their secondary forecast is the “Polls-Plus”. As you can see from the movement of the probabilities both of FiveThirtyEight’s forecasts have had (1) more movement and (2) lean Republican (or is conservative towards 50 percent) on average.

PredictWise20161027a

FiveThirtyEight would lose a lot of money if they traded against PredictWise. At any point FiveThirtyEight has two assets: cash and contracts. This creates two different points, as we can price the contracts at the liquidation price (i.e., what PredictWise would pay for them) and the expected price (i.e., what FiveThirtyEight thinks they are worth). As of midnight last night, FiveThirtyEight’s Polls-Only model would have $690,867 in cash and own 469,382 contracts for Trump, worth between $0.09 at PredictWise and $0.146 in expectation by FiveThirtyEight. FiveThirtyEight’s Polls-Plus model would have $745,378 in cash and own 703,075 contracts for Trump, worth between $0.09 at PredictWise and $0.168 in expectation by FiveThirtyEight. This represents a loss of between 24 and 27 percent of net worth for the Polls-Only, and 14 and 19 percent of net worth for the Polls-Plus.

PredictWise20161027b

A regular co-author of mine, Rajiv Sethi, after reading through a draft of this article, suggested another estimation strategy. He advised me to pretend that both forecasts are markets and, each day, a trader is arbitraging between the two markets (i.e., buying in the cheaper market and selling in the more expensive market). For this strategy, we assume that if the probabilities are greater than 5 pp apart, the trader would buy the lower value and sell the higher value. If/when they cross, the trader cashes out both accounts. This estimation strategy is explained in depth here.

In this strategy, the “market” or forecast where all the money ends up in is the trailing or less predictive forecast. Imagine there are 3 time periods: t1, t2, t3 and two forecasts: F1 and F2. On t1 both are 80, t2 F1 is 80 and F2 is 70, t3 both 80. In t1, nothing happens. In t2 the trader buys 100 contracts from F2 at $0.70 and sells 100 contracts to F1 for $0.80.  In t3, the trader covers the 100 contracts from F1 at $0.80 (for net of $0) and sells 100 contracts to F2 at $0.80, netting $10. The contract that moves down and comes back is the less predictive forecast, shown in this example as F1.

Again, when I ran this estimation strategy with PredictWise against both FiveThirtyEight’s forecasts, both FiveThirtyEight forecasts proved to be the less predictive forecast than PredictWise.

FiveThirtyEight does worse than PredictWise in both estimation strategies, because both of their models are somewhat predictable overall and specifically have predictable volatility. This is apparent in the large drops in net worth around the major swings in their forecasts, which PredictWise both leads and does so with less movement. This could be because FiveThirtyEight is limited by polling data which comes in slower than market data, or that FiveThirtyEight is more accurately demonstrating the ebbs and flows of sentiment rather than the ultimate outcome. This does not need to be one is right and one is wrong, as much as FiveThirtyEight is illustrating something different than PredictWise.

What I love about these methods is that they are not about the random outcome that occurs at the end, but the movement, as information grows and days disappear, as the eventual outcome approaches. We can liquidate the contracts on Election Eve and eliminate the element of chance and make this all about which forecast was more predictive of the movement and level in the forecasts as the outcome approached.