Economic Models, Opinion Polling and Prediction Markets
Andrew Leigh and Justin Wolfers have just published a paper on the 2004 election, comparing the predictive performance of economic models, opinion polling and betting markets.
If I can interpret their results as student grades, then opinion polls failed, economic models got a credit and the betting market a distinction. Their bitchiest comments are reserved for opinion polls, so please excuse the lengthy quotes and enjoy the cattiness.
The polls paint a more variable picture. By election-eve ACNielsen and Galaxy were predicting a Coalition victory, Newspoll declared a dead heat, and Morgan predicted a Labor victory. Three months earlier only Galaxy was predicting a Howard victory, and a year prior to the election the polls suggested, on average, a dead heat. The variability of predictions both across polls, and within polls over time, points to the difficulty of using them to form reliable forecasts.
Leigh and Wolfers converted the sampling errors to probabilities and found,
the ‘margin of error’ reported by the pollsters substantially overstates the precision of poll-based forecasts. Further, the time series volatility of the polls (relative to the betting markets) suggests that poll movements are often noise rather than signal.
But perhaps their most damning critique of opinion polls as a predictor of election outcomes was this.
In other words, we assume that the true standard error of the polls is equivalent to a poll of 25 voters that suffered only from sampling error. While this seems a rather extreme assumption, it generates a far more plausible series than when we take the pollster’s estimates of their standard error at face value. The resulting estimate ranges from 26% to 66%, still demonstrating more movement than the betting markets, but now moving within a more credible range. Moreover, this estimate appears to move in lockstep with the betting markets.
Even though this assessment of the polls strikes us as more credible, Figure 5 still suggests that the polls were systematically more pro-Labor than were the betting markets. Jackman (2005) contains a thorough assessment of the bias inherent in polls from each of the major polling organizations.
What does this suggest for those producing and publishing polls in future elections? Given their low predictive power, we propose that pollsters provide more guidance to their clients as to their (in)ability to forecast election outcomes. Our results indicate that — for forecasting purposes — the pollster’s published margins of error should at least be doubled. This is true for each of the four major pollsters covered here. At some point in the 2001-04 election cycle, ACNielsen, Galaxy and Newspoll each published figures which, if interpreted as a forecast, suggested that the Labor Party had less than a 0.5% chance of winning, while ACNielsen, Morgan and Newspoll each published figures suggesting that the Coalition had less than a 0.5% chance of being returned to power.
Consequently the media needs to display substantially greater caution in interpreting changes from one poll to the next. Indeed, even with the published margins of error, a 1% movement from one poll to the next is unlikely to be anything more than noise. But with the margins of error implied by our results, even very large movements are likely to be mere measurement error. Journalists who write about changes in poll movements without discussing the margin of error may well be guilty of misleading their readers.
The logic behind economic models to predict election outcomes is simple, ‘voters are more likely to re-elect incumbents who deliver a robust economy. This pattern can be motivated either as voters providing an incentive for politicians to deliver good outcomes, or as voters using available information to discern high-ability incumbents.’
The economic models predicted a Howard win, but not the size of the Howard win. Leigh and Wolfers found the ‘different models yielded substantially similar predictions, suggesting on election-eve a Coalition vote share of 51.3-51.7%.’ The actual result was 52.74 per cent.
Leigh and Wolfers found the betting market was the most accurate predictor of the 2004 election outcome. ‘The Coalition was the favourite from July 2003 (when the first election betting market opened) until polling day.’ Furthermore, on election eve, ‘the markets thought it more likely that the Coalition would be re-elected in 2004 than it did in 2001.’
A good paper, and relevant to the odd polling blips we saw earlier this month.