Oz Politics Blog

Simon Jackman has looked at the two key problems with opinion polls — sampling error and systemic bias — in the context of the 2004 election. His ’shock-horror’ conclusion is that both Morgan and Newspoll opinion polls demonstrated a systemic pro-Labor bias in the lead up to the 2004 election. More about that later, we will begin with an overview of Jackman’s paper.

Jackman began with an excellent introductory treatment of error margins and statistical testing to see whether the movement between one poll and the next is significant.

the sample sizes used by the published media polls (e.g., the 1,400 respondents typically seen in the Nielsen polls for the Fairfax papers) have a reasonable chance of detecting moderate to large swings in voter support. But the probability of detecting a small swing, say, on the order of a percentage point, is less than one in ten, given the sample sizes used by most commercial polls.

That is to say, there are more than nine chances in ten that any one percentage point move in the opinion polls, from one poll to the next, is nothing more than meaningless random noise. With most media commissioned opinion polls, a swing of less than 3 per cent between two polls is unlikely to be meaningful.

To address this problem of smallish sample sizes, Jackman looked at the potential to pool opinion poll results over time. By pooling two or more polls, the precision of the population estimate is increased. However, there is a danger here. If two polls had the same systemic bias (to the Coalition or Labor, it does not matter), the expected gain in precision could be entirely misleading.

Jackman sought to estimate the systemic bias of five polling organisations (Newspoll, ACNielsen, Roy Morgan, Galaxy, and the ANU/NineMSN internet poll) by calibrating their polls in the four months prior to the election with the election result. The analysis found,

The ANU on-line poll has the largest bias, under-estimating coalition support by 5.6 percentage points (on average) … Morgan has the next largest bias in these data, resulting in an under-estimate of coalition support of 4.7 percentage points. Finally, Newspoll is estimated to have 2.7 percentage point bias towards Labor that is statistically significant at the conventional 95% level. Recall that the pre-election Newspoll estimated coalition vote share at 50%, based on a large sample of 2,500 respondents; the disparity between this poll and the actual election result almost exactly mirrors the bias estimate, as it does for the other polling organizations.

Galaxy and Nielsen are estimated to have negligible biases (in the sense that their bias parameters can not be distinguished from zero). Again, these negligible bias estimates reflect the fact that Galaxy and Nielsen’s final poll estimates were quite close to the actual election outcome.

Jackman’s paper also presented a Bayesian statistical model for pooling results in a way that eliminated the effects of systemic house bias (the ’smoothing’ referred to in the paper’s title). This model depended on the hindsight of the election results, and therefore cannot be used predictively. Nonetheless, in an historical sense it is interesting to speculate at which points in the campaign the punters decided to vote the way they did.

Daily estimates of the Coalition’s share of 2PP vote intentions are shown in Figure 4 [reproduced below]. The solid line connects the 114 daily estimates of t , and the shaded region indicates a 95% confidence interval. As in Figure 3, campaign events are indicated with dotted vertical lines, and actual polls are overlaid with plotted points for point estimates and vertical bars indicating 95% confidence intervals…

the estimates reported here indicate that the coalition’s gains over the campaign amounted to about two percentage points, almost the size of the overall swing to the government in the 2004 election, and that they came relatively early in the campaign, as events such as the Jakarta embassy bombing and the leaders’ debate took place. The formal policy launches of both major parties seem to have made no impact on the overall breakdown of vote intentions.

This chart also demonstrates the pro-Labor systemic bias of three of the five the polling organisations in the lead-up to the 2004 election, with most published opinion polls falling on the Labor side of Jackman’s population estimate.

Although not highlighted on the chart, the dip in mid August suggests the potential grief the Mike Scrafton affair could have caused the Government if Senator Brandis had not found imperfections in Scrafton’s memory. The betting odds were also weakest for Howard around this time.