Oz Politics Blog

I found yesterday’s program on opinion polls quite interesting. As a consequence, I thought I would try to capture my rules of thumb on voter intention polling.

1. Opinion polling has a poor record at predicting election outcomes, even six months out from an election (as in, the polls get it wrong almost as often as they get it right)
   • To be fair, voter intention polling does not seek to predict how the nation will vote at the next election, but how it would have voted were the election were held last weekend. A possible exception is Morgan’s question: Who do you think will win the next election?
   • We know from past election studies that roughly half the electorate reports that it decided how to vote during the election campaign
2. Polls are typically volatile from fortnight to fortnight, but in most cases this volatility is ‘noise’ not ’signal’
   • Much of the electorate is largely disconnected from political processes — particularly between elections — and I suspect those who are disconnected typically respond to voting intention polling questions in terms of their loosely evolving preferences and their likes and dislikes, rather than any clearly formed voting intention
   • Also as noted above, some of the electorate is undecided outside of the election campaign period, and in this context some respondents give close to random answers to the polling organisations
3. All polling organisations have systemic biases inherent in their methodologies — these might be attached to sampling design, engagement modality (eg. face-to-face or telephone), question design, question order, answer prompting, the way in which the two-party preferred result is derived (by direct question or preference flows at the last election), or some other aspect of the polling process.
   • For example, I have noted for some time that the Morgan polls are typically to the left of Newspolls conducted on the same date (or is it that Newspolls are typically to the right of Morgan)
4. Individual opinion polls are not an accurate measure of public opinion
   • The consequence of the above factors (and others) is that the stated statistical error margin for any poll overstates its precision by orders of magnitude — Andrew Leigh and Justin Wolfers have estimated that “the true standard error of the polls is equivalent to a poll of 25 voters that suffered only from sampling error.” This is much more error than would be expected from the 2000 voters in a typical Morgan Poll or the 1400 voters in a typical ACNielsen poll.
5. Therefore, polls must be interpreted
   • Individual opinion polls must be considered in context: within the time series, in comparison with polls from other pollsters, against the backdrop of political events, and against other information sources (eg. betting markets and qualitative polling)
   • I typically reject as noise any significant poll movement that can not be attributed to an event that one would reasonably expect to change voting intention
   • Even if I can point to a plausible explanatory variable, I usually suspend judgment on a single poll result that could mark a discontinuity
   • To put it another way: don’t read too much into a single poll result
   • I look to the medium term trends in the polling, and I usually give precedence to the the moving average over (say) two months, rather than the individual fortnight on fortnight poll results
   • When considering a two-party-preferred (TPP) prediction, I look at how the preference flows were calculated — were they based on a direct question or preference flows at the previous election. If based on a question, I test the plausibility of preferene flows against the range of preference flows from previous elections
   • While I generally accept the direction of an extreme poll trend (as an indicator of the current mood, not as a prediction of the next election result), I usually have doubts about the scale of TPP vote predictions that are outside of the range of election experiences since the second world war (ie. any Federal poll that predicts a winning margin wider than 57-43)
   • A TPP moving average in the 49 to 51 per cent range can be particularly difficult to interpret
6. The media has a vested interest in sensationalising the noise in polling sequences — a dramatic swing to a party one fortnight is headline news, and in the followed fortnight the dramatic swing back to the other party is also headline news
   • My pet aversion in this area is the commentators who judge the Federal Budget by the following weekend’s poll — it is simply crap analysis — most punters do not analyse a Federal Budget between Tuesday and Friday and then adjust their voting intention in time for the pollster on Saturday — for most people, if they do adjust their voting preference, it is when they are negatively impacted by a Budget measure once it has been implemented

Are there other rules of thumb I should mention? Any you disagree with?

Update: an observation from Andrew Norton.

Andrew Leigh and Amy King have written a cracker of a paper on the benefits of being first on the ballot paper.

Their conclusion: Men gain 1.4 per cent of the total vote share when placed first on the ballot paper. Women gain no advantage. “About 1 in 70 voters will support the first candidate on the ballot paper, but only if that candidate is a man.”

Andrew’s blog on the paper is here.

For any given weekend since the last Federal election, it was not uncommon for ACNielsen and Newspoll to have estimates for the Greens that were statistically different. In plain English: at least one of the pollsters was wrong! This can be seen graphically.

This of course raises the question of systemic polling bias. To get a sense of the systemic differences between the pollsters, I have put together the following table. It gives the average poll results from the pollsters since the last election for each of the major party groups.

I am not convinced these results are necessarily meaningful. Morgan, for example, tended to report weekly on reduced sample sizes when it got anomalous results, which could skew its results in this analysis. For ACNielsen, I have excluded the results from June 2006, as its minor party primary vote prediction has not been made public yet.

Nonetheless, the results are indicative of a systemic polling bias problem - particularly with the Greens. I suspect this bias comes about from two sources. First, the way in which the pollsters develop their sample frame can bias the result. Second, the actual questions the pollsters ask and the sequence in which they are asked can also bias the prediction.

It is a significant issue, as these Green and minor party results feed into national two-party preferred estimate from each of the pollsters. From 2007, I expect Newspoll will report preference allocations according to the way in which people say they will vote, rather than use the preference flows from the previous election. Similarly, while Morgan reports both, I expect the focus to move to actual reported preference flows.

It is another useful reminder that interpreting polling results is as much an art, as it is a science. And it legitimises the practice of adjusting results from some polling organisations by a percentage point or two; although it is arguable which pollster is accurate and which is biased. For the record, I cannot quite come at the seven-point adjustment one of my regular commentators applies.

Note: in undertaking this analysis, I found an error in my earlier minor party charts in respect of the Morgan series. It has now been corrected. You may need to hit the refresh button on your browser to get the latest chart.

The Australian Electoral Commission has released an information paper on compulsory voting. It is an interesting overview of the history of compulsory voting in Australia, and the arguments for and against compulsory voting.

This may become a live issue. After each federal election, the Joint Standing Committee on Electoral Matters (JSCEM) conducts an inquiry into the election and considers public submissions. The 1996 JSCEM recommended that compulsory voting should be repealed. The 2004 JSCEM recommended that a full and separate inquiry be held into voluntary and compulsory voting. While the Prime Minister has said that the abolition of compulsory voting will not occur before the next federal lection there are a number of members of the government, including influential ministers such as Senator Nick Minchin, in favour of voluntary voting.

The Parliamentary Library has released a whole bunch of papers. The psephologically inclined might want to get their fangs into:

• Bills Digest for the Electoral and Referendum Amendment (Electoral Integrity and Other Measures) Bill 2005
• Research Note on the 2006 Canadian Election
• Research Brief on Electoral Systems
• Research Note on the 41st Parliament: middle-aged, well-educated and (mostly) male