Oz Politics Blog

I copped some flack yesterday for suggesting that the Morgan poll has a two percentage point left bias.

Now we all know there are commentators on this blog who routinely accuse Morgan of substantial biases. Those commentators typically go on to apply a correction formula derived from Colonel Sanders’ 11 secret herbs and spices and then they declare the Morgan poll guarantees the triumphant return of the Howard government for a fifth glorious term. So there can be no mistake: that is not my claim.

Nonetheless, I think there is a left-bias in Morgan that needs to be at least considered when interpreting Morgan polls. It would be reasonable for you to ask: What is the evidence for my claim of a Morgan bias? I have two lines of evidence.

The first is Simon Jackman’s academic research into the polls following the 2004 election. I previously blogged on Simon Jackman’s preliminary research. The preliminary paper no longer seems to be on the web — however, it looks like the research has been finalised and published as “Pooling the Polls Over an Election Campaign”, Australian Journal of Political Science, 2005, V40(4):499-517. There is a related slide presentation here. Jackman found that in the four months prior to the 2004 election Morgan typically under-estimated coalition support by 4.9 percentage points. To be fair to Morgan, the pollster changed its “two-party preferred” methodology following the 2004 election.

Second, notwithstanding Morgan’s changed methodology, since the 2004 election Morgan has typically predicted a Labor primary vote that was around two percentage points higher than ACNielsen and Newspoll for the same period. This tendency is clear in the next graph of the Labor primary vote moving averages for Morgan, Newspoll and ACNielsen.

With the Coalition primary vote predictions, a corresponding approximate two point bias is evident between Morgan and Newspoll. However, The relationship between ACNielsen and the other two is a little more complicated.

The two-party preferred (TPP) vote share predictions are a little more difficult to interpret, compared with the primary vote predictions for the major parties. There are two complicating factors.

First, the pollsters use different methodologies to calculate their TPP predictions. Newspoll and Morgan currently use the same methodology, which is based on the preference flows at the last election. ACNielsen calculates its TPP vote share prediction using responses to the poll questions.

The second problem is the pollsters get consistently divergent primary vote estimates for the non-Labor, non-Coalition parties. This divergence can be seen in the next two moving average graphs.

The consistent divergence in the Green primary vote predictions is of particular concern.

Notwithstanding the complexity of various inputs and methodologies, the outcome is that since 2004 both Morgan and ACNielsen have been tracking to the left of Newspoll with their published TPP vote share predictions. In the following graph I have plotted two moving averages for ACNielsen. The first — labelled “published” — is based on ACNielsen’s published TPP vote share predictions. The second — labelled “adjusted” — is calculated using the preference flows from the last election. When the ACNielsen TPP vote share is adjusted, it tracks closer to Newspoll than Morgan.

I accept it is arguable that the predicted TPP vote share divergence is a rightish bias in Newspoll rather than a leftish bias in Morgan. I prefer the left bias story on the basis of Jackman’s research into the 2004 election. I also prefer the left bias story because it is explainable. I suspect the leftish bias in Morgan is possibly an artifact of Morgan’s cluster sampling frame and the leftish bias in ACNielsen is possibly an artifact of ACNielsen’s TPP calculation methodology,

In conclusion, I think it not unreasonable to at least consider applying something like a two percentage point adjustment to Morgan’s TPP vote share and Labor primary vote predictions when interpreting Morgan poll results. This is consistent with Jackman’s observation that one might apply an a priori correction to polls based on their actual performance in the lead-up to the previous election. In Mogan’s case, that correction should recognise Morgan has changed the way in which it calculates its published TPP vote share prediction.

Note: I have not commented on what biases may or may not be evident in the Galaxy series, largely because I do not have enough polling data on which to form a view. Also note: I have used a three period moving average for ACNielsen, as it is published monthly, and a six period moving average for Newspoll and Morgan, which were being published on a fortnightly basis. This year, Morgan has been published weekly.

Update: Possums Pollytics has also looked at the Morgan bias question.

I have resisted the 1996 versus 2007 election comparison for some time; largely because they feel like very different elections to me. My other comparisons with 2001 and 2004 were to test whether Howard would repeat his earlier performances and “come from behind to win” in 2007. I thought there was not much to learn from a 1996 -v- 2007 comparison. As it turns out, I was wrong. However, the big lesson was not in the similarities (which are superficial), but in the underlying differences with the attitudinal polling.

I will come to that lesson shortly. First, let’s look at the similarities.

When you consider the primary voting intention polling in the lead up to the 2 March 1996 election and compare it with the lead-up to a hypothetical 15 December 2007 election, there are strong similarities. Both opposition leaders were installed around one year out from the election date. John Howard, in 2007, is polling close to but a bit below Paul Keating in 1995-1996. Keven Rudd, in 2007, is polling close to but a bit better than John Howard in 1995-1996. This can be seen graphically.

Notwithstanding these similarities, the attitudinal polling tells a very different story, and suggests a radically different dynamic is at work in the two election campaigns. Let’s begin with the satisfaction ratings.

John Howard’s satisfaction rating in 1995-1996 is very similar to his satisfaction rating in 2006-2007 thus far. The big difference is in Rudd’s satisfaction rating, which is much higher than Keating’s for the same period.

With the dissatisfaction ratings, Howard’s is around 10 points higher in 2006-2007 than it was in 1996-1997. But the big difference is between Rudd and Keating. Rudd is some 40 to 50 points ahead of Keating, in the lead up to this year’s election.

These satisfaction and dissatisfaction ratings can be combined into net satisfaction (ie. satisfaction minus dissatisfaction) ratings as follows.

It is in the net satisfaction ratings that we see the big difference between 1996 and 2007.

In 1996, John Howard was essentially neutral with the electorate in net-satisfaction terms. Yet in 1996 Keating was a big negative. The 1996 election was not so much about buying Howard, it was selling Keating. Consequently, I am going to categorise the 1996 election as a selling or disposal election.

In 2007, John Howard is still essentially neutral with the electorate in satisfaction terms. But Rudd is a significant positive with the electorate. This suggests that 2007 is shaping up as a buyer’s election. The electorate looks like it might just be in the mood to buy a new prime minister (rather than sell the old one). If there is a change of government in 2007, the motivating factor in the trade will be the purchase and not the sale. For the sake of the argument, I will categorise 2007 (as it looks so far) as a buying election.

This is a new and different dynamic in comparison with 1996. While the 1996 election may have been about taking to Paul Keating with a baseball bat, the 2007 election campaign (so far) is not about baseball bats. If the motifs for 1996 were hate and vengeance, in 2007 the motifs are love and attraction. Coalition supporters who take comfort from the absence of a baseball bat have missed the new dynamic at work.

If I can overload the metaphor, commentators should focus on the pick-up and not the “ex-” being dropped. They should focus on the exchanges of flowers and chocolates, and not the absence of baseball bats.

The scale of the new dynamic can be seen by subtracting the net satisfaction rating for the opposition leader from the prime minister’s net satisfaction rating. I have called this value the bias to the prime minister in satisfaction. In both the 1996 and 2007 campaign it was and is negative. That is to say, the satisfaction bias ran in the direction of the opposition leader. In 2007, the satisfaction bias against the prime minister is much starker than it was at this point in the 1996 election campaign.

There is one final graph I want to look at, largely to note that it offers little explanatory power. It relates to the polling question on who would make the better prime minister. That question favoured Howard in the first eight months prior to the 1996 election and Keating in the last four. But it had no impact on the primary voting intention. I suggest we ignore the question this year as well.

Interesting question: if I have called the new dynamic correctly, what can the Coalition do to woo back the electorate?

• Should the Coalition simply do nothing and wait for the wayward electorate to return to its first love?
• Is it a fear campaign along the lines of, “be careful with Kevin, you know he has the pox”?
• Does John show some more of his policy leg in an effort to outshine Kevin’s allure? Or,
• Should the Coalition opt for the complete makeover with Peter Costello as the new suitor?

Or is it all beyond repair with the electorate saying, “it’s over John; but don’t blame yourself — this is not about you, it’s about me”?

If I have it right, Newspoll is out on Tuesday, there is every chance that Morgan will publish a one weekend poll on Friday, and ACNielsen is out next Monday. When the polls come out, the papers will either say there has been a post Budget bounce for the government or there has not. Both headlines would be fatuous. Both analyses would be flawed.

It is nigh on impossible to assess a Federal Budget by a single poll on the following weekend. Anybody who attempts it either does not understand voter intention polling, or is deliberately setting out to deceive you. My advice is: don’t get sucked in. Don’t fall for the pop-psephology that will attribute a move (or the absence of a move) from one poll to the next to this event or that. It is rarely true.

To begin with, most voters do not analyse a Federal Budget between Tuesday night and Friday and then adjust their voting intention in time for the pollster on Saturday. The reality is that most voters are disconnected from the political process most of the time. Budget week is no different.

The real problem with voting intention polls are the stochastic perturbations (random noise) associated with polling. In practice, voting intention polls have a much larger margin of error than that predicted by statistical theory. For error purposes, a sample of 1000 voters would be better treated as if it were only a sample of 100 people and perhaps smaller. Andrew Leigh and Justin Wolfers have estimated that “the true standard error of the polls is equivalent to a poll of 25 voters that suffered only from sampling error”. As a consequence, any single poll result is fairly likely to fall almost randomly within plus or minus three percentage points of the actual population parameter.

With a high noise factor and a low underlying trend, if you take all possible consecutive poll findings, according to their probability of occurrence, it is entirely possible that the random noise would suggest a trend that does not exist in the population. Take, for example, the following Newspoll data from 2001 from March to immediately before the Tampa episode at the end of August. The trend over the period was a fall of 0.4 percentage points per fortnight in Labor’s two party preferred vote prediction. Of the 13 data pairs during the period, only four pairs had a declining trend of one per cent. Four of the data pairs suggested a dramatic improvement in Labor’s fortunes and five suggested a dramatic decline. None of the data pairs suggested the most accurate fortnightly trend prediction of zero (when you consider that Newspoll publishes it results in integer percentage points).

In summary, none of the consecutive data pairs gave the most accurate trend result possible. Only 30 per cent of the data pairs for this period gave a reasonable prediction of Labor’s fortunes. But similarly, 30 per cent of the pairs yielded a trend line in the opposite direction to the underlying trend in the population. 40 per cent of pairs over-estimated Labor’s demise in 2001. All up. 70 per cent of the apparent trends from the consecutive data pairs were misleading.

Whatever the papers say about a post Budget bounce, don’t believe it. The media pundits cannot reliably interpret two consecutive polls when the underlying population trends are likely to be small, and the random noise range associated with the polls is a number of multiples of the underlying trend. No-one can.

Please note: I am not arguing that the post Budget polls would be meaningless. All I am arguing is that individual voter intention poll results must be considered in context: within the time series, in comparison with polls from other pollsters, against the backdrop of political events, and against other information sources (eg. betting markets and qualitative polling). You will need a number of polls following the Budget to come to an informed view about how the government is travelling post budget, and whether that is different from how it was traveling prior to the budget. Unfortunately, you only need one post-Budget poll to come to an uninformed view on the efficacy of the Budget.

To counter the influence of random noise, I tend to place more faith in the moving average, than any particular poll result. The six poll moving average I use with Newspoll and Morgan, and the three poll average I use for ACNielsen, are designed to cancel out some of the random noise. It is not a perfect solution, but it is far better than over interpreting an individual poll result.

Two of the three moving averages suggest that the trend to Rudd since December 2006 has come to an end. It is impossible to tell whether the polls will plateau at this point, guaranteeing a Labor landslide; move slowly back to the government, giving Labor a narrow to comfortable victory; or move decisively back to the government and a fifth Howard term.

Links: Previously I have posted on my rules of thumb for interpreting polls.

You may not have noticed it, but a fortnight ago Royal Assent was given to the Electoral and Referendum Legislation Amendment Act 2007. After the substantial and sometimes controversial changes from the 2006 amendments to the electoral laws, this Act makes only a few minor and uncontroversial changes in respect of:

• electronic voting trials for the visually impaired and the Australia Defence Force; various changes to postal voting;
• electoral enrolment by Australians who are overseas (including Defence Force personnel);
• pre-poll voting arrangements; and
• defamation of candidates.

The Poll Bludger should be pleased as this Act repealed section 350 of the Commonwealth Electoral Act 1918. The old section 350 provided criminal penalties for defamation against electoral candidates. A candidate dissatisfied with the Poll Bludger’s (most excellent) electorate guide initiated a private prosecution against its author, William Bowe, for an alleged breach of section 350.

Today I was asked for my views on the relative merits of opinion polls versus betting markets as a predictive tool on election outcomes. I answered along the following lines.

If the polls and the betting market are giving different messages, I would tend to favour the betting market over the polls when it comes to making a prediction about the next election. It is not an iron-clad rule, just a strong preference.

Voter intention polls do not seek to predict the next election outcome. They seek to predict how the nation would have voted were an election held on the day of the opinion poll. In effect, they report the current state of play between the rival parties.

When polling is used to predict an election outcome, we make the assumption that nothing significant will change between now and the election (or that the current trend line will continue unabated). It is an heroic assumption. We all know the aphorism, “a week is a long time in politics”. Things can and do change.

In addition, some have suggested that a sizable proportion of the population is politically disengaged between elections, and they respond to polls in terms of their evolving preferences and reactions to news and events, rather than with a considered voting intention.

In practice, voter intention polls used predictively are less reliable the further out from an election they are. Even six months out, they are not particularly reliable.

Their reliability, even at close quarters, is further compromised by the high volatility in the predictions from polling organisations. The polls typically jump around from one report to the next. It is very easy to read too much into a particular poll, or how one poll differs from the previous poll. They have a high noise to signal ratio.

When individuals bet on the next election outcome, they are putting their money on a particular outcome. The bookmaker is constantly adjusting the odds so that no matter who wins the election, the contingent payout to the punters is less than the total amount bet. This way, the bookmaker should always make a profit (of course they can be caught by a large very late bet). Because the bookmaker is constantly the adjusting the odds, the odds reflect the weight of moneyed opinion for each of the outcomes. It is quite a simple arithmetic procedure to convert these odds to probabilities.

Over recent years, betting markets have typically predicted election outcomes earlier and more consistently than voter intention polls, with much less volatility.

But betting markets are not perfect. For example, the punter’s favourite did not win the 2005 New Zealand election. Although to be fair, immediately prior to the election the betting market predicted that NZ Labour had 9 chances in 20 of winning the election and the Nationals 11 chances in 20. In probability terms, that is pretty close to neck and neck. The final result — NZ Labour — was not that unlikely an outcome from the betting market’s perspective, as it had given them 9 chances in 20 of winning.

If the Coalition wins in NSW this Saturday, it would count as an election the betting market did not predict. It would also be an election the polls did not predict. Somehow, I think it unlikely.