Zero points or nine points?
Whenever two high profile polls come out at the same time I see lots of confused comments asking about who is right, or how come the polls are showing such different things. Today we have the well publicised ICM poll showing Labour and the Conservatives neck-and-neck and the regular YouGov/Sun poll giving Labour a nine point lead. On the face of it these are very different results. How come?
Well, there are two main causes of variation in polls. The first is the differences in methodology between polls from different companies - are they online or telephone, how do they sample, how they prompt, what things they weight by, what they weight to, how they deal with turnout, what they do with people who say don't know? All these things add up and have a party partisan effect, so one pollster may consistently produce figures that are better for Labour, or worse for the Lib Dems or whatever.
The second cause is normal random sample error. It comes from the fact that samples for polls are, to some degree, random. The margin of error is normally quoted as plus or minus three points, meaning that 95% of the time, the figures in the poll will be within 3 percentage points of the "true" figure. Now, in many ways this is a polite fiction - the formula assumes a genuine random sample, which no polls are, and ignore lots of other factors - but I would regard the plus or minus three points as just a good rule of thumb.
If you look at leads rather than shares in polls it is easy to forget just how large that 3 point margin of error is - imagine the real situation in the country was that the Labour party was on 38% and the Conservatives on 30%, and that the pollster conducting the poll had a method that was perfectly accurate. You would still get polls that has Labour varying from 41% to 35% and the Conservatives varying between 27% and 33% - that is, leads of between 2 or 14 points!
Anyway, let's come back to today's polls. ICM have a ZERO point Labour lead, YouGov have a NINE point lead. Let's dissect that and see where the difference comes from.
Looking at YouGov first, because they conduct daily polls we actually have a pretty good handle of YouGov's random error. If levels of support are actually pretty steady, the day-to-day ups-and-downs of YouGov's tracker will mostly be down to normal variation within the margin of error. For example, the average Conservative scores in YouGov polls so far this month is a little over 31%. Below is how each day's Conservative figure has deviated from that score.
So in four polls it was bang on average, in five further days it was one point above or below, on two days it was 2 points above or below. This is all what we would expect to see. We'd get a similar chart for Labour.
On average YouGov's polls this month have been showing a Labour lead of SEVEN points, so I expect the sample variation in today's YouGov poll is a bit on the Labour side, and two points of the difference in leads between ICM and YouGov is sample error in the YouGov poll.
Now let's move onto methodology effects. Earlier this month I produced the chart below showing the different house effects of the main polling companies - the partisan effects each company's methods have on their topline results.
As you can see, on average ICM show Labour leads that are about 3 points lower than those shown by YouGov. That's an average - it too will vary from month to month. The cause is largely down to two differences between the companies polls - ICM weight their data according to how likely people say they are to actually vote. They also make some estimates about how people who say don't know will actually end up voting, assuming that 50% will end up backing the party they did back in 2010. YouGov don't do either of these things.
If we look at ICM's detailled tables here we can see the effect these two approaches had. ICM started out with CON 33%, LAB 38%. After weighting it by how likely people said they were to vote this moved to CON 35%, LAB 37%. After reallocating don't knows it became CON 36%, LAB 36%. There will be other, less tangible differences in methodology, but crudely speaking another five points of the nine point difference came from the different methodologies of ICM and YouGov.
That leaves 2 points difference to account for, which is all likelihood is down to random error in the ICM poll. Whereas we get 20 odd YouGov polls a month, we only get one from ICM so we don't have the data to average out all ICM's polling data this month and see if this particular poll is a bit too Labour or bit too Conservative. But because we can't measure it, doesn't mean sample error isn't still there.
In summary, about half the difference between ICM and YouGov's polls towards is down to methodological differences, about half is down to normal random sample error (YouGov a bit more Lab than usual, ICM probably a bit more Tory than usual). Case closed.
Except, it probably isn't. You're probably still asking "Which poll is right?", is the Labour lead zero or nine? If polls are all over the shop, what possible use are they? The answer is that one single poll taken in isolation isn't that much use, the solution is not to take them in isolation, look at the big broad picture, the underlying trends, the averages over many polls.
And, for the record, Labour have probably got a lead of around about 7 points, like it says up at the top right of the page.
