YouGov methodology tweaks

YouGov have been dominating the polling in the pre-election period since the daily polling began, especially since the other companies still seem to be on their "peacetime footing". At some point many of them will also step up the frequency of their polling as we get closer to the election, but for now YouGov is getting the attention, and two parts in particular of their methodology have been provoking a lot of questions.

Weighting Labour identifiers

Shortly after starting the daily polling YouGov made a slight change to their political weighting. Previously their political weighting was just on party ID. It's important to note that party ID is NOT the same as past vote: some people identify themselves with one party, but abstain or vote for someone else - perhaps tactically, perhaps as a protest (e.g. "I'm Labour through and through but I'm voting for the Lib Dems because of Iraq"). Nor is it the same as voting intention....

In 2005 only 72% of people who said they identified with the Labour party said they actually voted for the Labour party. 13% voted Lib Dem, 9% stayed at home, 6% voted for other parties (this was mostly a Labour phenomenon, the overwhelming majority of people who identified with the Lib Dems or Conservatives also voted for them).

To take account of this YouGov have always sampled loyal Labour ID (people who identify with Labour and voted for Labour in 2005) and disloyal Labour ID (people who identify with Labour, but did not vote for them in 2005) separately. In the past, they never felt the need to also weight by it - the other weightings were enough to keep it in broadly the correct proportion.

The switch came about on the back of the testing of daily polling before its launch. One day produced an outlying result and further investigation showed that the cause seemed to be too many "Loyal Labour ID" respondents. To prevent that happening again, we decided to start weighting the loyal and disloyal Labour ID respondents separately as an extra control on the representativeness of the sample.

Naturally before we actually did it on any published polls it was heavily parallel tested. We reweighted three weeks worth of polling using both the new and old weights, and in almost every case it made no difference at all to the topline figures (in most cases the changes were in the region of 0.1%). So it doesn't make the results more Labour or more Conservative, all it does it make things a tiny bit more stable.

Degree of weighting

The other issue that has aroused comment is the amount of weighting YouGov do to their samples. As I've said several times before, this doesn't actually matter that much when it comes to final results - what is important is what the demographics of the sample are AFTER weighting, is that representative? Having to do more weighting does reduce the effective sample size of a survey, so risks making it more volatile, but that's about it - and so far YouGov's daily polling has not been unduly volatile.

However, for those who are interested the YouGov figures in the last few weeks have needed more weighting than in previous months. The main reason is the shift in the way YouGov sample. People used to be invited to specific surveys - people got an email inviting them to do a survey about, say, fish, they followed the link and did the survey about fish. If the fish survey was already closed, they got sent away. This allowed YouGov to invite samples that were pretty accurate demographically to begin with and reduce the need for weighting afterwards, though some weighting was still needed because you got different response rates from different groups. This was a plus, but it also had disadvantages - you could end up with too many respondents, or panellists could end up being sent away after taking the trouble to respond to an email. You also couldn't do very accurate one day surveys since "fast responders" were different to "slow respondents".

So last autumn YouGov switched to a new system. Now people are sent a non-specific invite and, when they arrive at the YouGov system, they are allocated to whichever survey needs someone in their demographic group (for example, were I to arrive at the site the system would look at all the open surveys and see which one's quotas were most in need of a 25-40, middle-class, Times reading man from the South East, and send me over there). What this means is that surveys don't get excess replies, respondents never get sent away empty handed, and that even very fast surveys get an even mix of fast and slow respondents, allowing accurate one day surveys.

However, because you can't tell who is going to respond to which survey, it also makes it more difficult to calculate the proportions of people to invite to get a sample that needs the least possible amount of weighting - hence the slightly higher levels of weighting (though it's worth saying they remain relatively low compared to some of the weights needed for quasi-random samples).

They'll come down over time as the panel team get used to the patterns of response amongst different groups (looking at recent results, for example, we could probably do with inviting fewer men over 55 - too many are responding!) but in the meantime the effect upon polls should be just to make them a tiny bit more volatile.