This week's Polling Inquiry
On Tuesday the BPC/MRS's inquiry into why the polls went wrong publishes its first findings. Here's what you need to know in advance.
The main thing the inquiry is looking at is why the polls were wrong. There are, essentially, three broad categories of problems that could have happened. First, there could have been a late swing - the polls could actually have been perfectly accurate at the time, but people changed their minds. Secondly respondents could have given inaccurate answers - people could have said they'd vote and not done so, said they'd vote Labour, but actually voted Tory and so on. Thirdly the samples themselves could have been wrong - people responding to polls were honest and didn't change their minds, but the pollsters were interviewing the wrong mix of people to begin with.
Some potential problems can straddle those groups. For example, polls could be wrong because of turnout, but that could be because pollsters incorrectly identified which people would vote or because polls interviewed people who are too likely to vote (or a combination of the two). You end up with the same result, but the root causes are different and the solutions would be different.
Last year the BPC held a meeting at which the pollsters gave their initial thoughts on what went wrong. I wrote about it here, and the actual presentations from the pollsters are online here. Since then YouGov have also published a report (writeup, report), the BES team have published their thoughts based on the BES data (write up, report) and last week John Curtice also published his thoughts.
The most common theme through all these reports so far is that sampling is to blame. Late swing has been dismissed as a major cause by most of those who've looked at the data. Respondents giving inaccurate answers doesn't look like it will be major factor in terms of who people will vote for (it's hard to prove anyway, unless people suddenly start be honest after the event, but what evidence there is doesn't seem to back it up), but could potentially be a contributory factor in how well people reported if they would vote. The major factor though looks likely to be sampling - pollsters interviewing people who are too easy to reach, too interested in politics and engaged with the political process and - consequently - getting the differential turnout between young and old wrong.
Because of the very different approaches pollsters use I doubt the inquiry will be overly prescriptive in terms of recommended solutions. I doubt they'll say pollsters should all use one method, and the solutions for online polls may not be the same as the solutions for telephone polls. Assuming the report comes down to something around the polls getting it wrong because they had samples made up of people who were too easily contactable and too politically engaged and likely to vote, I see two broad approaches to getting it right. One is to change the sampling and weighting in way that gets more unengaged people, perhaps ringing people back more in phone polls, or putting some measure of political attention or engagement in sampling and weighting schemes. The other is to use post-collection filters, weights or models to get to a more realistic pattern of turnout. We shall see what the inquiry comes up with as the cause, and how far they go in recommending specific solutions.
While the central plank of the inquiry will presumably what went wrong, there were other tasks within the inquiry's terms of reference. They were also asked to look at the issue of "herding" - that is, pollsters artificially producing figures that are too close to one another. Too some degree a certain amount of convergence is natural in the run up to an election given that some of the causes of the difference between pollsters are different ways treating things like don't knows. As the public make their minds up, these will cause less of a difference (e.g. if one difference between two pollsters is how they deal with don't knows, it will make more of a difference when 20% of people say don't know than when 10% do). I think there may also be a certain sort of ratchet effect - pollsters are only human, and perhaps we scrutinise our methods more if we're showing something different from anyone else. The question for the inquiry is if there was anything more than that? Any deliberate fingers on the scales to make their polls match?
Finally the inquiry have been asked about how polls were communicated to the commentariat and the public, what sort of information is provided and guidance given to how they should be understood and reported. Depending on what the inquiry find and recommend, in terms of how polls are released and reported in the future this area could actually be quite important. Again, we shall see what they come up with.
