Why Does Everyone Disagree on MRP?

It will probably be worth all the polling companies who conduct MRP (all listed helpfully here by Peter Ingelsby) coming together to do a post mortem after the event to diagnose the cause of the large spread / range of forecasts. This is separate, but related to the range of vote share forecasts that currently operate between an even wider variety of polling companies.

Currently the MRP forecasts range from More in Common’s projection of 380 Labour seats vs 180 Conservative seats to Electoral Calculus’ 450 Labour seats vs just 60 for the Conservatives in a clear third place. This is an enormous spread. What’s the cause? And who is right? There are some clear answers to the first question, but we fundamentally won’t know the answer to the question until the exit poll drops at 10 pm on 4 July.

There are fundamentally six reasons why the forecasts are different

The “slope” of gradient of the Conservative party’s vote share is not known - because national elections in which the incumbent drops c.20% are very rare. The “slope” here means basically how a party’s vote share is distributed across the country. Not all distributions are equal. If the Conservatives only suffer from uniform national swing, it would mean that c.20% vote points would be taken off each seat from the 2019 vote share on average to get an estimate of Conservative voting intention by seat. That is of course impossible as 69 seats have a 2019 vote share of less than 20%. Proportional swing would imply taking the ratio of the Conservative 2019 vote (44.5% for GB) and today (say 22.25% for argument’s sake) =  multiplying out by 0.5x roughly the vote share from 2019 by seat. Basically this means you lose more votes in safe seats (70% seat turning into 35%) than a unsafe seat (20% turning in 10%). Proportional swing is devastating as it shifts almost all safe seats into marginals, whereas uniform implies there are seats that can never really shift hands. Unhelpfully in British history swing can be both uniform, mix of uniform and proportional or just proportional as you can see below. Also helpful/unhelpfully Professor Stephen Fisher has noted there are examples of different types of swing in big change elections across different democracies. For what it’s worth, the swing at GE1997 was UNS/proportional mix, the recent locals were UNS/proportional mix. But at the locals when the Reform party was standing it was basically proportional. If the conservative vote share change is between Uniform National swing and proportional they’ll end up with a seat count around 120-160.  proportional 90-120ish. Beyond proportional means below 90 although the way down to 50s.

Second most important reason the forecasts all differ is that when the conservative vote share sits between 20-25% it is uniquely sensitive to pollsters differing voting intention that is powering through the models. Analyst James Blagden has neatly summed this up as “snakes and ladders”. In other words we are in the “death valley” of forecasting. Couple of points on the Conservative vote share, or away from Labour uniquely switch more seats than would typically be the case in an election

Thirdly is the issue of timing. MRP models use data that can range from days to weeks. It can very often be backward looking to a greater degree than usual polls. That’s because MRP models typically need samples north of 15,000 which can take a large amount of time to collect. This matters in 2024 because a lot has happened. Nigel Farage has returned and boosted the Reform party (which may subside post Ukraine comments). Rishi Sunak’s D-Day gaffe, and “gamble-gate” could all materially affect  voting intention numbers

Fourthly, different companies have different samples flowing through their models. We’ve seen an explosion of modelers and of sample providers. This has the potential to have exacerbated and magnified the differences in seat forecasts. Some MRP providers (like Survation) have phone polling through their models. Other forecasts (like Electoral Calculus’) use non-traditional sources such as FindoutNow for sampling. Others (such as Ipsos’ Verivan division) use random probability samples (meaning you are asked, based on heavy analysis of what makes up a perfect, randomly stratified sample)

Fifthly MRP models have different turnout assumptions. Much of these assumptions are opaque. At Focaldata - the company where I am Chief Research Officer - we trained our turnout model on the 2015,2017 and 2019 elections. There is no guarantee that either this is right, or that others are right or wrong

Sixthly, the actual specifications of both the multi-level modeling and the stratification frame can be quite different between providers. These are not quasi-Census databases. There are statistical models which bake hundreds of micro-analytical and political choices in modeling.

Lastly, there are eight existential questions facing MRP - with no easy answers:

1. Is MRP there to get the aggregate picture of seat counts right or individual quirky seats that you would never expect? What’s the trade off between the two?
2. If there are sampling biases (such as the Spiral of Silence)does MRP actually magnify such potential sources of error?
3. Is MRP really a good tool to predict smaller parties where their potential gains are limited, and vote distributions highly unusual; and not necessarily driven by demographics?
4. Should MRP providers offer first past the post forecasts or probabilistic ones? Which should be favoured?
5. What model specifications should be made available to the public? Does that help the public or in fact erode competitive advantage?
6. How should the media communicate the findings and uncertainty? For example the probabilistic seat forecasts available here indicate that in many seats the confidence bands (correct 90% of the time) can be as wide as 10-15 pts)
7. What’s the best way to understand the seats that nobody seems to agree on?

Are MRP results more accurate at the beginning of a campaign (like 2019) or later? Does data quality irrevocably get worse as campaigns go on?

It’s clear the UK eco-system is probably in need of a “meta-aggregator” along the lines of the US’s 538 that can parse, rate pollsters and provide a meta-aggregation. This may in time help the media parse all of these complicated, conflicting, and high frequency of results

Additionally - unrelated to MRP - is the issue that others such as Matt Singh have pointed out from the small amount of seat polling he has conducted, is that there a lot of Don’t Knows, and that they are located very heavily in Conservative leaning areas, and are very disproportionate 2019 Conservatives. Should these voters return disproportionately to the conservatives in the coming days this could materially change the outcome of the race.

Okay so what’s going to be the result?

I genuinely think it is acceptable to be comfortable with not being surprised by a Conservative seat count as low as 60 - and in third place - and a Conservative seat count at 160 (or even higher). This election was always a toggle between whether the result would be like 1997 or 1993 like Canada. We are none the wiser heading into it, but that’s okay. Totally valid, excellent pollsters have different projections and have made those projections in good faith.

For what it’s worth I think the result will be somewhere in the middle - and the degree to which “gamble-gate” dissipates or doesn't and Farage peaks or doesn't will determine whether this is 1997 mark two, or something much more dramatic.