The Good, the Bad and the Wildly Wrong Pollsters of the General Election 2024

Yesterday we looked at how well, or rather how badly, we did with the difficult to predict "wildcard" seat micro-forecasts we made for the general election last month. Today we will look at how all the pollsters did in comparison with their macro-forecasts. It is fair to say that the industry collectively did not acquit itself well in this election cycle, Danny Finkelstein in The Times wrote a good piece last month arguing that "Pollsters were so wrong they need a rethink, Labour’s lead was half the 20 points the industry said. It needs to admit it has a problem and try asking new questions." For understandable commercial reasons pollsters are rarely keen on publicly analysing why they are not very good at their job. So unfortunately this is unlikely to happen, nevertheless be in no doubt some pollsters were appallingly bad and their fancy MRP extrapolations were about as useful as astrology. Yet their analyses during the campaign were reported across newspaper front pages.

Some pollsters will hide behind their vote share predictions - which were generally better than their seat predictions - that ignores the whole supposed point of MRP techniques. MRP was the new, new thing and by and large it failed. Below is a table showing the last publicly released prediction of all the pollsters and forecasters we could find:

The actual election result is in the first column on the left. These are the figures the general public looks for on election night before looking at their own constituency result. We have calculated the "aggregate variance" which we define as the sum of the absolute differences (plus or minus) of each party's total forecasted by a pollster. In plain English, "how much was the pollster out by in terms of seats?"

So by this measure Britain Predicts can take a bow, they were the best pollster on this metric out by just 46 seats (7.31%). Savanta can hang their heads in shame, they were out by 210 seats (33.3% of their forecasts) thanks to overestimating Labour's haul by 105 seats and underestimating the Conservative seats total by 68. That is not within the margin of error, it is not a standard deviation or two out, that is a miss by a country mile. Here are the pollsters ranked in order from worst to best:

You will note that UK Polling Report came sixth by this metric - given the amount of abuse we got on Twitter for having a higher projection for Conservative seats than most pollsters it is particularly gratifying that we beat most pollsters precisely because of that higher projection.

Given how so many pollsters are divas there will be objections to this analysis, some will point out how accurate their vote share forecast was or how they did on forecasting actual seat results. When it comes to the granularity of MRP forecasting YouGov did best on that metric calling 481 constituency seats right, Savanta and WeThink did worst calling only 419 seats right. We came in the middle of the pack calling 448 seat outcomes correctly without using MRP. (Source data.)

On a personal note I would like to thank Andy for all the algorithm coding work over the election period and my daughter for the data entry work, editing and compiling. They can be proud that we beat most of the professional pollsters. We will start all over again with the new dataset of MPs soon.