On small sample sizes
This is a cross-post from the YouGov website here. It's territory I've covered here in the past (and here, and here.) I trust most of my regular readers will know full well that cross-breaks with small sample sizes are not to be trusted. Any news report with comparative results for people of different religions needs particular skepticism, there have been proper polls with parallel representative samples of different religious groups, but a bit of digging does often find it's someone balancing a story on a sub-sample of 8 or something.
- - -
In the small print of opinion polls you'll often find a ‘margin of error’ quoted, normally of plus or minus 3%. This means that 19 times out of 20, the figures in the opinion poll will be within 3% of the ‘true’ answer you'd get if you interviewed the entire population.
A poll of 1,000 people has a margin of error of +/- 3%, a poll of 2,000 people a margin of error of +/- 2%. The smaller the sample, the less precise it is and the wider the margin of error. Strictly speaking, these calculations are based on the assumption that polls are genuine random samples, with every member of the population having an equal chance of being selected. In many cases this isn't true ? polls are carried out by quota sampling, or from panels of volunteers. Even polls done by randomly dialling phone numbers aren't truly random, as the majority of people decline to take part. Even so, the margin of error is still a good rough guide to how precise a poll in, and indeed, when measured against real events like general elections most polls are indeed within the margin of error of the real result.
However, it is important to note that a margin of error applies to the whole sample. All pollsters who are members of the British Polling Council, like YouGov, will publish computer tables showing the detailed results of the poll, which will include crossbreaks breaking down respondents by age, gender, social class, region and other demographics. While these offer great insight into patterns of public opinion, they do, naturally, have smaller sample sizes. For example, a poll of 1000 people will normally have around 500 men and 500 women, and the margins of error on those figures will be around +/- 4%
For smaller demographic groups, sample sizes are even smaller and these bring with them much larger margins of error. For example, a poll of 1000 people would have a margin of error of +/- 3%, but if there were only 100 Scottish respondents within that poll the Scottish figures would have a margin of error of +/- 10%. This means unless the difference between what Scottish respondents said was different to what the rest of the sample said by more than 10 percentage points, it would not be statistically significant. It could just be random error.
The error is particularly common when looking at responses of ethnic minorities or religious minorities in national polls. Britain is an overwhelmingly Christian or secular country, meaning that in any properly representative poll of the British population, only a small percentage of respondents will be Muslim, Hindu or Jewish, and any crossbreaks by religion or ethnicity will be based on very small numbers with very large margins of error.
Many newspapers, for example, have picked up on a recent report by thinktank Demos based on YouGov polling, claiming it shows that Muslim Britains are more patriotic than average. In our poll 83% of British Muslims agreed with the statement that they were proud to be a British citizen, which was indeed higher than the 79% of the general population who agreed. However, this poll contained only 42 Muslim respondents, giving a margin of error of +/- 15% for Muslim respondents. Take into account the margins of error on the poll, and we only can be confident that between 68%-98% of British Muslims are proud to be British, compared to 77-81% of the general population. In other words, it is impossible to draw any firm conclusions about whether Muslim Britain are more proud to be British, less proud to be British, or exactly the same as everyone else.
Remember, it isn't just the sample size of the overall poll that counts, but the sample sizes of the crossbreaks too. It is very rare that crossbreaks of fewer than 50 or 100 respondents will tell you anything reliable or useful.
