Polling: science and art
September 21, 2016 Leave a comment
Wow. So loved this feature in the Upshot yesterday that so clearly demonstrates how much of polling is not science, but also art. Yes, there’s plenty of science when it comes to accurately obtaining a truly random sample and using statistical inference based on the science of that sample, but the art can be just as, if not more important, in determining the final results.
Basically, your random sample is never going to perfectly match the demographics of the population about which you are drawing inferences (e.g., you only care about 700 NC voters insofar as they can help you infer the opinions of the millions of actual NC voters). Now, if you just want “public opinion” this is easy. If your sample is only 12% Black but the Census data says 15% Black, you just weight accordingly. But elections are different. Nobody actually knows what the percentage of Hispanic voters or under-3o voters will be on election day. All we can do is make informed guesses and weight to those guesses. But, they are guesses. And there’s often quite defensible reasons for choosing 14% Black or 17% Black or 45% over 65 or 50% over 65 or whatever.
Not to mention, when polls try and determine “likely voters” that’s another educated guess where pollsters use a variety of defensible methods to predict who will actually vote or not. In close elections, these choices make a huge difference in who is winning or losing in the polls. And as the Upshot feature points out, ultimately a bigger difference than the simple margin-of-error calculations based on sample size.
Anyway, the Upshot gave raw Florida polling data to four different pollsters and asked them to come up with an estimate for a final poll. They all showed a close race, but the results diverged not inconsiderably. Here’s some charts that show the different results, and importantly, the assumptions behind them:
Who’s right? Well insofar as this is a snapshot in time, we’ll never know. Then again, come the November exit polls, we can see whose assumptions about the composition of the electorate got it right and who got it wrong.
I so love that the NYT did this so that people can understand how polls really work. This is definitely going into multiple future syllabi.