How to read a poll (and how not to conduct one)
October 24, 2016 1 Comment
Nate Cohn wrote this a couple weeks ago, but with Republicans hanging all their hopes these days on the massive outlier polls from IBD/TIPP it’s worth posting Cohn’s piece on how to make sense of any single poll. Lots of good advice, but I especially liked this part:
Comparing Where They Were
To get a sense of whether a poll is good or bad news for a certain candidate, I usually compare the results of the poll with the polling averages or the last poll conducted by the same pollster.
■ If the poll is very different from the polling average, there’s a good chance it’s an outlier. [emphasis mine]
■ If the poll shows a big shift from a prior survey, I also wonder whether the previous poll was an outlier. If so, a candidate might appear to rebound simply because he or she was unusually weak in a prior poll. So compare that prior poll with the average of the time, too.
■ It’s also worth looking at whether the candidate has gained or lost vote share. When candidates fall without good reason, I often assume they’re likelier than not to win back their former supporters. I definitely take note when candidates have won more supporters than they’ve won before. If that happens a lot, it’s a real sign of strength…
■ I really don’t look at party identification. We have no idea what the “right” partisan breakdown of the electorate really is: It’s an attitude, not a nearly fixed characteristic.
Depending on the news or the national political environment, Republican-leaning or Democratic-leaning voters can switch in and out of the “unaffiliated” or “independent” column. It is very clear that there are more self-identified Democrats than Republicans in the country, which has been true for about a decade.
But I will look at party registration, if it’s available from the voter file. That’s a pretty fixed characteristic: It doesn’t swing with the mood.
Also, an amazing story of polling gone wrong. You may have noticed one particular poll (LA Times) that seemingly has a huge pro-Trump bias. Turns out it is largely due to a single 19-year old Black man. Nate Cohn again:
There is a 19-year-old black man in Illinois who has no idea of the role he is playing in this election.
He is sure he is going to vote for Donald J. Trump.
And he has been held up as proof by conservatives — including outlets like Breitbart News and The New York Post — that Mr. Trump is excelling among black voters. He has even played a modest role in shifting entire polling aggregates, like the Real Clear Politics average, toward Mr. Trump.
How? He’s a panelist on the U.S.C. Dornsife/Los Angeles Times Daybreak poll, which has emerged as the biggest polling outlier of the presidential campaign. Despite falling behind by double digits in some national surveys, Mr. Trump has generally led in the U.S.C./LAT poll. He held the lead for a full month until Wednesday, when Hillary Clinton took a nominal lead.
Our Trump-supporting friend in Illinois is a surprisingly big part of the reason. In some polls, he’s weighted as much as 30 times more than the average respondent, and as much as 300 times more than the least-weighted respondent…
Alone, he has been enough to put Mr. Trump in double digits of support among black voters. He can improve Mr. Trump’s margin by 1 point in the survey, even though he is one of around 3,000 panelists.How has he made such a difference? And why has the poll been such an outlier? It’s because the U.S.C./LAT poll made a number of unusual decisions in designing and weighting its survey…
A typical national survey usually weights to make sure it’s representative across pretty broad categories, like the right number of men or the right number of people 18 to 29.
The U.S.C./LAT poll weights for many tiny categories: like 18-to-21-year-old men, which U.S.C./LAT estimates make up around 3.3 percent of the adult citizen population. Weighting simply for 18-to-21-year-olds would be pretty bold for a political survey; 18-to-21-year-old men is really unusual.
On its own, there’s nothing necessarily wrong with weighting for small categories like this. But it’s risky: Filling up all of these tiny categories generally requires more weighting.
A run of the U.S.C./LAT poll, for instance, might have only 15 or so 18-to-21-year-old men. But for those voters to make up 3.3 percent of the weighted sample, these 15 voters have to count as much as 86 people — an average weight of 5.7.
When you start considering the competing demands across multiple categories, it can quickly become necessary to give an astonishing amount of extra weight to particularly underrepresented voters — like 18-to-21-year-old black men.
Anyway, maybe the LA Times poll is right and almost everybody else is just wrong. But, I strongly suspect we’ll have proof otherwise come November 8.