A lot of you had ideas on how we might do a little better in reaching people for our surveys.
We’re already in the field with our penultimate wave of New York Times/Siena polls — this time focused on four or five key House races — so let’s go to the mail and answer readers’ questions about our surveys.
This week, our inbox was full of replies to our recent note on the grim reality of telephone polling: Less than 1 percent of dials yield a response. A lot of you had ideas on how we might do a little better.
Maybe the most frequent suggestion was some version of this:
I think some of us who no longer answer calls from unknown phone numbers might answer if the call identified itself as from a polling firm. — Deb M
My mom also suggested this last weekend. It would certainly make the poll cheaper. But as I told her, I think this might be a mistake. We want a representative sample. I don’t think the way we want to increase our response rates is by further attracting the kinds of politically engaged folks who would be excited to take a political poll. We already have many highly engaged voters as is.
Another question came from someone who is no stranger to survey research:
Why doesn’t The Times move to an online probability sample? — Cliff Zukin, a former president of the American Association for Public Opinion Research
Before I answer, I just want to flag a key word in this question: probability. A “probability sample” is one in which every person has a known probability of being selected for the survey. To take an example: If we randomly dial telephone numbers, everyone with a telephone number (basically everyone) has a chance of participating; thus, it’s a probability sample.
Many online polls, however, are not probability samples — think Morning Consult or YouGov. These polls survey only people who previously signed up to participate in an online panel. It is very challenging to craft a representative survey with the idiosyncratic folks who decide to join an online panel after clicking on a random banner ad.
The State of the 2022 Midterm Elections
Both parties are making their final pitches ahead of the Nov. 8 election.
- Where the Election Stands: As Republicans appear to be gaining an edge with swing voters in the final weeks of the contest for control of Congress, here’s a look at the state of the races for the House and Senate.
- Biden’s Low Profile: President Biden’s decision not to attend big campaign rallies reflects a low approval rating that makes him unwelcome in some congressional districts and states.
- What Young Voters Think: Twelve Americans under 30, all living in swing states, told The Times about their political priorities, ranging from the highly personal to the universal.
- In Minnesota: The race for attorney general in the light-blue state offers a pure test of which issue is likely to be more politically decisive: abortion rights or crime.
An online probability sample, on the other hand, would have the rigor of a telephone poll. The most common way to pull it off is to mail people an invitation to participate in a poll online. In many cases, the respondents are recruited to join a longer-term panel, where the pollster can contact them over and over. A lot of firms now use these kinds of online probability samples: Pew Research, Associated Press/NORC, Ipsos/KnowledgePanel and now CNN with SSRS, to name just a few.
If we stipulate for a moment that this would be cheaper — and it may not be, by the way — there’s an argument this could work for The Times in certain cases. But there’s one big limitation for us: It’s hard to conduct an online probability sample by state or district, and most of our polls are state or district polls.
We couldn’t build a large enough panel in all the states (let alone districts) where we might want to conduct a survey. Without a panel that we can recontact on-demand, we’re stuck with a one-off mail-to-web poll in which we mail people letters inviting them to participate in an online poll. It can take a long time.
The last CNN/SSRS mail-to-web poll, for instance, was fielded over the course of 32 days — from Sept. 3 to Oct. 5 — and released on Oct. 13. They probably wrapped up the questionnaire well before Sept. 3, given the need to print and mail questionnaires. I’m glad CNN is trying this, but personally the result felt stale to me.
That said, I do think there’s room for something like this to be part of our portfolio. It might be useful far from an election. Or if the data is of especially high quality, perhaps it can be used to calibrate cheaper surveys.
To that point, here’s an idea: cold, hard cash:
By your own account you have to pay a substantial amount of money for one completed phone interview. Two hours of salary and miscellaneous expenses. Why not pay the interviewee for his or her time and trouble? For $20 or so, a reasonable number of people would talk to you. — Tom Hill
That’s a good thought. In fact, it’s such a good thought that we’re trying this in a large mail-based study of a key battleground state, right now! More on this in a few weeks.
Source: Elections - nytimes.com
