Pollsters have been faced with some high-profile flops in recent years. It’s becoming more common-place to see headlines of polls that predict very different political races than those that actually occurred.
In 2014, the people of Scotland voted against national independence by a wider-than-polled margin. Stateside, pollsters tended to underestimate the midterm wave by the Republicans. Many observers have recently suggested that polling is not an appropriate tool for studying subjects like religion.
In the face of mounting challenges and diminishing accuracy, how can survey researchers have confidence in their findings? Scott Keeter of the Pew Research Center recently addressed these concerns.
Two rising trends are causing a slip in the reliability of political and other polling worldwide: the vastly increased use of cellphones and the continuous decline in respondent willingness to answer surveys. But despite these two major challenges, the industry still relies on polls to determine the thoughts, feelings, and actions of populations. Mr. Keeter has answered some of the key questions and challenges swirling around the polling discussion:
Low Response Rates do not mean Unreliable Polls. Although non-response bias tends to be greater when response rates drop, the presence of the latter does not automatically insinuate the former. When conducting telephone surveys, for example, if the researchers accurately weight landline vs cellphone respondents to match the demographic composition of the population then the non-response bias is kept to a minimum.
Falling Response Rates is not a new concern. Experts in market research have been noting and discussing response rate declines for over 25 years, says Keeter. And Pew Research has been documenting the trends along the way, which has recently led to greater awareness of the challenges (and better ways to face them!).
Respondent Willingness is dropping for numerous reasons. People are harder to contact due to their increasingly busier lives and their more mobile technological tendencies. Technology also leads to a increased tendency to ignore unknown phone calls. Personal perceptions are increasingly leaning towards privacy concerns around data collection and that surveys are burdensome. (all the more reason to keep them simpler and more logical!)
Response Rates are nearly identical on landlines and cellphones. Although it generally takes more time to get completed surveys from respondents on mobile (due to manual dialing, requirements to interview only the responder, and increased numbers of minors with cellphones), response rates are almost exactly the same.
The team at Pew Research has been facing and overcoming the challenges of falling response rates for a long time now, and can be looked to guidance through these changing times. despite the challenges, they still devote great effort to ensuring their surveys represent the general population. This sometimes requires numerous callbacks over several days and constantly weighting samples to better represent mobile users.
Pew Research Center’s team is regularly improving their current survey techniques while simultaneously testing alternate ways to measure public attitudes, behaviors and actions. The future of research will likely involve a combined format of surveys and Big Data. Striving to do the best work possible while maintaining transparency is the current plan, and we can all take some leadership from Pew and Keeter on this direction.