A great write-up from professional survey-creator Scott Weinberg has shone some light on the risks of using mass-recruiting panels for your online surveys. We’ve edited down an excerpt below. A reminder to be sure to maintain panel health and quality as strictly as you monitor your survey quality!
“An MR leader in a huge tech company said something interesting on a call I remember vividly. He asked: ‘when is the last time you washed your rental car?’ The context here pertained to online sample. The problem is this: why would you ever wash your rental car? Why change the oil? Why care for it at all? You use it for a day, or a week, and you return it. Mass-recruited online panelists are no different. You use them for 5 minutes, or 20, and return them to the provider.
If we actually cared about them, the surveys we offer them wouldn’t be so stupefyingly, poorly written. Half of the surveys that mass-recruited panelists complete are flat out laughable. Filled with errors and illogical questions. Around a quarter consist of nothing but pages of matrices. If you’re an online panelist, they’re simple boring. Are the semi-professional survey-takers in a mass-recruited panel really going to thoughfully answer 30 questions about laundry detergent for a dollar? Ever think about who is really taking these surveys?
One of the saddest observations is the lazy mechanics of sample purchasing. In the last couple years, sample quality is simply assumed. When buying access to mass-recruited panels, MR firms assume suppliers check their own panel quality (like the rental car analogy!). Quality is not winning sample provider orders. Quantity and margin are.
Now that I’ve thoroughly depressed you, one may wonder, is there any good news? Yes – ‘invite-only’ panels give the best shot at good, clean sample. When you open your front door to anyone with a web connection, and tell them there’s money to be made, well, see above. As a quality check, design surveys with an open-end comment capture, to avoid fraudulent panelist activity. It’s much more difficult to fake/auto-complete good, in-context open ended verbiage. Yes it takes a bit more work on the back end, and there are many solutions that can assist with this.”
In many cases, it’s best to bite the bullet and manage your own panels rather than dive into the discount bin of mass sample providers. Let us know if you have any questions about getting started!