Avoiding Survey Bias – Ensuring your surveys get the right responses

Surveys are a valuable source of information to derive insights from. There are many factors that can affect the quality of the data you’re collecting. Survey Bias is one of the most tricky elements to keep under control when gathering data. It’s extremely difficult to discern if the survey results that you’ve accumulated have been tainted by bias in any way.

What complicates matters is that the bias is not always an intentional act on the part of the respondent. Sometimes people just tell you what they think you want to hear. In some cases, like in the healthcare or insurance industry, biased survey results can often result in conclusions that are negative or dangerous.

Humans are inherently biased by default, so the burden falls on the shoulders of the researcher to ensure that they manage to get what is unbiased and without prejudice.  

Are you sure you’re not the one responsible?

When creating a survey, you need to be careful not to frame your questions in a way that leads respondents on. It’s important to avoid polarizing or emotionally triggering questions unless absolutely necessary as an upset respondent is likely give biased responses.

It’s also advisable to keep your language as simple as possible. A person who’s unfamiliar with your product or service needs to be approached in a clear and direct manner, or you risk getting incorrect data.

Good design = Good responses

Responses to web surveys can be influenced by their overall design. A cohesive, well-designed survey interface is likely to get better quality responses. This design language needs to extend to mobile platforms as well – with smartphones being as ubiquitous as they are today, there’s a huge likelihood that your respondents will attempt your survey on a mobile phone or tablet

A modern survey platform can assist you in creating well designed, mobile friendly online surveys that help capture the data you need.

Know your demographic

Studying your demographic is a key part of making and distributing a survey, but did you know it’s a major factor in survey bias as well?

Surveys distributed to the wrong audience can lead to incomplete surveys and biased responses which will pollute your data.

Structure your survey

A key factor in reducing response bias is by making sure that the respondent has answer choices which are concise and not too wordy.

It’s best not to include additional questions within the primary question itself, as it can confuse or irritate your respondents.
You must also keep in mind that the people taking part in your survey may not be familiar with the topic they’re being queried on, so overwhelming them with information can have a negative impact and result in biased answers.

Do your respondents have an escape route?

Sometimes, even after supplying them with the necessary information, the respondent may feel that they just don’t know enough to answer the question appropriately.
This is perfectly acceptable and you need to account for it in your surveys by giving them the option to say so!

A simple “No”, or “Undecided”, or “I don’t know” or a “Beam me up, Scotty” can go a long way in ensuring the accuracy of your data.

Avoiding Survey Bias = Better Data

Surveys are data sensitive and can result in you having to deal with a mountain of analysis. Modern online survey tools can definitely help you in collating this data so you can glean insights from it, but you need to feed them with the right questions. 

Although it’s not feasible for your respondents to answer questions in a completely unbiased manner, a robust and carefully crafted survey can go a long way in helping you minimize bias.


Behind-the-Scenes with our Client Feedback Team

We have been making some of the industry’s most flexible survey tools for over 25 years. We’re always evolving our powerful multichannel platform, which includes launching new products, major releases, and regular maintenance and feature updates. Through it all, we’ve definitely become experts in the craft of survey software.

But savvy product managers know that there’s a gap between the knowledge that comes from creating and refining a product as a developer, and as an end user who is actively using it daily. To close that gap, our Voxco Online product team has long been tapping into the user perspective – what features do they need the most, how do they spend their time on the platform, what would they do differently?

Informally reaching out to users for feedback had always been happening, but just last year, our product team formalized a client outreach program that maximizes the impact of the feedback on the final product. The outreach has been invaluable; not just for our team, but for clients who could help shape the product’s development.

Now that it’s a formal part of our process, here’s a behind-the-scenes look at our client outreach programs – how they work, and how much of an impact they have had on the online survey tools. Let’s take a peek under the hood:

User Surveys

We make survey software, so it was a natural jump to use the platform to ask our own clients for feedback. To gain quantitative data about in-demand features, the surveys asked multiple choice and ranking questions that aimed to prioritize the relative importance of new features. At the end of each survey, qualitative data was collected via a series of open-ended questions that asked users to precisely describe what key features would maximize their organization’s use of the tool, and how existing functionalities could be further enhanced.

Advisory Committee

Post-survey, we wanted to continue the qualitative conversations, and dig deeper with active users on how they could maximize their productivity on the platform. So we assembled a team of 15 of our most active clients, representing varying industries, organization sizes and survey types. In the pursuit of more qualitative insights about specific upcoming features, we formalized this group into a client advisory committee.

When we are in the early planning stages of a major new release, we can now approach a team of active users as a sounding board. The conversation starts as an overview call to get their input on how they use a specific feature set. We then use this rich feedback to further define product requirements and create interactive mock-ups of the new feature set. We hop on a second call to present the mock-ups to their teams, and fine-tune the details over time.

Data galore!

It was nice to see complimentary patterns from our varied client types that helped validate our internal team’s existing direction. The client survey offered instant clarity on which new features were must-haves, which helped the product team to immediately start prioritizing the next few releases.

We were amazed by the long, detailed answers given to the open-ended questions. It was clear that clients were very open to the communication channel and were excited to have their voice heard. The responses helped stimulate new ideas for future features and started many active discussions among the development team.

We have witnessed very high involvement levels from all members of the advisory committee, and the resulting conversations feel very collaborative. The members often choose to gather project-specific suggestions from their larger team, and even share the feature mock-ups with them for more robust user feedback. It’s quite clear that the advisory committee enjoys being consulted and collaborating on feature development.

Overall, the entire process has helped us define a very solid road map for the next year worth of updates, and with the help of many of our users, we have populated our first 2 major releases with many of their most commonly requested features.

Coming soon: results

The earliest major release that was directly impacted by the new client feedback is coming out in just a couple of months! And the client feedback team has already moved on to collecting data on a future release that will enhance our reporting tool. Next month, we’ll share with you some of the results of the first few client outreach initiatives right here on the blog.

If you’re a user of the industry’s most flexible survey platform, keep an eye out for our surveys to join in the conversation – you could help us define the future of Voxco Online. Stay tuned!


4 Most Important Factors in Survey Project Design

The bi-annual GRIT report was released by Leonard Murphy at Greenbook last week. As usual, it presents a very telling snapshot of the state of the market research industry. Which issues are important and which are less so. What is hot. And what is not.

The report surveys almost 1,600 individuals who work in market research, 80% suppliers (mostly from MR firms), and 20% buyers (clients). Some of the most interesting results come from the comparison of how both sides view the same issues.

As creators of survey software, the findings that generated the most inter-office conversation were those about what researchers felt mattered most in research project design. Specifically, this summary bar chart of the more interesting findings of respondents’ Top 2 most important factors:

1. Research Trustworthiness

Most important to both suppliers and clients is how trustworthy the research is. And rightfully so – that’s what clients are paying for, and the insights generated are what will lead to organizational shifts.

2. Sample Quality

A direct contributor to #1 above, a full half of the supplier and client respondents chose to include in their Top 2 most important factors that the recruited database or panel needs to be of high quality and representative.

3. Respondent Focus

Somewhat surprisingly, there was a steep drop in clients and researchers acknowledging the imjportance of the amount of time that a respondent takes with the research project/survey and the focus they give it. But there were still a quarter of the researchers (and a fifth of the clients) who chose this option in their Top 2. More on this in a moment.

4. Survey Scope Clarity

The only result in the top 4 where more clients selected it than researchers, 26% of clients want to ensure that survey participants have a clear idea of the scope of the project.

Of Concern

As noted in three of the four orange highlight points in the chart above, there was a significant difference in the results that favored the researchers’ Top choice (that the research should be good) compared to those that measured respondent experience.

In addition to the steep drop off before respondent focus and clear communication of project scope, responses addressing respondent compensation and satisfaction indicators received less than 10% of researcher responses, and less than 5% of client responses! (see last three answers in chart above)

As Murphy states, “It seems the relationship between researchers and research participants has become very one-sided – what we expect from participants is not aligned with what we feel our obligations are to them.”

If you build it, they will come. Build great surveys. Communicate well with your respondents, and treat them with the respect they deserve. By keeping your respondents satisfied, you will get better results. We need to shift our focus outwards and create the kinds of survey projects that foster great results.

Download the full GRIT report HERE – it’s free and jam-packed with rich industry insights.

Evaluating-Organization-technology-Strategy-Voxco-Blog-400x250 (1)

Feature Update: Enhanced questionnaire structure view

This week we started rolling out the newest version of our Voxco Online SaaS platform to include a highly requested feature. In the past, after questionnaires were written and programmed, the default preview display made it difficult to present the questionnaire structure to others and request their feedback. Well no more!

We have reviewed and updated the entire questionnaire structure preview to make it more helpful when presenting questionnaires to others.

These changes will make the questionnaire structure preview easier to understand on-screen, including the ability to add survey settings and block settings directly via the display. We’ve also added an option to export the questionnaire structure display into Microsoft Word as an HTML file. The Word file can then be formatted, edited, and presented to others:

Now Voxco Online users can easily share questionnaire structure with non-users, who can see all questionnaire language and survey logic in one place. This is especially helpful for market researchers who regularly present questionnaire content and flow to their clients for their feedback and approval.

If this is something that could benefit your market research team, give it a try! You can access the new questionnaire structure preview directly from the survey editor with the slider under the side navigation (choose ‘Structure’):

You can then export the previews to an HTML file by choosing ‘Print All’ or ‘Print Selected’ in the side navigation, and then choosing the new ‘Save as HTML for Word’:

Stay in touch! We’re regularly updating the software with new features, many of which are requested by clients.