Skip to main content

Response Style Bias and How to Overcome It

Happy Face Neutral Face Sad Face with Buttons
A few weeks ago, Research Access had an article that talked about cleaning data from panels.  One of the methods was to remove all the "extreme" responses that occurred in rating scales.  If you've ever received a survey with too many matrix rating questions and then mindlessly checked all the lowest ratings, mid-ratings or high ratings -- you know who you are and you know what I'm talking about.

This put a bug in my mind about those people who never give the highest rating -- or those people who consistently give a high rating.  In many ways, they are no better than those panelists who were mindlessly answering their survey.  In other words -- their responses were suspect in being valid answers to help us make a decision.

Then I ran into this article by Jeff Henning and realized that I wasn't the only one thinking about this.  In fact, people much smarter than I'll ever be have done loads of research on the topic and have come up with several solutions that I'm going to show you here.

Henning's article actually gets into the discussion about cultural response bias, and you can read more about that in his article.  The following are question types that Henning listed that will help you overcome general response style bias:


  • Binary scales – As far back as 1946, to minimize response style bias, Lee Joseph Cronbach (ofCronbach’s alpha fame) advocated using two-item scales: yes/no, agree/disagree, dissatisfied/satisfied, describes/does not describe. Essentially this treats everyone as Extreme Response Style responders.

  • Choose-many questions – Presenting a list of choices and instructing the respondent to “select all that apply” is an economical form of binary scale, prompting respondents to choose the items they agree with, find important or are dissatisfied with.

  • Ranking questions – Another way to avoid traditional response bias is to use ranking scales, where each choice on the scale may be used only once: most important to least important, most satisfactory to least satisfactory, most likely to least likely.

  • MaxDiff scaling -- Maximum-difference discrete-choice models are a more sophisticated type of ranking question, typically showing attributes four at a time and asking the respondent to select the best and worst attributes from each set: the attributes with the maximum difference. Research has demonstrated that MaxDiff scaling is superior to rating scales for cross-cultural analysis. (Steve Cohen & Leopoldo Neira, 2003, “Measuring Preference for Product Benefits Across Countries: Overcoming scale usage bias with Maximum Difference Scaling”.)


Lessons Learned: Surveys should provide additional data for your decision -- not MAKE Your Decision

So it isn't just panels that can "dirty" your data enough to mis-represent how respondents really feel.  Response bias comes in as many shapes and forms as there are human opinions.  The key point here is that decisions are ultimately made by human beings.  Managers and business owners can use survey data as an additional information resource to help them decide -- sort of like the  "Ask the audience" option in "Who Wants to be a Millionaire" but that doesn't mean you shouldn't use your own judgement.

Comments

Popular posts from this blog

Results of the Presidential Job Index Survey (week ending 7/1/2010)

Quick update on the results for the presidential job index survey for the week ending 7/1/2011. As a reminder this is a weekly survey that we conduct and we publish the results for the survey every Friday. To participate in this week's survey, please download the SurveySwipe application and take the survey.

Remember, you get points that you can redeem for various rewards when you take our surveys. We have already distributed several paid apps like Angry Birds for free.

Highlights

Almost 900 respondents took the survey.
52% vs 53% last week of the respondents approve of the job that President Obama is doing.
Respondents included 28% Democrats (31% last week), 25% Republicans (22% last week), 24% independents (27%) last week and 2% with the Green Party (3% last week).



You can view the reports here or directly on your phone.

Top 5 Infographics of the Week: Cyber Security

Earlier this week, we experienced a power outage with our data center at Internap. During this short period of time, our customers were unable to access the site. Luckily, it didn't last long and we are back up and running now thanks to our dedicated support team that responded quickly and worked diligently.

Cyber security is, and always has been, a very important issue for us as we continue to grow and meet the demands of a technology-driven society. Nothing is more paramount than the safety of your valuable data. And apparently we weren't the only ones affected, as LinkedIn was experiencing difficulties on the same day and Internap had 3 massive outages earlier this year. Check out these infographics with stats on the importance of cyber security and the impact it can have on small-medium sized businesses.

Top 5 Infographics of the Week: Internet of Things

In 2008, the number of things connected to the internet exceeded the number of people on Earth. By 2050, we are expected to surpass 50 million different devices connected to the internet. And we're not just talking smartphones and tablets - we mean everything. Wearable tech, household appliances, smart homes, public transportation, hospitals and even cattle have all been using sensors to transmit data automatically. It is an exciting yet scary time to be alive. The various interactions between these different entities are opening up doors for more apps, more software and more auxiliary services. With so many "things" connected online, the possibilities for information exchange becoming wider everyday. Check out the top 5 infographics to learn more about the IoT.