Skip to main content

Questionnaire Length: The Long and Short of Participant Engagement

Survey length in online interviews continues to be a bone of contention among people all along the marketing research continuum. While some within the industry recommend keeping questionnaires short to promote participant engagement, others often dismiss this advice. Short questionnaires, they assert, won’t unearth the depth and breadth of information that is needed.

As a sample provider offering guidance based on research, SSI suggests keeping online interview length at 20 minutes or less. Generally, when interview length increases, fatigue also increases and, conversely, attention span decreases potentially damaging data integrity. In effect, researchers who insist on longer questionnaires, sincerely believing they’ll get more information, may, in actuality, be sabotaging their efforts.

Research Design

In order to more fully understand the possible effects of survey length, fatigue and subsequent response quality, SSI recently fielded two surveys: one long and one short. This study replicated a ground-breaking study conducted in 2004 by Sandra Rathod and Andrea la Bruna which concluded, among other things, that data quality suffers as interview length increases.

The surveys, both then and now, utilized a block design and were divided into four blocks of questions, each representing a different subject matter. The blocks were randomized for each respondent so that the effect of survey length on response quality could be investigated by comparing whether the different order of the blocks led to different response patterns as the block position varied in the survey.

Fatigue Effects

One of the hypotheses of the 2004 study was that respondents would take less time and exert less effort later in the questionnaire than they would earlier in the questionnaire, due to fatigue. This hypothesis was proven in 2004: as the same block of questions was moved further back in the study, the time taken to complete it gradually reduced.

It could be argued that the decrease in block completion time was due to increased familiarity with the question set. It is true that the question blocks were similar in their construction and contained somewhat similar questions. However, additional evidence on panelist fatigue shows that at least some of the increased speed was due to fatigue.

Panelist Fatigue and Satisficing

One of the behavior outcomes of cognitive fatigue is satisficing – doing just enough work to satisfy the task. To see if this behavior was present in the 2004 study, researchers looked at a question, in each block, that it was possible to skip. This question offered a set of scales, presented in the form of sliders. The slider bar was positioned at the mid-point so it was possible to click on “next” without moving the slider and still leave some data behind. The likelihood of skipping the question rose as the skippable question was encountered further and further into the questionnaire.

In 2009, SSI found precisely the same pattern. The first time the skippable question was encountered it was more likely to be completed than when it was seen on subsequent occasions.  This was particularly true for the long survey. A reduction in elapsed survey times did not mitigate the effect. The long survey, at nearly 25 minutes, was still too long.


In both 2004 and 2009, the long survey proved itself too long. It fatigued the respondent and led to satisficing behavior. When questions could legitimately be skipped, they were. Perhaps the most unsettling finding was that the instances of cheating, deliberately telling a falsehood in order to skip an entire section, also increased as the survey progressed.

Following the 2004 study, researchers indicated that there is a “critical point in online survey response when the fatigue effects become significantly more pronounced. That critical time is around the 20 minute mark. If researchers work to keep surveys shorter, it will not only help ensure response quality, but it will also make for more motivated and responsive respondents.”

Today’s research confirms that interview lengths of 20 minutes or less can produce wonderful and engaged responses if well designed. The fact that there was much less satisficing and cheating in the short survey attests to this. If researchers work to keep surveys shorter, it will not only help ensure response quality, but it will also make for more motivated and engaged participants.

Note: For a copy of the white paper Questionnaire Length, Fatigue Effects and Response Quality Revisited, with the results of the studies cited in this article and to share your comments, go to

About the Author: Pete Cape is Global Knowledge Director for Survey Sampling International. SSI provides access to more than 6 million research respondents in 72 countries. Sources include SSI proprietary panel communities in 27 countries and a portfolio of managed affiliates. SSI can potentially access anyone online to give their opinions via a network of relationships with websites, panels, communities and social media groups.
Reblog this post [with Zemanta]


Popular posts from this blog

Results of the Presidential Job Index Survey (week ending 7/1/2010)

Quick update on the results for the presidential job index survey for the week ending 7/1/2011. As a reminder this is a weekly survey that we conduct and we publish the results for the survey every Friday. To participate in this week's survey, please download the SurveySwipe application and take the survey.

Remember, you get points that you can redeem for various rewards when you take our surveys. We have already distributed several paid apps like Angry Birds for free.


Almost 900 respondents took the survey.
52% vs 53% last week of the respondents approve of the job that President Obama is doing.
Respondents included 28% Democrats (31% last week), 25% Republicans (22% last week), 24% independents (27%) last week and 2% with the Green Party (3% last week).

You can view the reports here or directly on your phone.

Top 5 Infographics of the Week: Cyber Security

Earlier this week, we experienced a power outage with our data center at Internap. During this short period of time, our customers were unable to access the site. Luckily, it didn't last long and we are back up and running now thanks to our dedicated support team that responded quickly and worked diligently.

Cyber security is, and always has been, a very important issue for us as we continue to grow and meet the demands of a technology-driven society. Nothing is more paramount than the safety of your valuable data. And apparently we weren't the only ones affected, as LinkedIn was experiencing difficulties on the same day and Internap had 3 massive outages earlier this year. Check out these infographics with stats on the importance of cyber security and the impact it can have on small-medium sized businesses.

Top 5 Infographics of the Week: Internet of Things

In 2008, the number of things connected to the internet exceeded the number of people on Earth. By 2050, we are expected to surpass 50 million different devices connected to the internet. And we're not just talking smartphones and tablets - we mean everything. Wearable tech, household appliances, smart homes, public transportation, hospitals and even cattle have all been using sensors to transmit data automatically. It is an exciting yet scary time to be alive. The various interactions between these different entities are opening up doors for more apps, more software and more auxiliary services. With so many "things" connected online, the possibilities for information exchange becoming wider everyday. Check out the top 5 infographics to learn more about the IoT.