Skip to main content

What Is The Right Sample Size for My Survey?

How many respondents are really enough?
There are two schools of thought about sample size – one is that as long as a survey is representative, a relatively small sample size is adequate. Perhaps 300-500 respondents can work. The other point of view is that while maintaining a representative sample is essential, the more respondents you have the better.
This is a big issue because it impacts all sorts of decisions including the length of your survey, your collection mechanism, and, of course, you’re sampling rate.
So what’s the right answer?
If you aren’t integrating your survey data with web behavioral data, then a relatively small sample size might be okay (I’d emphasize the might). But if you want to combine behavioral analysis and survey data, then forget a sample 300 or 500 respondents. Those numbers simply won’t work.
Let me give you a real-world example showing why that’s true. We’re working right now with a client that samples approximately 1000 site visitors a month for their satisfaction survey. They asked us to do a study of the impact of using one of two internal search tools on their site on both overall site satisfaction and visit accomplishment.
On this site, search is used in about 10% of visits. Since the site gets more than 10 million visits a month, that still yields a heckuva lot of behavior to study – more than 1 million search visits every month. No problem there.
But our representative sample only captured about 100 respondents who’d used search.
Between the two search tools, one served about 70% of the queries. So for the second search tool, we had about 30 respondents to deal with. Getting the picture?
For our analysis, we wanted to track visit reason vs. satisfaction vs. outcomes for searchers. With some visit reasons only accounting for about 10% of visits, there were cases where we were supposed to analyze the outcomes for all of 3 visitors.
And that’s with a survey size of 1000 and a relatively simple cross-tabulation of visit intent and one fairly common behavior. Sure, we could add lots more months to the picture. But tracking behavior over extended periods of time adds all sorts of complications to the analysis. The combination of seasonality, site change and macro-economic change make this dangerous. Very few of our client sites remain constant for six months.
If doing behavioral analysis with 1000 survey respondents is challenging, imagine what it would be like with a sample size of 300. Impossible.
So what’s the right answer?
If I had my druthers, I’d recommend that high volume sites strive for a much higher sample size – something like 15K would be nice on a monthly basis.
Is that too much for your user experience to bear? Obviously, the answer depends on your site volume and your take-up and completion rates.
You can’t do much about volume, but if your survey length is impacting your take-ups or completion rates, then I’d be willing to sacrifice a whole bunch of questions to get to the increased size. The fact is that on many 30-40 question surveys, we’d only expect to use at most 5-10 of those questions in a behavioral analysis. I’d bet even money that your analysts feel that same way and that a heavy majority of questions on many long surveys hardly ever get studied at all.
At some point you may have to make a decision: do you want a whole lot of really shallow information or do you actually want to do analysis on a narrower set of data?
So when it comes to behavioral analysis combined with survey integration, the right answer is pretty obvious. A representative sample is essential, but size really does matter. Let yourself get talked into a 300 person sample, and you might as well throw all that work you did to integrate online survey data with behavioral data in the junk pile.
Here are some more tips on how to determine your survey sample size.


Popular posts from this blog

Results of the Presidential Job Index Survey (week ending 7/1/2010)

Quick update on the results for the presidential job index survey for the week ending 7/1/2011. As a reminder this is a weekly survey that we conduct and we publish the results for the survey every Friday. To participate in this week's survey, please download the SurveySwipe application and take the survey.

Remember, you get points that you can redeem for various rewards when you take our surveys. We have already distributed several paid apps like Angry Birds for free.


Almost 900 respondents took the survey.
52% vs 53% last week of the respondents approve of the job that President Obama is doing.
Respondents included 28% Democrats (31% last week), 25% Republicans (22% last week), 24% independents (27%) last week and 2% with the Green Party (3% last week).

You can view the reports here or directly on your phone.

Top 5 Infographics of the Week: Cyber Security

Earlier this week, we experienced a power outage with our data center at Internap. During this short period of time, our customers were unable to access the site. Luckily, it didn't last long and we are back up and running now thanks to our dedicated support team that responded quickly and worked diligently.

Cyber security is, and always has been, a very important issue for us as we continue to grow and meet the demands of a technology-driven society. Nothing is more paramount than the safety of your valuable data. And apparently we weren't the only ones affected, as LinkedIn was experiencing difficulties on the same day and Internap had 3 massive outages earlier this year. Check out these infographics with stats on the importance of cyber security and the impact it can have on small-medium sized businesses.

Top 5 Infographics of the Week: Internet of Things

In 2008, the number of things connected to the internet exceeded the number of people on Earth. By 2050, we are expected to surpass 50 million different devices connected to the internet. And we're not just talking smartphones and tablets - we mean everything. Wearable tech, household appliances, smart homes, public transportation, hospitals and even cattle have all been using sensors to transmit data automatically. It is an exciting yet scary time to be alive. The various interactions between these different entities are opening up doors for more apps, more software and more auxiliary services. With so many "things" connected online, the possibilities for information exchange becoming wider everyday. Check out the top 5 infographics to learn more about the IoT.