Q and A from 10-Point Checklist for Questionnaire Design Webinar held April 7th, 2010

QuestionPro and SurveyAnalytics recently presented with Kathryn Korostoff of Research Rockstar on the topic: 10-Point Checklist for Questionnaire Design.

We had over 105 attendees who logged in from all over the world and had a very warm reception.

For those who missed it you can download slides, recordings, or pod-cast at the following: http://grandview.rymatech.com/pmv/webinars/2010/04/10-point-checklist-for-questionnaire-design.php

Many of the attendees presented us with interesting questions that we wanted to share with everyone. Please note that these answers are from our perspective and depending on the circumstances will not always apply to all research projects.

Q1: When sending an email via email how can we avoid having it being marked as SPAM?

A. On QuestionPro and Survey Analytics, we have a Spam Index. We run your uploaded email lists through a series of filters and preset algorithms to objectively determine the quality of your list. The possible values are:

1. Good - No Major Issues Found


2. Fair - Some Issues Found


3. Poor - List is highly SPAM suspect. Issues Found. QuestionPro WILL NOT accept email delivery for this list without manual approval from our support staff.


4. Restricted - Not Acceptable - This list will not be accepted by QuestionPro

Q1a: How can I guarantee that emails sent out will not be marked as spam by my company's mail servers?

A. General rule of thumb if planning to use your own company email, we recommend sending emails in batches of ten. Usually anything higher than that would result in being flagged for SPAM.

We cannot guarantee that your company's mail servers will accept emails from us (for your survey) - Different companies have different algorithms and policies as to what is considered as SPAM and what is not. PLEASE CONTACT YOUR INTERNAL IT STAFF as ask them about their ANTI-SPAM policies.

Additional Help Links on SPAM:

http://www.questionpro.com/help/11.html

http://questionpro.com/help/349.html



Q2: Where can you get sample to do a research study?

A. There are many online panel research companies that you can find online.

The surveys can be programmed to integrate with third party panel / sample providers.

Usually, panel management programs will distribute the survey URL that you provide them. They will want to pass in variables that they use to track respondents, into our system, so that we can forward these variables back to them after respondents complete the survey.

We have already integrated the following panels with QuestionPro.

• Authentic Response.


• e-Rewards.


• Generation Voice


• Greenfield


• GMI.


• Peanutlabs.


• Precision Research


• Rewards Central


• Survey Sampling International (SSI)


• United Sample

You just need to select the panel that you are using in the drop down menu and save the changes, the redirection links are updated automatically in the system

Q3. How much does sample cost?

A. This depends on the type of sample you need to do your study. In general, the lower the incidence the higher the sample will cost. Please contact a panel provider for pricing quote.

Q4: Do you have any best practice recommendations on surveys that use multiple pages or questions on just one page?

A. Although this question may apply more towards paper surveys, I wanted to tackle this from a perspective of an online survey. I would experiment with formatting your survey in way that would keep your respondents engaged in your survey.

Examples include adding/not adding page breaks, different question types, pictures, applying logic, etc. With my experience most surveys usually start out with easy questions, then move towards harder questions in the middle of the survey, and ending the survey with easy question types again.  You may also include a progress bar, which helps your respondent gauge how much longer your survey will take.

Progress Bar: http://www.questionpro.com/help/69.html

Q5: What practices can you engage in to best protect anonymity?

A. Speaking from our point of view, one of the challenges researchers face is the requirement for two directly conflicting issues

1. The ability to track who has responded to the survey and who has not -- for sending out reminder emails, giving out prizes or compensation etc.

AND

1. For human subjects protocols or other privacy reasons, ensuring that email identifications not be linked to the response data.

From a technical standpoint, if we need to track who has taken the survey and who has not, the survey researcher also implicitly has the ability to track the response for each individual.

To overcome this issue, the Respondent Anonymity Assurance has been introduced. QuestionPro asserts that once RAA is enabled on a survey, although computer generated identification numbers for individuals will be generated, the survey researcher will not have access to both the respondent's email address as well as the response data at the same time.

For more information please visit: http://www.QuestionPro.com/security/raa.html

Q6: We recently have sent out two B2B surveys.  The first went to 1200 people through email with a link to the website.  We have only had 8 responses.  How can we get more people to take the survey?



A. We would suggest looking into how to sent your emails first. Did you send it via our own company email or did you use a survey software tool? Going back to our answers in Q1 and Q1a, it may be that using our company email would result in having it marked as SPAM. See answers above for our solution. If you sent the survey from QuestionPro or SurveyAnalytics email management system, the next step is to send a reminder email of those who have not responded to your email. The general rule is generally no more than 3 contacts via email max: 1 email + 2 reminder emails afterwards.

Here is how to do reminder emails: http://www.questionpro.com/help/72.html



Q7: What are some follow-up strategies you recommend?



A. To follow up on surveys you are currently fielding we would recommend the following:

1. Review your current results: Are you meeting your quotas/specs set out prior to fielding the survey? If not, this is the time to look at your current sample list and target specific sample to reach your quotas.


2. Review the amount of sample you have left. Is it enough to help you reach your quotas?


3. Review all the ways you sent our your survey. Did you send it in an email? Did you post it to your website? Social network sites, etc?


4. Send out Reminder emails. The great thing about QuestionPro and SurveyAnalytics is that we only target the non-responders and leave those who have taken the survey alone.


5. If you have the budget perhaps you may want to add other survey gathering options such as phone surveys, intercepts, etc. to help increase completion rates.  www.Quirks.com is a great resource to implement other options.


6. Rethink your quotas. Given the amount of time a study has been fielded, it may be time to recalculate your incidence rates and readjust quotas to meet your budget.

Q8: For B-B online surveys, what's best day of week, time of day to send the survey request out (invite sent out via email)



A. I wish we could say that Tuesday at 12:00 PM PST is the perfect time to B2B customers. Unfortunately we do not live in that world. Some say it really depends on which industry or location you are targeting, and others say there’s no time like the present. With my experience I have found the best strategy is to pair up your online survey with a strategy to follow up with your survey + incentive to complete your survey. It can be a reminder email or a phone call. Either way, following up is key to reaching your quotas especially for B2B market research.



Q9. Do you think questionnaires are the best source of information from your target population/audience?



A. From the online survey software standpoint, it is a very powerful and inexpensive way to gather and collect data for analysis in real time.  The sooner you gather information the faster you or your clients can react to the opinions of your target audience. It’s standard to pair survey research with other forms of research such as focus groups, one-on-one interviews, on-line diaries, etc. Each research project is unique and it depends on your target audience which forms of research would be most applicable.