FAQ

Designing a survey is complex. Things to consider are:

1. How diverse is the population; the more diverse and different the population is from one another, the higher the sample of respondents should be. Usually, this is measured statistically, by standard deviation, variance or a similar means.

2. What form does the population take (statistical graph); A population that has a normal statistical distribution needs to be treated differently than a population that is skewed to the left or right. A population that has very fat tails, known as kurtosis, needs to be treated differently than one that does not. A great example of how a survey can be designed incorrectly, is the now infamous story of Long-Term Capital Management. This hedge fund was full of brilliant Nobel laureates and mathematicians, well-funded, but still ended up losing it all because they didn’t consider carefully enough, kurtosis!

3. Your budget! What is ideal mathematically may not be ideal financially. Before considering a survey project, you should understand your budget constraints and what is possible. Surveys can be very expensive when done correctly.

4. How many in the population are eligible to participate in the survey, known as Incidence Rate. More importantly, how much of the population you contact will actually participate in the survey. Take for example a survey that requires 100 B2B respondents. If it takes 15 qualified respondents to get one to participate, and the incidence rate is 1 in 10, then it would take 150 contacts to get one successful survey. At Tailor Research, we call this the classic needle in a haystack exercise, which can be quite expensive. Ideally, the population all qualifies when reaching-out for surveys.

5. Define your goal clearly and realistically. The more time you spend defining what the goal and parameters of the survey are, the better. In other words, a well thought out survey, is one that clearly defines the sample of survey respondents. This may take research, to understand the questions you want answered and who is most likely to answer them well. A good survey will have a clear purpose that surveyors/respondents understand and keep in mind, while answering questions. The desired confidence level and margin of error required are crucial in the design of the survey. Once again, it is important to understand what is possible with the given budget and what is realistic. Phone surveys are more expensive than email blasts and each has its advantages—understanding the benefits and drawbacks is important to designing a survey.

6. The order of the questions in the survey matter. Many times, an incorrectly designed survey will have the most difficult questions first, which causes many problems. For one, the respondent is more likely to give up on the survey, which will increase the cost of the survey because more potential respondents will need to be contacted to get one good response. Also, respondents are human (or at least most of the time), so psychologically they need time to “warm-up” to difficult questions and a survey in general. Respondents usually don’t like participating in surveys, so making the survey as easy to digest as possible is important to the overall design of the survey.

7. The number of answers to a question matter. In general, although not always, an even number of answers in a multiple-choice format is better than an odd because people tend to choose the middle answer more frequently than other answers. With no middle answer, less bias is introduced. Also, having too many answers can lead to respondent fatigue, which often leads to the first choice being chosen.

8. The length of the survey is one of the most critical aspects of a survey. Often, surveys are designed to be too long. What is the issue with a lengthy survey? The issue is that respondents get fatigued while conducting a survey. The lengthier the survey, the more fatigue. An ideal survey will take no longer than 10 minutes to complete, before fatigue sets in. The ideal amount of questions is no more than twenty, however that assumes just a few open-ended questions. The more open-ended question, the fewer questions overall.

9. Carefully word your survey questions. When writing questions, try to write the question in a neutral way. For example, “You liked the software solution?” Y/N, is not as carefully worded as “What are your thoughts about the software solution?” (a) It was average (b) It was bad (c) It was good (d) I will need to get to know the product better

10. Stay away from absolutes, like always, never, all, etc., when asking a question because often that will bias a survey to a negative response.

Both phone-based and digital surveys have their advantages. The advantages for phone-based surveys are that they allow the caller/surveyor to navigate to the correct respondent, verify the respondent’s knowledge, clarify the intent and meaning of questions with the respondent, as well as clarify answers given from the respondent. Also, open-ended questions are easier to elicit full answers. Because respondents are verified by the caller, respondents cannot “game the system” like they can in digital surveys. In other words, digital surveys often have the same respondent filling out a survey under different profiles in order to gain the fee/reward. Phone-based surveys are often better for B2B surveys.

The advantages of digital surveys are they are less expensive, they can reach more respondents quicker, and sometimes respondents feel freer to answer questions honestly. Digital surveys are often better for Consumer surveys.