We often here the results to a poll or survey while watching the news, or see a graph online followed by the comment that the “margin of error is plus or minus 5%”. So what exactly does this mean? Can you really trust the numbers?
Every survey or poll has a margin of error. The margin of error is directly related to the number of people surveyed out of a population group. The number of people surveyed is known as the sample size. The larger the sample, the smaller the margin of error (to a point). If we were able to survey everyone, there would be no margin of error. This would be called a census.
So, if the results to a poll state that there is a 5% margin of error, this means that results are within 5% of what they would be if we exhaustively interviewed everyone (a census). They may be right on, they may be off by 2% or 3%, but the most they would be off would be 5%.
So why sample? Why not just take a census?
A census can work if you are dealing with a small group, such as a classroom full of 30 students. But marketing research typically deals with fairly large populations where a sample is more effective, practical, and really the only reasonable option.
A sample is also more accurate because we have more control. If we try to survey an entire population, or an extremely large sample of a population, we increase what is called non-sampling error. Non-sampling error happens when there are too many variations in the sampling process (different interviewers, time, events, etc.). Thus, the margin of error will drop as we increase our sample size, but the likelihood of non-sampling error increases.
When sampling it is important to get equal representation from the demographic being sampled. We need only sample one spoonful of potato soup in order to know that the whole pot is going to taste the same, we are confident that it is all equal. And we certainly would only take a sample of someone’s blood, not a census. If we are looking to ascertain the opinions of high school teachers regarding changing the start time for high schools in the state of Washington, we wouldn’t simply survey a dozen teachers in Spokane and be done with it. We would want to survey teachers from all over the state. Our survey would take into account the population and geography of the target demographic in order to obtain equal representation. The target population must be representative of the sample population.
In the 1936 presidential race, Literary Digest magazine predicted a landslide win for Landon over Roosevelt. However, as we are well aware, it was in fact Roosevelt who won by a landslide. The problem with the Literary Digest poll was that while the target population was eligible voters, they conducted the poll by calling only people listed in the telephone directory. In 1936 there weren’t very many people with telephones and fewer listed in telephone directories. However, Literary Digest managed to poll a total of 10,000,000 people across the country. Because of the sampling frame (telephone owners in the directory) were mostly wealthy republicans, Literary Digest predicted a victory for Landon. Gallop (relatively unknown at the time) surveyed 1,000 people correctly and predicted Roosevelt by a landslide. Literary Digest was out of business in less than a year.
There are several online tools that can be used to determine an adequate sample size. One useful tool can be found at http://www.raosoft.com/samplesize.html. Here you can input the margin of error you are willing to accept, the size of the population you are sampling, and you will get the correct sample size you need. For example: If we are surveying a population of 2,000, we learn that for a margin of error of 9% we need to survey 112, for a margin of error of 6% we need 235, for a margin of error of 5% we need 323, for a margin of error of 2% we need 1,090.