Tuesday, June 12, 2012

Why I Hate Polls and Surveys

I wrote this for a class when I was attending the University of Maryland University College - Asia (whew...can it be any longer?!). The assignment was:

  1. Research the results of any poll or survey in which you disagree with the results.
  2. Describe why you disagree with the results.
  3. Offer a brief explanation of how the survey/poll could have been improved (results, survey base, etc).

"A recent Gallup poll in the United States indicated that around 40% of Americans do not believe that humans evolved from less advanced forms of life. That contrasts to 59% acceptance in Canada, and upwards of 80% in some European countries.", taken from here.

Without getting into the Science vs Religion debate, I can accept these statistics. However, I wanted to look at the survey base to find out how accurate it might be, and this is what I found:

"Results are based on telephone interviews with 1,018 national adults, aged 18 and older, conducted Feb. 6-7, 2009, as part of Gallup Poll Daily tracking. For results based on the total sample of national adults, one can say with 95% confidence that the maximum margin of sampling error is ±3 percentage points."

Really?  1,018 adults surveyed over three days can give a confidence rating of 95% that the margin of error is only "'±3 percentage points."?  If I could skew data to show only what I want to show, then I'll just start referring to Gallup for my references from now on!

Here is what I find wrong with this poll:
  1. They don't list what city the poll was taken in, which DOES matter.
We are talking about the opinion of 1,018 individuals being taken as a reflection of the whole of the United States of America. Because of this, the city that was polled is critical to the data represented.  What if the poll was taken in New York, New York? What if it was taken in Montgomery, Alabama? 

If the city lies in the "Bible Belt", the results are skewed in favor of the Biblical representation of the origin and growth of our species.

 If it is taken in a city with a "healthy" amount of non-religious citizens, it may be skewed in favor of the Darwinian representation.

And if the results are taken in cities where the religious community (of mixed religions, not just Christian religions) and the non-religious community are well-balanced, the results are closer to what they SHOULD be, in order to adequately reflect the opinion of an entire nation based on a miniscule percentage of citizens in a specific city.

But it doesn't state this. We don't know if they polled one city, or a number of cities. They don't give us that information, which is ridiculous! "1,018 national adults" surveyed over a period of three days cannot adequately reflect the opinion of the nation as a whole.  How is this small subset of people, each with their own religious or scientific bias, supposed to constitute "40% of Americans"?

According to the Wolfram Alpha Computational Search Engine, the United States population between ages 18-64 in 2009 was 189.3 million people (189,300,000). 1,018 adults out of 189.3 million adults = 0.000537770734284205 % of the 189.3 million adults. Round up a bit to the nearest millionth for an "easy" 0.000538 %.  That is the percentage of the US population that they surveyed. So I think it's safe to say that we can throw their results into the trash.

What should have been included in the survey was the pool of adults in the city that was polled.  1,018 adults out of how many in the city?  This would give a reference for comparison, a ratio, for comparing it to what the average national opinion might be.  Get the ratio of the people who were surveyed (say, 1018 out of 10,180, which is 1/10th), and use that for the basis of stating what the "National Average" is.  1/10th of the adult population in 2009 is 18,930,000.  Grab that percentage and it would result in a more reliable answer.

I may as well survey a married couple, and say that their opinions reflect that of the entire city. If one is religious and one is not, then "50% of the American populace in Okinawa does not believe in evolution". Or take it a step further, and survey ten families (10 men, 10 women). Assume that it comes out 80% in favor of creationism. Can I then go on to say that "80% of the American populace in all of Asia (China, Korea, Japan, Taiwan, etc) does not believe in evolution"? I could, but my statistical results may very well be wrong.

This is the "Age of the Internet". We do everything online: shopping, socializing, and even find entertainment via a staggering amount of venues. Why are polls limited to such a small number of people taken as the opinion for the nation as a whole? Why not put the survey online, and gather data in this way?

There is always a fear of invalid data (the same person taking the survey multiple times, or a person who enters information as a joke), but invalid data can be given in a face-to-face survey and telephone-surveys as well, so there will always be a certain amount of data which is invalid based on those situations. Surveys and polls which are used to define the opinion of a large mass of people are already hard to retrieve data for, and I understand this. But with a survey that included data from such a miniscule subset of the population, with the location of the survey being undisclosed, lends credence to my opinion that the survey itself is nothing more than a joke.

No comments:

Post a Comment