500 people are enough for a representative sampling of the WHOLE COUNTRY!?

[Bonobo]

I call bullshit.

Yes, I have heard people that claim to understand the arcane mysteries of sampling and statistics, but common sense says it just isn't possible that a survey of 500 people could be in any way, shape or form, accurate.

CNN put up a "nationa;" gallup poll this morning that shows a tight race of 44% to 41% with a margin of error of a few pct. points.

Please, someone, tell me that I'm right in calling bullshit on this. It just makes no sense to me. 500 people? Even if you chose carefully, it strains credulity....

Thoughts appreciated.

[mediaman007]

1. I''m not a statistician, but I can't imagine that 500 people

can represent the consensus of 200 million voters.

[Bucky]

33. I've studied statistics. And yes, within a margin of error, a scientific sampling works quite well.

You sound like a Creationist saying "Well, geeze, I couldn't possibly be descended from a monkey. That's crazy talk!"

Study the science of how it works. Understand taht you depend on the scientific reliability of statistical sampling every time you eat meat or vegetables, every time you ride on a bus or airplane, and every time you take medication for what ails you. The safety and efficacy of all these things that you personally depend on is guaranteed by scientific, representative sampling.

So shut up.

[Stand and Fight]

2. Funny thing is...

I thought that you were talking about Congress at first... Either way, I agree with you wholeheartedly.

Okay, 535 may be over that number, but it's close enough...

[TheKentuckian]

3. I'm thinking a representative sample

should be at bare minimum 1%. I'd tend to say 5% can give you a pretty accurate sample but 500 and 1,000 people is probably too small to get an accurate gauge.

[EstimatedProphet]

12. Why?

[CTyankee]

30. The margin of error

I'm not a statistician either but I do remember learning that the smaller the sample size, the greater the M.O.E. IT makes sense if you think about it...

[EstimatedProphet]

35. All other things being equal, yes.

I see what you mean now. However, in a truly normal population the mean and standard deviation can be estimated pretty accurately, assuming no sampling bias exists. For example, with height as the variable a relatively small random selection (just a few hundred people) can give a pretty precise estimator of average human height for the entire population.

Typical margin of error in these polls is around 4%. That's actually a lot. It's acceptable to the pollsters, though, because they are going for the most bang for the buck, and that comes with repeated polls which cost more money.

[Tarc]

4. Yes, but when most of the polls show the same or similar numbers

http://www.cnn.com/2008/POLITICS/06/24/poll.of.polls.chart/

It is rather reliable, yes.

[Captain Hilts]

7. There's no big 'pollster conspiracy'. They don't work together.

It actually is a specific statistical process.

[Captain Hilts]

5. It must have a large margin of error. Gallup is a lazy pollster that doesn't balance its data well.

[Fluffdaddy]

6. What, now that the polls are tightening up we are choosing not to believe them?

This is going to be a very very close election people. Year 2000 close, it's about time we all realize this.

[grantcart]

8. actually CNN's poll of polls shows Obama with his biggest national lead

Here are some thoughts on minority under representation in a change election


http://journals.democraticunderground.com/grantcart/133

[napi21]

9. 500 people CAN give an indication of multimillions IF the 500

is selected properly! In every business (politics included) there are certain strong indicators that are rarely wrong at predicting a specific outcome.

ie: On a MUCH smaller scale, when I was the Cash Manager of a retailer with 2,000 retail locations, one of our jobs every Monday AM was to transcribe the weekly sales info for the entire chain. I had a staff of 4, but it still took 3-4 hours to transcribe all those calls and get a printout and a total #. As you can imagine, Sr. mgmt would begin stopping by my office about 30 mins into the process, all looking for a prediction. It didn't take me too long to realize that the total week's sales # could fairly easily be predicted (and be pretty close to accurate) by the sales at 3 stores. These 3 stores were in 3 diff. parts of the country, and I never did figure out WHY this always worked, but it did. I started giving early "guesse" after finding out the info on those 3 particular stores. THEN I'd have those same Sr. mgrs asking me "How do you do that?" I never told them. I'd just smile and say "Caus I'm GOOD!"

I have no idea how GOOD these polsters are at picking their 500 people though.

[Jackpine Radical]

10. All polling hinges on the accuracy with which the sample represents the population.

Many things can interfere with getting an accurate sample. For example, if you do phone sampling, you get only those with landlines, who hapen to be at home to answer their phones. You can try to stratify the sample--e.g., if you find too few young people in your draw, you can go out & try to find more young people. But how do you know that the young people you find are representative of young people as a whole? It depends on how you found them.

The ideal conditions of a sample are met when each person in the target population (e.g. US voters) has an equal chance to appear in the sample. In reality, these conditions are never met. The reason one poll differs from another (beyond simple random error) is that the various pollsters have different, proprietary ways of compensating for the discrepancy between their sample and the ideal sample.

A poll of 500 can accurately estimate the population values to the extent that it meets the ideal criteria. Given that assumption, you can predict those margins of error rather simply:

Standard error=sqrt(p*(1-p)/n)

and Margin of Error (2 tailed, .05)=1.96*standard error.

[EstimatedProphet]

11. It can be, if the trends are very defined

The pollsters are making assumptions that trends in candidate choice are well-defined, which I think is for the most part a reasonable assumption. For why this works:

http://www.stat.berkeley.edu/~stark/Java/Html/lln.htm

[Kurt_and_Hunter]

13. Your "refutation by personal incredulity is noted"

500 is a little low by the modern conventions of polling, so your question is cleverly framed, but 1,400 is generous and adding another 10,000 to that sample would do little to make a poll more accurate.

And in terms of seeming plausability, 500 and 1400 are both sub-microscopic slices of 300 million people.

Whether a scientific reality strikes you as personally plausible is not probative.

You can also call bullshit on the idea that feather and a bowling ball fall at the same rate in a vacuum, or that the Earth goes around the sun.

Both are roughly as surprising as the fact that the reliability of polls essentially maxs out at surprisingly small sample sizes.

There was a time when "calling bullshit" on basic precepts of mathematics would have been recognized on DU as equivalent to "calling bullshit" in the idea the Earth is round, but those days seem to be fading.

[drm604]

14. Statisticians will disagree with you.

If the sample is truly random then, with a sample size of 500, a survey will have a margin of error of plus or minus 4 percent at a 95 percent level of confidence. This is true for any large population regardless of the size of the population.

Here's a discussion of this: http://www.isixsigma.com/library/content/c040607a.asp

Margin of error – the plus or minus 3 percentage points in the above example – decreases as the sample size increases, but only to a point. A very small sample, such as 50 respondents, has about a 14 percent margin of error while a sample of 1,000 has a margin of error of 3 percent. The size of the population (the group being surveyed) does not matter. (This statement assumes that the population is larger than the sample.)

Here's a much more mathematical discussion for those who want to tackle the math: http://en.wikipedia.org/wiki/Sample_size

Of course there are caveats. The sample must be truly random, the questions must not be slanted, etc. Those are the things that will be manipulated if someone wants to skew a result.

Do you have a link to that survey?

[AllentownJake]

18. +-4

They would need 600.

[drm604]

23. I was going by the chart on isixsigma.com

but they may be off a little. The point still remains that in a properly done survey 500 is representative, depending on what you consider to be an acceptable margin of error.

[endthewar]

27. Here is the table that you should use to understand election polling:

http://www.washingtonmonthly.com/archives/individual/2004_08/004536.php

[slackmaster]

15. Yes, that is enough if the sample is properly stratified

Take a basic statistics course, or just read a textbook for one.

[AllentownJake]

17. Actually it isn't

See my post below.

1067 would be an appropriate sample so the sample size would need to double at a 95% +-3

[slackmaster]

19. Yes, but for 95% +- 5% out of 100 million you need only 383

500 MAY be enough depending on what confidence interval and degree of precision you are trying to achieve.

Because of the difficulty of selecting a truly representative sample from the real-world population of US voters, even what seems like a ridiculously large sample set can still give inaccurate results.

The only poll that really matters is the election.

[AllentownJake]

20. +-5

Is a ridiculous metric for a political race. That means they could be 10 points off. +-3 is about the norm.

[endthewar]

25. "That means they could be 10 points off."

Again, get a refund from that class and look at this table instead to understand what margin of error means for election polling:
http://www.washingtonmonthly.com/archives/individual/2004_08/004536.php

[slackmaster]

32. +-3 doesn't seem much more useful to me

Considering how close the popular vote has been in the last two elections.

[endthewar]

24. "95% +-3"

Better get a refund on that statistics class. :rofl:

Here's an easy to understand table for election polling: http://www.washingtonmonthly.com/archives/individual/2004_08/004536.php

I wrote a journal article about it a while ago, but I don't have it anymore, so I hope that the table is easy enough to understand.

[AllentownJake]

26. Sorry I do audit sampling

500 people of a sample of over 120 million potential voters is a no sale for me.

[endthewar]

29. Further explained

Statistically speaking, 500 people yields a margin of error of about 4%. This is something that is calculated by a mathematical equation that can't be disputed. So from a statistics standpoint, 500 people is considered "sufficient."

From a polling standpoint, you do have a point, but not about the sample SIZE but perhaps about the sampling METHOD. Criticisms of the sampling method would be the same if one sampled 500 or 500,000 people. For example, the most common criticism of sampling methods for this election is LAN lines vs. cell phones. Many people believe that people who only use cell phones and don't have a LAN line tend to be younger voters. This group is leaning heavily in favor for Obama, by a double-digit landslide. So if the sampling METHOD selectively excludes them, then Obama supporters will be undersampled.

So to sum it up, the appropriate criticism is not about the sample SIZE but instead could be about the sampling METHOD.

I should stop now, or else I'll keep rambling on, since I love talking about math and am a doctoral student in engineering right now. :rofl:

[AllentownJake]

28. BTW better read your own article

"Generally speaking, national polls use sample sizes of about 1,100, which translates to an MOE of 3%. State polls often use a sample of 600, which produces an MOE of 4%. Subsets of polls sometimes have MOEs of 5% or higher."

So the +/- 5 is not really used that much in national polling.

[endthewar]

31. The +/- 5 has NOTHING to do with the INTERPRETATION of an election poll

Surely you realize that if a candidate is ahead by 20% people would be comfortable with a small sample size with only a margin of error of 5%, right?

The real question is, "What is the percent likelihood that Candidate A is truly leading Candidate B?" You can use that table to answer that question. For example, if a poll has a 5% margin of error and the one candidate is ahead by 3%, (from the table) that candidate has a 73% chance of TRULY leading the other candidate.

So the poll really tells you the percent likelihood that a candidate is ahead. Is a 73% chance of being ahead make that candidate's strategist feel comfortable? Depends on the strategist. Everybody has their own cutoff on percent likelihood that a candidate is leading in order for them to feel comfortable that the candidate is truly ahead.

[AllentownJake]

16. Enjoy the Sample Size Calculator!

http://www.isixsigma.com/offsite.asp?A=Fr&Url=http://www.surveysystem.com/sscalc.htm

[suston96]

21. Accurate? Which pollster ever claimed that their polling is "accurate"?

No pollster has ever claimed "accuracy" in polling.

[endthewar]

22. Very important table to help understand polling data

500 is a large enough sample size. However, the REAL question is whether or not all groups are represented in that sample size. For example, does the polling method under-represent young voters because they are less likely to have a LAN line and only a cell phone? You should focus on things like that instead of the sample size.

Here is a great link to better understand polling data: http://www.washingtonmonthly.com/archives/individual/2004_08/004536.php

I'm convinced that 99% of the American people don't understand margin of error, so take a look at that table. A 4% margin of error in a poll with a 3% difference between the candidates translates into a 78% likelihood that one candidate is leading the other.

Also, as a rule of thumb, anyone who says that a poll is a "dead heat" because it's "within the margin of error" doesn't understand the first thing about polling statistics. I could write a more detailed explanation, but DU isn't paying me to educate everybody now. :rofl:

[Bucky]

34. You are wrong in calling this bullshit.

I've studied statistics. And yes, within a margin of error, a scientific sampling works quite well. To deny it makes you sound like a Creationist saying "Well, geeze, I couldn't possibly be descended from a monkey. That's crazy talk!"

Study the science of how it works. Understand taht you depend on the scientific reliability of statistical sampling every time you eat meat or vegetables, every time you ride on a bus or airplane, and every time you take medication for what ails you. The safety and efficacy of all these things that you personally depend on is guaranteed by scientific, representative sampling.