Opinion Polls - Fact or Bunk?

[MethuenProgressive]

Any good books on the subject?
I just can't see how 1000 not-quite-random Egnlish speaking people, with their answering machine on "off", who are home at supper time yet with the time to answer a dozen long questions, who are of unknown education background and honesty - can be so depended upon by policy makers!
:argh:
Seriously, any good books on the subject?

[depakid]

1. Any basic statistics or qualitative and quantitative methods text

that covers the methodology of random sampling, validation of instruments and discusses the various forms of bias will clue you in.

Not sure if there's a specific book out there (I'd bet there is).

Bottom line though, if you've taken some stats courses and/or done research, you'll see how laughable and easily manipulated most of these cheap media polls are.

[MethuenProgressive]

2. I know a bit about random samples

from working in quality assurance in a huge factory for twenty years. The samples must be as equal as possible - you must be sampling the same thing done the same if you want the study to be valid. And if your studying something with huge numbers, your sample's value increases with the size of the sample. That's two of the reasons I just can't see the validity of over the phone polls.

[depakid]

4. Yep- and there are also serious problems with clustering

Not to mention bias.

There are very specific procedures you have to use to ensure you get a random sample from a representative population- otherwise, even with the best of intentions, you end up with "selection bias."

For example, you could wait at a random (and rotating) bunch of corners around a city and ask every third person who went by a question. However, if you only did this 9-5, you'd be missing sets of people who may share certain characterists that are different from the 9 to 5'ers.

Similar things happen with telephone surveys. In the past- classic blunders occurred because not everyone had telephones. The famous "Dewey Defeats Truman" headline came about partially because of that.

Interestingly enough- that year, local "popcorn polls" taken at movie theaters predicted the outcome more accurately- and so did the "feedsack poll."

In Kansas City, a feed supply company conducted their own informal poll by putting donkeys and elephants on the feed sacks, giving farmers the chance to register their preferences by which sacks they purchased. LOL. Silly, yet in this case it worked!

Even though neither of these were "random samples" in the statistical sense- they managed (maybe through luck) to get it right. There's no real way to know- so they're not "scientific."

We have similar problems today with telephone polls. Some people- especially younger people, may not have landlines- they use cell phones (selection bias). Others have taken to screening their calls- and so won't pick up (non-response bias).

These people may be like the people who didn't have telephones in 1948- their preference was to vote for "a common man" like themselves- yet they were systematically weeded out of the survey. Today, my guess would be the bias would be skewed toward more traditional and older people, who I would guess have more conservative leanings. It seems to me -they'd be more likely to pick-up and talk to you.

There are also problems (and aveunes for manipulation) in how you do your clustering- and since the sample sizes are pretty small relative to the population- getting your clustering wrong- skews your data.

Campaigns (and researchers) know this- so they pay big money- and/or engage in may hours of preparation before doing their surveys or "internals."

Cheap media polls, on the other hand- LOL.

They're typically used to influence public opinion- not reflect it- although they may be useful for seing trends over time.

[C_U_L8R]

3. The Old Republican Bible

"How To Lie With Statistics" by Darrell Huff (1954)

http://www.amazon.com/gp/product/0393310728/102-1354219-2288104?v=glance&n=283155

[unblock]

5. you'd be amazed at how effective 1000 is as a sample size.

think of it this way: if you sampled 1000 atoms randomly out of the pacific ocean, odds are REALLY, REALLY GOOD that you'll show 2/3 of them are hydrgen and 1/3 are oxygen. yes, there are contaminants and salt and organisms and all sorts of other things, but mostly it's water, and a random sample of 3 molecules might show anything, but 1000, it's VERY hard to get it wrong, +/- 3%.

the key is making sure your sample really is RANDOM, and you did point out a few of the problems, but the sample size itself isn't likely one of them.

[TahitiNut]

6. It's tough to explain probability to folks who play lotto.

:shrug: I play lotto, too. :evilgrin:

[Gormy Cuss]

9. So true.

}(
I went to Vegas with family and was the party pooper because I viewed the money gambled as money spent on entertainment, rather than as an investment in search of that big win.
}(

[annabanana]

11. That is the ONLY way to guarantee a good time..

When a win comes along, it's frosting.

[Clark2008]

14. That's the ONLY way to view it, in my case.

I have no luck at all - none what-so-ever; therefore, it never enters my mind that I could "win big."

Actually, I don't even gamble at all anymore. My state just more than a year ago finally allowed the lottery (over the protests of fundie nuts who thought their religion should dictate to everyone) and I haven't ever purchased a ticket. Waste of my entertainment dollar, to me. I'd rather put that buck or two toward a good six-pack or a dinner out.

[Greyhound]

13. I call lotteries "the I don't understand math tax".

:rofl:
OTOH, I do throw a buck away every now and then.

[Deja Q]

7. Relevant but dependent on two conditions:

1. EVERYBODY in the US is polled.
2. The US is seen as a country and not a rusty cog in the new world order known as globalization.

[moondust]

15. EVERYBODY?

That would seem to suggest that a poll rating Bush conducted in Oklahoma or Idaho could have vastly different results than one taken in San Francisco. Ha!

I suspect that some if not all political pollsters may tweak their sampling by zip code or area code to produce desired results. I can't prove it.

[Gormy Cuss]

8. Here's a link on polls to get you started.

Some polls are easy to spot as stinkers. Some others can be categorized as suspect if you understand simple statistics or survey methodology. I can't give you a single reference on the subject, but this article is a good start for the layperson.

20 Questions A Journalist Should Ask About Poll Results

Third Edition
Sheldon R. Gawiser, Ph.D. and G. Evans Witt

Polls provide the best direct source of information about public opinion. They are valuable tools for journalists and can serve as the basis for accurate, informative news stories. For the journalist looking at a set of poll numbers, here are the 20 questions to ask the pollster before reporting any results. This publication is designed to help working journalists do a thorough, professional job covering polls. It is not a primer on how to conduct a public opinion survey.

The only polls that should be reported are "scientific" polls. A number of the questions here will help you decide whether or not a poll is a "scientific" one worthy of coverage – or an unscientific survey without value.

Unscientific pseudo-polls are widespread and sometimes entertaining, but they never provide the kind of information that belongs in a serious report. Examples include 900-number call-in polls, man-on-the-street surveys, many Internet polls, shopping mall polls, and even the classic toilet tissue poll featuring pictures of the candidates on each roll.

...

The results of the well-conducted scientific poll provide a reliable guide to the opinions of many people in addition to those interviewed – even the opinions of all Americans. The results of an unscientific poll tell you nothing beyond simply what those respondents say.

By asking these 20 questions, the journalist can seek the facts to decide how to report any poll that comes across the news desk.

http://www.ncpp.org/qajsa.htm

[TahitiNut]

10. Q: Is an election "an unscientific survey without value"??

:evilgrin:

After all, it's NOT "scientific" since those who express their opinion (vote) are not randomly selected from the population of all citizens of voting age, but self-select.

[Gormy Cuss]

12. As a matter of fact, it is an unscientific survey

when only 40% of those eligible bother to voice an opinion and self-select. It has no value as an opinion survey but as we know it has high value in terms of effect on policy decisions.

Maybe we should hand out lotto tickets to everyone who votes as an incentive.}(