Polling expert finds major problem with poll that found Edwards ahead in Iowa

[antiimperialist]

I was happy, being an Edwards supporter, when I heard that a recent Insider Advantage poll had found Edwards ahead of Obama and Clinton in Iowa. But at the same time I felt surprised and skeptical.
Anyway, today Mr. Mark Blumenthal, a veteran polling expert, analyzed the crosstabs from this poll and found that,

The second problem is potentially bigger. InsiderAdvantage typically emails us a few pages of cross-tabulations that we have sometimes posted to the site, but which they rarely post to their own site. We did not receive those crosstabulations for today's survey, perhaps because of the story I am about to share. The site RealClearPolitics has posted a more limited version for the Republican and Democratic results.

Take a look at the Democratic tab, and if you look closely, you'll see the problem: According to the crosstabs, Barack Obama gets 19.6% of the vote from men, 17.8% from women but 24.3% from all voters. Needless to say, that result is impossible, especially since they report 392 interviews conducted among men, 585 interviews among women and 977 overall (and since 392+585=977).

This problem is present for Obama and not Edwards and Clinton, but the mere fact that the crosstabs make no sense casts doubt upon this poll

link to analysis by Blumenthal: http://www.pollster.com/blogs/the_insider_advantage_crosstab.php

[calteacherguy]

1. K&R. nt

[rinsd]

2. Insider Advantage has some weird demographic samples

Discussion of undersampling 17-44 year olds in IA poll here - http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=132x3859457

Discussion of undersampling black voters in SC poll here -

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=132x3818678#3818736

[K Gardner]

3. This is fascinating and only enforces the absurdity of relying on polling data

and, more absurdly, of trying to guess a caucus or predict eligibility based upon skewed polls. I do not understand the blind faith placed in some of these poll numbers, when most reasonable people here KNOW the MSM drives the news, KNOW that votes and ergo polls can be rigged, and KNOW that corporations have a say in what we see. Thank you for posting.

[patrice]

5. Rigging:

Is it possible that there's sooooo much fine-tuned information out there now that poll results can be reverse engineered and depending upon how the population from which the samples are drawn is defined, you can pretty much cover your tracks or at least disguise them?

[K Gardner]

10. I'm no statistician. But I do know that very few people take the time to delve

into the bare bones of polling data. As is so evident here on DU, some people will believe almost anything if someone types it out - without taking the time to research or verify it. I don't find it a stretch to think our obsession with "polling data" is a manufactured phenomenon that can be manipulated at will.

[patrice]

16. In my graduate work, we talked about the use of numbers in describing people,

i.e. quantitative descriptions, compared to other more semiotic information, referred to as "qualitative" information and frequently obtained in more in-depth interviews.

I think of the qualitative data as representing those meta-phenomena of which the quantitative info, e.g. polling data, is but a wide variety of sub-manifestations within their particular times and settings.

The candidate who can reliably access valid meta-qualities wins.

[Think82]

4. Obama gets a majority of transsexuals.

: }

[Quixote1818]

6. This was an interesting comment at the link

It seems Insider Advantage nailed Iowa in 2004:



>>>>You seem to have a real thing for this one company. Did you read Bob Novak's column of January 19, 2004? InsiderADvantage used this same process and nailed the Iowa caucus. You need to do a little research yourself before you do damage to a company and get yourself sued. I dare you to post this. By the way, Novak's column ran right before the caucus of '04. " InsiderAdvantage, which previously has polled mainly in the South, says the contest may not actually be that close in Iowa. Calculating second-choice preferences that may be decisive in the complicated caucus system, the poll gives Kerry 33 percent of the actual caucus vote to Dean's 26 percent." Robert Novak Chicago Sun Times January 19,2004/Creators Syndicate. Hey mystery pollster, have you got the guts to apologize or even post this. Bet not.

[antiimperialist]

7. But did they have impossible statistics in 2004?

That is the question. you can't have a total support that is higher in average than both the men's and women's support.

[Quixote1818]

8. Yes you can. It depends on how the poll needs to be weighted

You said: "That is the question. you can't have a total support that is higher in average than both the men's and women's support."

Yes you can. It depends on how the poll needs to be weighted


They take the stats they got and compare them to the actual # of people in each age group and gender group tend to vote.

Obama got 35.3 and 35.2% of the younger vote 18-44 years old, but they were a very small part of the sample probably because younger voters only have cell phones. So what they did was to calculate how many younger voters there are in NH compared to the older voters and weighed that in because he does so well with younger voters and they were such a tiny part of the sample only (34)(88) people out of 977 polled.

Take a look: http://www.realclearpolitics.com/docs/InsiderAdvantage_Majority_Opinion_Democratic_Iowa_caucus_poll.html

[antiimperialist]

9. Age and gender are two different things

You forgot to equate the vote of people older than 18-44; on the other hand, not only did Blumenthal categorically said Insider Advantage's total is impossible, but we can easily verify it because there are only two genders: male and females, and it's impossible to say for example, that 50% of people prefer the color blue over yellow, if only 20% of women and 15% of men prefer blue, and this is regardless of the distribution of men and women.

[Quixote1818]

11. No I didn't. You don't understand weighted polls

Look at the sample size for each age group:

There were ONLY 122 people sampled age 18 to 44. (Obama did really well here)

There was a HUGE sample of 855 people older than 44. (Obama did poor here)


Age 18 to 44 was only 12% of the total number polled which was 977 people. That means you have to add weight to the sample of 122 people between 18 to 44 and take away weight from the sample of the 855 people older than 44. Since the 18 to 44 age group was so badly under sampled and Obama did so well in this age group then it makes perfect sense that once weighted he would do much better than what the tab columns say. Gender has zero to do with the final result because the poll needed to be weighted.

[antiimperialist]

12. Give me a scenario in which total support can be higher than both male and female support

You pick the weight. Pick any number for male and any number for female. Then use the percentages found in the insider advantage poll. It will be impossible for you to demonstrate that Obama can have total support higher than both his support among males and support among females.

[Quixote1818]

13. It looks odd because

The sample of younger voters was so small (12%) and went for Obama in such huge numbers (35%). Normally, the tabulated total won't look so strange but they obviously couldn't get hold of the age group that obviously LOVES Obama.

Say I polled 100 people and 90% of them were over 50 years old. In reality perhaps 50% of the electorate is under 50 so the poll needs to be weighted in a big way. Lets say Obama gets 100% of those under 50 and zero % of those over 50. Thats only 10 votes out of 100 people polled. So the tally will say that Obama only got 10% regardless of gender. So even though the % of men and women votes for Obama would be 10% thats not the REAL number he got from men and women because those numbers are not weighted to the real % of each age group I gave above.

Now lets weigh the poll and give that 10% under 50 years old it's real % of the population which is 50%. That means we need to multiply Obamas number by 5 to equal 50%. So even though he only got 10% of the votes, once it's been weighted then he ends up with 50% of the votes based on the actual number of people under 50 years old.

As far as Male and Female support, that needs too be weighed to however in this case not nearly as much as the weight that needed to be given to those in the younger age group. It doesn't matter how many women and men are in the poll if you have to weigh other parts in a big way. Lets say in this poll he got 4% from women and 6% from men. Thats what the tabulated score will say which looks bad! But if you multiply those # to equal the weight based on age then his real numbers are 20% from women and 30% from men but those won't show up that way in the chart because those are not weighted in the chart they display.

[Adelante]

19. Does this make a difference in your thinking, Q?

(Weighting isn't something I have a good grasp of, I should say.)

They gave me a number for the statistician that weights the data. I called, but heard nothing back, then got caught up in our office move and other more pressing stories. I finally heard back yesterday from Jeff Shusterman, the president of Majority Opinion Research (the company that conducts the InsiderAdvantage surveys) and he confirmed what should have been obvious to Towery and Reese: Only the total column in their crosstabs is weighted. Thus, for reasons that still perplex me, they choose to leave the columns for subgroups unweighted.

Before posting this item, I went back to Towery and Shusterman and asked for an explanation of the purpose of releasing weighted values for all respondents, but unweighted results for subgroups. Here is Shusterman's answer:

The purpose of the InsiderAdvantage/Majority Opinion polls are to provide a snapshot for major media outlets of the race at the time of polling and, as the election day approaches, to accurately predict the outcome of the election for which we have a substantial record of success. This snapshot and eventual prediction are contained in the total column of the cross-tabulations, which is accurately weighted. By contrast, our polls are not conducted to advise campaigns or to provide interesting subtext for academics or bloggers, so we do not weight or place emphasis on the other banner points.

If that's the case, I am not sure I understand why they choose to run "inaccurate" cross-tabulations at all, much less send them to us and to RealClearPolitics. Readers ought to take all of the this "interesting subtext" into account when trying to decide which polls to rely on (and we will save for another day the issue of what weighting up subgroups by factors of three or more does to the reported "margin of error").

http://www.pollster.com/blogs/the_insider_advantage_crosstab.php

[progressive_realist]

18. Self-delete

Self-delete for not enough :donut:

[JDPriestly]

14. That was immediately noticeable when you looked at the poll.

Not knowing about polls, I just wondered what was going on. Must have been a typo.

[Quixote1818]

15. No look at the numbers closely before they were weighted

Obama got 35.3 and 35.2% in the 18 to 29 and 30 to 44 age groups but those age groups were only 12% of the sample. What it looks like to me is they couldn't get hold of the younger voters (who love Obama) because they all have cell phones so they had to give incredible extra weight to those age groups.

Those 18 to 44 are certainly much bigger than 12% of the voting block and since he got 35% of the voters from 18 to 44 his numbers came out to 24% after the younger vote was weighed in properly.


http://www.realclearpolitics.com/docs/InsiderAdvantage_Majority_Opinion_Democratic_Iowa_caucus_poll.html

[cascadiance]

17. Perhaps the way they polled stating one's gender wasn't what they had from all participants...

Might be kind of flawed in terms of having complete sets of data in terms of who they gather data from, but if a huge chunk of them didn't provide their gender to the pollsters (for whatever reason that might have happened), those voters might have substantially not supported Obama, and thus you could have got the skewing of statistics you see here, since those not identifying their gender wouldn't have had their votes counted when looking at the support from different genders.

[maximusveritas]

20. Not to mention

The person running that company is a conservative Republican.