Reply #10: OK, here's a response [View All]

[Febble]

it's a bit long, and it's taken me a while, but FWIW:

First the good news:

I do think Steve makes a good case for investigation. I agree with him that Ohio stinks, voter suppression is rampant, whether structural or intentional (and it is hard not to believe intentional), and that the exit poll discrepancy cannot be due to chance. Given the first two, it is reasonable to suspect that the exit poll discrepancy may have been due to fraud.

And I like his term "Precinct Level Discrepancy"(PLD) “Within Precinct Error” (WPE) has become synonymous with a very particular measure of PLD, which I think Steve agrees is not the best, although inevitably it is the one that will feature in his analyses as it is the one used in the E-M report. And it is important as it appears that precinct level was where the red-shift occurred.

So, thus far I’m with Steve. The case for investigation is good, and PLD is where the bodies appear to be buried.

Steve also presents some interesting observations. Although state level aggregates are difficult to interpret, it is of interest that completion rates show a bit of a tendency to be higher in redder states, and that PLD appears to be higher in states with higher African American populations. (I identified a possibly related pattern which is that WPE was higher in bluer states). State level is too coarse a grain with which to investigate potential causal factors in these relationships, but they are, nonetheless, interesting, and worthy of further investigation.

Steve also says in one slide that the only two causes of PLD are "non-response bias" and "count corruption". I agree with this as long as you take a fairly broad definition of "non-response bias" to include actual sampling bias i.e. people who do not respond because they are not even approached. There is evidence that the actual sampling may have been biased (PLD was greater where sampling rate was low). This point is important in interpreting some of his other observations.

Because here is where Steve and I start to part company.

He then goes on to say that interviewer effects cannot account for all the discrepancy - although he then says that it cannot be ascertained without looking at the data. This latter thing is true. You need a multiple regression model to determine whether interviewer effects can or can't account for all the PLD. But from what is publicly available we do not know whether it can or can't. And it is also true that some polling factors may be collinear with fraud. So although I accept that with the data publicly available we cannot be assured that interviewer effects account for all the discrepancy – without the data we can also not be certain that they do not. There is a difference between saying that “interviewer effects cannot account for all the discrepancy” and saying that “on the data given, we cannot know whether effects account for all the discrepancy”.

The completion rate by precinct partisanship argument is of interest, but IMO not at all conclusive. At the risk of sounding like an apologist for the exit polls, there are many reasons why the completion rate patterns are perfectly consistent with non-response bias as an explanation for the exit poll discrepancy (particularly if this is taken to include sampling bias). Firstly we know, from a scatterplot presented by Mitofsky at AAPOR (and again at this debate) that variance in completion rates was vast, and neither negatively nor positively correlated with Bush’s share of the vote. This means that staring at aggregate values tells you very little about whether those aggregates are meaningfully different. Secondly, total completion rate may vary independently from differential completion rate. For example, voters of from both groups may be more likely to respond in, say suburban than urban precincts; however, for a given total completion rate, there may nonetheless be a tendency for Bush voters to respond less. Thirdly, non-response bias may operate at the level of voter selection rather than voter refusal. If Bush voters were less likely to be selected, but equally likely to participate IF selected, completion rate would not vary by precinct partisanship; red-shift would nonetheless be manifest in the PLD. And the fact that PLD was greater where interviewing rate was lower (and therefore where there was more opportunity for non-random sampling) strongly suggests that bias in voter selection was a factor in producing redshift.

Steve presents a plot in support of his claim of “implausible” Kerry response rates which is simply wrong. I believe it was produced by Ron Baiman for UScounts votes, and plots a calculated value for Kerry and Bush response rates based on a formula that can be readily shown to give “impossible” values for response rates for voters for the minority candidate under conditions of sampling error alone. A full rebuttal of this plot is way to geeky for this post, but if anyone’s interested I can demonstrate that Ron’s formula can easily generate computed “response rates” for voters for one candidate that exceed 100%, even if the only source of error is sampling error. And it is not possible to conduct a poll without sampling error.

And now we come to the apparent rise in PLD in “high Bush” precincts. This is, I believe, an artefact of the WPE as a measure, and of the instability of aggregate measures where Ns are relatively small. I believe that Steve agrees with me that the measure I proposed, and called ln(alpha) is a better measure. However, even when WPE is used as the measure of PLD, Steve’s statement that “there’s no PLD at all in the Kerry strongholds” is simply untrue – the mean PLD may be zero in the Kerry strongholds, but there is plenty of it. Moreover if ln(alpha) is used as the measure, it is clear that the only sense in which PLD is”lower” in Kerry strongholds than in Bush strongholds is that in the high Bush category (incidentally, not “quintile” – which would imply equal numbers in each category) is that there is a shortage of extremely blue shifted precincts – and in fact one very blue-shifted precinct only escapes the arbitrary “high Bush” category line by a single percent. And in fact, as Mitofsky demonstrated at the Miami AAPOR meeting in May, there is no significant tendency for PLD to be higher at the Bush end of the spectrum than at the Kerry end. The correlation coefficient between ln(alpha) and Bush’s vote-share is insignificantly different from zero.

A few more miscellaneous comments:

The swing state thing is interesting. Could be evidence of fraud. Could be a spatial manifestation of the temporal phenomenon suggested in the E-M report whereby red-shift appears to be greater in years in which interest in the election is high.

Voting technology. I simply don’t think anything can be drawn inferred from this. As Steve draws attention to, the number of urban precincts with paper ballots is absurdly small – too small to do any statistics on. As with many of these analyses, it’s kind of interesting, but you need to know what is collinear with what else to draw any conclusions.

Reported Vote: Steve seems to have cottoned on to TIA’s case. My response is the same as my response to TIA’s. I won’t repeat it here, except to say that it strikes me as being in no sense a slam dunk for fraud, despite popular belief.

Confidentiality. I simply disagree with Steve that the data should be publicly released. Quite apart from being a violation of the ethical guidelines of the professional organisation of pollsters, it is, simply, unethical to make data public in a form in which respondents could be identified – and here we are talking about the sanctity of the secret ballot. And I also believe that E-M owe their interviewers (casual employees) a duty of care. If precincts can be identified, so could interviewers. The demographics of both responders and interviewers are important variables. It is simply wrong to allow these variables to be released in a way that make either identifiable. “Blurred” data on Ohio was released. Perhaps other sets could be similarly prepared and released. But confidentiality matters. I would be dismissed from my own job if I published data in a form in which my respondents could be identified. You just can’t do it.

So to summarise my response to Steve’s talk, for anyone who is interested, and doesn’t think I’m a freeper:

Freeman and Mitofsky agree that there are only two possible causes of the Great Exit Poll Discrepancy: Fraud and Non-response Bias (and I’ll qualify that by including selection bias in Non-response bias).

So whodunnit? Fraud or Non-response Bias:

Steve, as a good prosecuting counsel builds a nice case for Fraud as the killer of the Exit Poll. His evidence that a crime was committed is sound. His evidence that Fraud was the culprit is suggestive. Some is flawed. Some is wrong. But regardless of which is which, it is built into a solid-looking edifice. Things look bad for Fraud.

But even solid looking cases can come apart with a watertight alibi. And I think Mitofsky as defence counsel presented the alibi for Fraud.

Here it is:

If Fraud was responsible for the Great Exit Poll Discrepancy, then where the PLD is greatest, the boost to Bush’s vote should be greatest. After all, the point of fraud is to win the election. Cui bono?

If Non-response Bias was responsible for the Great Exit Poll Discrepancy, PLD should vary independently of Bush’s advantage in the vote count.

Mitofsky correlated a measure of PLD with a measure of advantage to Bush, namely the increase in Bush’s share of the vote relative to 2000 (a year in which mean PLD was near zero).

And there was no relationship. PLD is not correlated with boost to Bush’s vote share.

So, carefully as Steve’s case was built, I simply don’t think it stands.

Of course Fraud could still have stolen the election. But it looks to me that with regard to the Great Exit Poll Discrepancy, he’s innocent.



(edited for grammar and typo - I'm sure there are more)