University of Hypocrisy

Today, Mark Blumenthal reports that the American Association of Public Opinion Researchers (AAPOR) censured Dr. Gilbert Burnham who is one of the authors of the Iraq War casualty paper.

Burnham was censured for not disclosing all of the study materials that were requested by AAPOR in their investigation, although it seems like he turned over just about everything. One of the main pieces that AAPOR claims was not turned over (though the allegations of what Burnham actually failed to be turned over is nowhere in the AAPOR press release) is the question wording. But, it seems (as Blumenthal notes), he did release his question template. Granted, it is not a survey instrument; but, I wonder if AAPOR has an Arabic translator to read the actual survey instrument -- and if so, how many members of AAPOR would be able to understand it. In response to the press release, at least two people (Blumenthal and Science blogger Tim Lambert) asked AAPOR to clarify what, exactly, Burnham failed to release to AAPOR. The answer

We do not provide public reports of the investigations but if there are other specific questions that I could answer, I would be happy to try to do so.

Basically, Burnham gets censured for not turning over all of his materials for review by an organization (of which, it should be noted, Burnham is not even a member) that cannot be bothered to turn over their own evidence used in the report. I have a lot of respect for AAPOR, and think that they did some great things during the election—aided tremendously by Mark Blumenthal—to make sure that the public and journalists knew how to interpret polls, but this is just absurd.^[1] What's good for the goose...

I am all for more transparency in data. I think that more needs to be done in social science (and public health) to deal with replication and increasing access to data^[2]; however, for an organization to claim that they can censure a non-member for ethical violations and then turn around and violate the same spirit of their rule strikes me as something members of AAPOR (I am not one) should address.

^[1] As a side note, I should mention that there are some problems with the Iraq studies. Their response rate is very high, particularly since they only went to sampled houses on a single day (although this National Journal summary quotes famous soc blogger Kieran Healy noting that this response rate is not unusual for Iraq) and, because of their sample size, the confidence intervals are very, very large. The appropriate venue for such debates, however, is in the academic press to challenge the validity of the findings, not for some organization to nitpick on violations of ethics of non-members.

^[2]In fact, a recent special issue of Sociological Methods & Research pertains to this very issue. I highly recommend the issue of the journal for many of the interesting points that are raised.