Sunday, December 2, 2007

Saletan's Mea Culpa: "Whoosh"

Cosma Shalizi has a worth-while follow-up to journalist William Saletan's eugenicist-riddled idiocy and epilogue to said idiocy. Saletan claimed to rely upon published, presumably peer-reviewed writings and rebuttals when doing his "research" for his now-easily discredited series of articles. Sez Shalizi:

But let me back up a minute to the bit about relying on "peer review and rebuttals to expose any relevant issue". There are two problems here.

One has to do with the fact that, as I said, it is really very easy to find the rebuttals showing that Rushton's papers, in particular, are a tragic waste of precious trees and disk-space. For example, in the very same issue of the very same journal as the paper by Rushton and Jensen which was one of Saletan's main sources, Richard Nisbett, one of the more important psychologists of our time, takes his turn banging his head against this particular wall. Or, again, if Saletan had been at all curious about the issue of head sizes, which seems to have impressed him so much, it would have taken about five minutes with Google Scholar to find a demonstration that this is crap. So I really have no idea what Saletan means when he claimed he relied on published rebuttals — did he think they would just crawl into his lap and sit there, meowing to be read? If I had to guess, I'd say that the most likely explanation of Saletan's writings is that he spent a few minutes with a search engine looking for hits on racial differences in intelligence, took the first few blogs and papers he found that way as The Emerging Scientific Consensus, and then stopped. But detailed inquiry into just how he managed to screw up so badly seems unprofitable.

The other problem with his supposed reliance on peer review is that he seems confused about how that institution works. I won't rehash what I've already said about it, but only remark that passing peer review is better understood as saying a paper is not obviously wrong, not obviously redundant and not obviously boring, rather than as saying it's correct, innovative and important. Even this misses a deeper problem, a possible failure mode of the scientific community. A journal's peer review is only as good as the peers it uses as reviewers. If everyone, or almost everyone, who referees for some journal is in the grip of the same mistake, then they will not catch it in papers they review, and the journal will propagate it. In fact, since journals usually recruit new referees from their published authors or people recommended by old referees, mistakes and delusions can become endemic and self-confirming in epistemic communities associated with particular journals.
Shalizi, of course takes great pains to note that he is not in any way, shape, or form trashing the peer review process but merely making it clear that it is a social process - albeit a considerably more rigorous social process than its alternatives. The quality of that process will vary from journal to journal, and even if the journal follows a high-quality peer-review procedure its editors and reviewers themselves can be blinded by the idols of the theater. Hence, some variant of caveat emptor is advised when figuring out whom to cite, etc. Get familiar with the journals that cover the research topic in which you are interested (learn their editors' assumptions, and so forth) - this would be advice given not only to those beginning scholarly careers but also to those outside that particular area of inquiry who wish to cover it for newspapers and magazines. Saletan failed badly in large part because he did not do so.

No comments:

Post a Comment