Tuesday, October 30, 2007

Estimates of the Iraqi death toll: a follow-up

Discussions on the merits of the Lancet study (published in the fall of 2006):

Huge rise' in Iraqi death tolls

An estimated 655,000 Iraqis have died since 2003 who might still be alive but for the US-led invasion, according to a survey by a US university. The research compares mortality rates before and after the invasion from 47 randomly chosen areas in Iraq. The figure is considerably higher than estimates by official sources or the number of deaths reported in the media.It is vigorously disputed by supporters of the war in Iraq, including US President George W Bush.

Johns Hopkins Bloomberg School of Public Health estimate that the mortality rates have more than doubled since the invasion to overthrow Saddam Hussein, causing an average of 500 deaths a day. In the past, Mr Bush has put the civilian death toll in Iraq at 30,000, and hours after details of the latest research were published he dismissed the researchers' methodology as "pretty well discredited".

The Johns Hopkins researchers argue their "cluster sample" approach is more reliable than counting dead bodies, given the obstacles preventing more comprehensive fieldwork in the violent and insecure conditions of Iraq. "I stand by the figure that a lot of innocent people have lost their life... and that troubles me, and it grieves me," Mr Bush told reporters at the White House. "Six-hundred thousand or whatever they guessed at is just... it's not credible," Mr Bush said.

Sharp rise

The researchers spoke to nearly 1,850 families, comprising more than 12,800 people in dozens of 40-household clusters around the country. Of the 629 deaths they recorded among these families since early 2002, 13% took place in the 14 months before the invasion and 87% in the 40 months afterwards. Such a trend repeated nationwide would indicate a rise in annual death rates from 5.5 per 1,000 to 13.3 per 1,000 - meaning the deaths of some 2.5% of Iraq's 25 million citizens in the last three-and-a-half years. The researchers say that in nearly 80% of the individual cases, family members produced death certificates to support their answers.

Reliable data is very hard to obtain in Iraq, where anti-US insurgents and sectarian death squads pose a grave danger to civilian researchers. The survey updates earlier research using the same "cluster" technique which indicated that 100,000 Iraqis had died between the invasion and April 2004 - a figure that was also dismissed by many supporters of the US-led coalition.

'Survivor bias'

While critics point to the discrepancy between this and other independent surveys (such as Iraq Body Count's figure of 44-49,000 civilian deaths, based on media reports), the Bloomberg School team says its method may actually underestimate the true figure. "Families, especially in households with combatants killed, could have hidden deaths. Under-reporting of infant deaths is a widespread concern in surveys of this type," the authors say. "Entire households could have been killed, leading to survivor bias."

The survey suggests that most of the extra deaths - 601,000 - would have been the result of violence, mostly gunfire, and suggests that 31% could be attributable to action by US-led coalition forces. The survey is to be published in a UK medical journal, the Lancet, on Thursday. In an accompanying comment, the Lancet's Richard Horton acknowledges that the 2004 survey provoked controversy, but emphasises that the 2006 follow-up has been recommended by "four expert peers... with relatively minor revisions".

A computer scientist with a blog called Good Math, Bad Math (who appears to be politically neutral) says:

I've gotten a lot of mail from people asking my opinion about the study published today in the Lancet about estimating the Iraqi death toll since the US invasion.

So far, I've only had a chance to skim the paper. But from what I can see about it, the methodology is sound. They did as careful an analysis as possible under the circumstances, and they're very open about the limitations of their approach. (For example, they admit that there were methodological changes compared to earlier studies to reduce the risk to members of the survey team; and there were several data collection errors leading to invalid or incomplete data which was then excluded from the analysis.)

My guess would be that this study is a pretty solid upper bound on the death toll of the war. Population-analysis sampling based techniques like this do tend to produce larger numbers than other analyses, but over the long term, while the sampling techniques tend to over-estimate, those higher numbers have tended to be quite a bit closer to the truth than the lower numbers generated by other techniques.

When I compare this to what the US government has been trying to feed us, I find that I trust these results much more: this study is open and honest, tells us exactly how they gathered and analyzed the data, and is honest and forthcoming about its limitations and flaws. In comparison, the official US estimates are just black-box numbers - our government has refused to provide any information on how their casualty estimates were produced.

Faced with that contrast, and the history of causalty recording and analysis in past wars and natural disasters, I'm strongly inclined to believe that while we will probably never know the real number of people who've died as a result of our invasion of Iraq, the figure of 600,000 deaths as of today estimated by the Lancet study is far closer to the truth than the US government estimate of 30,000 as of last december.

Believe me, nothing would make me happier than being wrong about this. I really don't want to believe that my country is responsible for a death toll that makes a homicidal maniac like Saddam Hussein look like a pansy... But facts are what they are, and the math argues that this mind-boggling death toll is most likely all too real.

He then followed that post up with:

As expected, the Lancet study on civilian deaths in Iraq has created a firestorm on the net. What frankly astounds me is how utterly dreadful most of the critiques of the study have been.

My own favorite for sheer chutzpah is Omar Fadil:

I wonder if that research team was willing to go to North Korea or Libya and I think they wouldn't have the guts to dare ask Saddam to let them in and investigate deaths under his regime. No, they would've shit their pants the moment they set foot in Iraq and they would find themselves surrounded by the Mukhabarat men counting their breaths. However, maybe they would have the chance to receive a gift from the tyrant in exchange for painting a rosy picture about his rule.

They shamelessly made an auction of our blood, and it didn't make a difference if the blood was shed by a bomb or a bullet or a heart attack because the bigger the count the more useful it becomes to attack this or that policy in a political race and the more useful it becomes in cheerleading for murderous tyrannical regimes.

When the statistics announced by hospitals and military here, or even by the UN, did not satisfy their lust for more deaths, they resorted to mathematics to get a fake number that satisfies their sadistic urges.

You see, going door to door in the middle of a war zone where people are being murdered at a horrifying rate - that's just the peak of cowardice! And wanting to know how many people have died in a way - that's clearly nothing but pure bloodthirst - those horrible anti-war people just love the blood.

And the math is all just a lie. Never mind that it's valid statistical mathematics. Never mind that it's a valid and well-proven methodology. Don't even waste time actually looking at the data, or the metholodogy, or the math. Because people like Omar know the truth. They don't need to do any analysis. They know. And anyone who actually risks their neck on the ground gathering real data - they're just a bunch of sadistic liars who resort to math as a means of lying.

That's typical of the responses to the study. People who don't like the result are simply asserting that it can't be right, they know it can't be right. No argument, no analysis, just blind assertions, ranging from emotional beliefs that the conclusions must be wrong to accusations that the study is fake, to claims that the entire concept of statistical analysis is clearly garbage.

The Lancet study is far from perfect. And there are people who have come forward with legitimate questions and criticisms about it. But that's not the response that we've seen from the right-wind media and blogosphere today. All we've seen is blind, pig-ignorant bullshit - a bunch of innumerate jackasses screaming at the top of their lungs: "IT'S WRONG BECAUSE WE SAY IT'S WRONG AND IF YOU DISAGREE YOU'RE A TRAITOR!"".

The conclusion that I draw from all of this? The study is correct. No one, no matter how they try, has been able to show any real problem with the methodology, the data, or the analysis that indicates that the estimates are invalid. When they start throwing out all of statistical mathematics because they don't like the conclusion of a single study, you know that they can't find a problem with the study.

Majikthise, also from last fall:

The right wing noise machine is clanking and shuddering. They're outraged about this study, published in the Lancet. The study estimated that 665,000 more Iraqis have died after the US invasion than would have been expected based on pre-invasion death rates. (I discuss the study in more detail here)

Here are today's talking points. Or should we say talking flails? There aren't many actual points here:

1. 655,000 is an awfully big number. That would mean that this war killed a whole lot of people. (Jane Galt)

2. If 770 extra people were dying in Iraq every day, why don't we hear about them on the news? (Gateway Pundit)

3. The study was published before the election. (Instapundit) (Political Pitbull)

4. The peer-reviewed paper must be bogus because the editor of the Lancet goes to anti-war rallies. (Anti-Idiotarian Rotweiler)

5. The pre-invasion death rates are too low. Surely, Saddam was filling mass graves two months before the invasion. (Chuck Simmins)

6. Those peacenik scientists just wish there were more dead Iraqis. ("When the statistics announced by hospitals and military here, or even by the UN, did not satisfy their lust for more deaths, they resorted to mathematics to get a fake number that satisfies their sadistic urges," Omar Fadil.)

7. I just know the study's wrong, but I can't figure out how. Math people? (AllahPundit)*

8. Sure the study's methodology is standard for public health resesarch. But don't forget that public health is a leftwing plot. (Medpundit)

9. These "statisticians" say that you can take a small sample from a large population and learn a lot about the whole population. As if. I'll believe those 665,000 Iraqis are dead when they tell me so. (Tim Blair)

Cowards, all of them. They own this war, but they won't face up to the fact that their little adventure helped kill over half a million people.

More debunking from Tim Lambert, Appletreeblog, and Liberal Avenger. Stone Court scrutinizes Jane Galt.

At the blog Kiko's House we get the following summary:
I've now had an opportunity to read the article and accompanying documentation in The Lancet on the controversial new study that concludes nearly 655,000 Iraqis have died because of the war. My initial skepticism has been replaced by, well . . . non-skeptical anger.

Over the years I've been involved in a fair amount of cluster sampling such as that used in the study by the Johns Hopkins Bloomberg School of Public Health, and the resulting mortality numbers seem pretty solid to me, if mind blowing.

For me, keys to the accuracy of the numbers are:
The pro-active nature of the sampling. Virtually all previous surveys on civilian deaths have been passive; that is, statistics were gleaned from secondary sources. With the Hopkins survey, the numbers were compiled in the field through interviews with the families of the victims.

Who did the sampling.
Four-member teams of medical doctors fluent in English and Arabic led by supervisors did the interviews. Death certificates and other documents were examined to determine exact cause of death.

How the numbers were crunched. The statistics were derived from 50 population clusters determined by size of area. Heavily populated Baghdad had 12 clusters while Kerbala and four other sparsely populated areas had one each.
Most surveys on Iraqi civilian war deaths have been passive in the extreme. This is because most have been based on news media reports. The consensus conclusion of these surveys has been that between 40,000 and 50,000 Iraqis have died since the U.S.-led invasion in March 2003.

This survey method has two big problems:
* The media has concentrated on the carnage in Baghdad, where the city's Central Morgue and the Ministry of Health have kept seemingly accurate mortality tallies which are grim in their own right.

But beyond those figures media accounts are typically anecdotal since there seldom is any follow-up reporting to validate their accuracy.

* There has been no reliable way to tally deaths in the outlying provinces, which increases the Baghdad-centric bias of these media account-driven surveys.

I have long suspected that deaths outside the capital have been higher on a per capita basis than in the capital, in part because of the absence of news media representatives and because deaths in villages and rural areas tend to go unnoticed by all but immediate families and tribal members anyway. The Hopkins study confirms that is indeed the case.
My confidence level in the study also was raised because efforts were made to iron out problems in a 2004 study that might have skewed the results or made accurate results more difficult to obtain.

An example: Survey sites were determined by random numbers applied to streets or blocks rather than with global positioning units as had been the case two years ago. The random approach is less skewed and therefore more accurate. GPS units also were ditched because people using them might be viewed with suspicion and be put at risk in a country far more violent than in 2004.

Some other nuggets from the article on the survey:
* The civilian death rate has increased every year since the war began.

* Two out of three victims have been males, typically ranging in age from adolescence to middle age.

* Non-violent deaths during the same time period included cardiovascular conditions (37 percent), cancer (14), chronic illness (13), infant deaths (12), accidents (11) and old age (8).

*The Department of Defense keeps records of Iraqi deaths. The most recent accounting is in a classified 2006 report called the Measuring Stability and Security in Iraq. When the Pentagon says it doesn't know how many Iraqis have been killed, it's lying. If and when the report is released, expect the DoD's numbers to be dramatically lower than the Hopkins survey.
The survey team was candid about possible biases that might skew the results in one direciton or another.

These included the large-scale migration out of Iraq which has decreased population size, households not available on an initial visit were not called back because of security concerns, and there were instances when entire families were killed and there was no one to interview. (Don't you hate when that happens?)

The article's conclusion is chilling:
"In the Vietnam war, 3 million civilians die; in the Democratic Republic of Congo, conflict has been responsible for 3.8 million deaths; and an estimated 200,000 of a total population of 800,000 died in conflict in East Timor . . . We estimate that almost 655,000 people -- 2.5% of the population in the study area -- have died in Iraq. Although such death rates might be common in times of war, the combination of a long duration and tens of millions of people affected has made this the deadliest international conflict of the 21st century and should be of grave concern to everyone."
I have highlighted in red print those passages I found particularly salient to the discussion.

I'll have to say that much of the criticism towards the Lancet studies and the ORB studies cited in an earlier post strike me as eerily reminiscent of arguments in favor of Holocaust Denial (sometimes euphemistically referred to as "revisionism"). The talking points indeed seem to amount to little more than "that's too big a number," the science behind the analyses is bogus, that merely using a hand-held calculator and official Green Zone scribed reports is preferable to actually relying on research conducted as well as one can in a war zone, based on a representative sample of real human beings, and in which the vast majority of the interviewees produce death certificates to back up their claims. Why don't we "hear about these awfully high daily death tolls in the news (after all, our mass media would never lie to us, except, supposedly to diss "Dear Leader")? When in doubt, I suppose one can always resort to the usual accusations that those who read the reports, comb through the methodology, and who accept that the analyses are likely accurate are somehow "unpatriotic moonbats" or whatever names they wish to hurl.

I hinted at the fact in a previous comment that the methodology employed by ORB, which itself has an excellent reputation for methodological rigor, was used to estimate the death toll from 1994's Rwanda genocide. As far as I am aware, there seems to be little debate out there as to the accuracy of that genocide's death toll or the methods utilized to estimate the Rwanda death toll. Where's the disconnect? Is it okay to accept the methodology employed by ORB or the Johns-Hopkins researchers who published in Lancet when it's someone else's genocide, but completely unacceptable when coming to grips with what is arguably our own?

One thing that would be worth mentioning: we scientist types (whether physical, biological, medical, behavioral, or social) tend to be rather conservative in the sense of being cautious and skeptical. When faced with a finding that seems excessive or counterintuitive, the first impulse is to question its veracity. When I first read about the initial Lancet report, my initial impulse - and this comes with the usual caveat that I have been against this war from the get-go - was "that can't be right. How did they come to their conclusions?" The numbers in those studies were, to say the least, horrifyingly large. After reading the original article and its follow-up, I have been left with little doubt that the estimates from that research are about as good as we're going to get from an on-going war zone, and are likely to stand the test of time once the war ends.

At this point, I have found the right-wing "arguments" against the current approximation of 1 million Iraqi deaths due to the US invasion and occupation to be unconvincing. Time will tell.

No comments:

Post a Comment