Bean Counters (Lancet numbers vs Estimates of those killed by Saddam)

I found myself thinking about the furor over the Lancet study over the last few weeks. There was something vaguely troubling me about the criticism. Then it sort of hit me. A sample of my thoughts follow.

It used to be common to fill a jar with different colored beans and turn it into a contest by asking who could figure out the ratio of the different kinds of beans or determine the total number of beans in the jar. Most people estimate the ratio of beans in the jar through a method called “sampling.” You take a random handful of beans and determine the ratio. Assuming you have a representative sample, you’re projection is likely to be surprisingly accurate. Recently the Lancet, one of the world's oldest public health journals used a similar method to estimate the number of deaths in Iraq. They did this once before in 2003 and came back with a shockingly high number in the hundreds of thousands. The 2006 followup (note registration required) estimated some 650,000 deaths in Iraq since the war started. Iraq has only 27 million people, so the number is more than a little eye opening.

The President himself rejected the study and a number of right wing sources jumped forward to suggest that the Lancet’s estimates were impossible. There were even a number of skeptical left-wing voices, among them the iraq body count an anti-war site that has tracked the number of dead including civilians by using reports of deaths in western media. Their figure as of today was a maximum of 49,760 or about 8 percent of the Lancet study’s number, a huge discrepancy.

By the way, they point to a UN sponsored report called the Iraq Living Conditions Survey ILCS, to argue that another sampled survey came up with a much lower number of dead in 2004. The right wing groups like to point to the Iraq Body Count refutation as proof that even the left is uncomfortable with Lancet. For those who are into unintentional humor, the ILCS was a project of the United Nations Development Programme. This may be one of the first times the right has cited, albeit indirectly, a UN program approvingly.

I realized fairly quickly that my last statistics class was the basic introductory class that all liberal arts majors took as breadth requirement for a bachelor’s degree. I can’t and shouldn’t argue confidence intervals, variances, etc. It’s clear to me that a number of individuals who know less than I are doing so anyway. I would mention that the 95% confidence interval cited by the Lancet is a pretty standard number for studies that use samples of random respondents of roughly 1,000. You have to go to something like 10,000 repsondents to get to 96% so most pollsters stop with a sample of 1,000.

Beyond saying that Lancet is peer reviewed and generally well respected, I have to say that I’m simply not qualified to attack the methodology. I would point out a couple things though. We routinely rely on a very similar methodology and sample size in a number of areas. Billions of dollars a year in ad revenues depend on the Nielsen survey which for years has used a sample of Nielsen families. More to the point, the political polls that everyone seems to take so seriously depend on the same basic methodology as the Lancet study. Every now and then, you’ll hear someone complain that 3 Nielsen families played a joke and cost someone millions of dollars or that say an “Exit poll” in 2004 oversampled female voters, but for the most part our culture is more than comfortable with “sampling” as a way to tell us how many, how much, etc.

In fact, the pollsters themselves like to point out that we routinely depend on another kind of sample to make critical health decisions. Blood tests are a random sample.

This is my peeve though about the furor over the Lancet study. The same right wing sources insisting that the Lancet finding must be a gross exaggeration even an impossibility are the same folk who estimate the number of people killed by Saddam in the hundreds of thousands. Do they ever ask where that number came from. If you used Iraq Body Count’s method to do the estimate, that number might be in the hundreds not the hundreds of thousands. Few if any of those killed by Saddam were reported to mainstream western media at the time. The actual number of bodies recovered in mass graves remains relatively low, it’s just that the number turns scarier when one asks how many more “undiscovered” graves of this sort are out there. If one tracks down the figure of hundreds of thousands of dead during Saddam’s reign, one finds that it’s often boosted with estimates of the number killed during the Iran-Iraq war or those who may have starved to death during the Western embargo during the 90’s. Both figures obviously raise some issues if you’re trying to directly attribute the number to Saddam.

When you break it all down, the right has been using methods not at all far removed from Lancet’s in its attempt to estimate the number killed by Saddam. The bottom line answer turns out to be that the higher estimates of the Saddam number comes from a projection of some kind. Does the media ever question the number so casually thrown out by some who justify the American invasion?

I don’t know if the Lancet number is right. I do see surprisingly few attacks on the actual methodology. Most of the attacks simply say “the number itself violates logic,” which is somewhat similar to what happened when geologists in the 19th century started estimating the age of the earth in billions instead of thousands of years. Those who believed in the traditional religious view didn’t question the methodology, they simply declared that the scientists’ number was somehow illogical or impossible.

I also know that death certificates and news reports tend to grossly underestimate the number of dead in a war. Chaos prevents people from documenting each and every death. Not only is it hard to find the bodies, but it’s nearly impossible to keep records because the records themselves get destroyed. Even some of the numbers for the holocaust, which the Germans documented perversely well, might be on the low side because not all the killings took place inside the camps.

Yes, it’s a horrifying number, but even if it’s off by a factor of six, it’s a horrifying number. Even if the sample is flawed and only represents areas of Iraq that have relatively higher casualties, the finding is horrifying.

The problem is that the Administration will attack the Lancet study then out of the other side of its mouth talk about the significance of positive poll results. They know most of us won’t put the two together. In fact, they’re counting on it.

11/21 note: It's worth mentioning that after the US midterm elections, the Iraqi government queitly upped its own estimate of civilian deaths to 150,000. I need to check the Iraq bodycount site to get their reaction...since they insisted that their method was not a drastic undercount. Obviously, this helps the credibility of the Lancet study even though it's still more than 4 times the official count.


chancelucky