320: What’s In A Number? — 2006 Edition

Ira Glass

OK, here's a simple way to measure how violent things have gotten in Iraq. Calling around this week to five English-speaking Iraqis. Every one of them knew somebody who's died in the violence since the US invasion. A doctor in Baghdad, who's active in politics, told me he knows over 300 people who've died. A guy in Babylon says it's pretty safe where he lives, but his cousin was shot by Americans when he got too close to them on the road. A medical student's cousins died from an IED exploding in the street. And he has friends who died too.

Ira Glass

Just today told me, when he went to the store near his house, he saw three people were killed. No one knows why. Ali, a bookstore owner in Baghdad that I talked to picked up a list of people he knows who've died. Friends, relatives neighbors.

Ira Glass

It's at the point now where if you go up to a dead body and try to clear it away, you're in danger of getting shot yourself. It's at the point now where Iraqis are getting their names, addresses, and phone numbers tattooed on their bodies. So if they're killed, their bodies will be identified and returned to their families.

People I talked to said they're scared all the time. Everybody said that it was better when Saddam was in power. Not that they like Saddam. Mohaned, an architect, described the choice between Saddam and the current situation in this way.

Ira Glass

Seems like every day we hear these stories and numbers from Iraq. Car bombings, more deaths, sectarian killings. But today on our radio show, we try to understand what you would think would be one of the simplest facts about the war, once and for all.

From WBEZ Chicago, it's This American Life, distributed by Public Radio International. I'm Ira Glass. Our program in three acts today. Now, in the second half of the show we have a very unusual recording of US forces trying to make things right with Iraqis, after the US accidentally killed 12 civilians. In the other acts of the show we talk about a number, and in one of the acts a small news story that nobody ever really seemed to take very seriously at the time. And what was really behind that story. Stay with us.

Ira Glass

Act One. Truths, Damn Truths, and Statistics. Well, the number that we're going to talk about right now-- and we are going to talk about a number-- the number is how many civilians have died as a result of a war. Fact is, nobody is even pretending to have an accurate count. Iraqi Ministry of Health for a while, early on in the war, was compiling morgue figures from across the country and making them public each week. But that practice was stopped. There's also a privately run nonprofit web site called iraqbodycount.net, which keeps a running tally of confirmed deaths in Iraq. That's as opposed to an estimate of all the deaths in the country.

And there have been two studies published in the British medical journal, The Lancet. One of the studies came out just a few weeks ago. The other came out two years ago. And both of these studies arrived at casualty numbers that are so high, they shocked many people.

Right now, in this first act, we're going to return to a story that one of our producers, Alex Blumberg, did for our show a year ago about that first Lancet study. Later in the show, in act three, we're going to look at the study that just came out. Anyway, here's Alex.

Alex Blumberg

Everyone will tell you counting casualties in wartime is hard. First of all, you need to do something called a large-scale mortality survey. And second of all, you need to do it in the middle of a war zone.

The first attempt ever made in Iraq was this Johns Hopkins University study published in The Lancet, a British Medical Journal, in late October 2004. A couple of days before the US presidential election. It concluded that probably a 100,000 Iraqis died as a result of the war. This figure was astonishingly high. 10 times higher than any other casualty estimates at the time.

Because of this, and because the study itself got almost no traction in the press, I remember thinking at the time it came out that it was probably bogus and slanted. I'm guessing a lot of people, if they heard about the study at all, felt the way I did.

But recently, in trying to find out how many Iraqis have died in the war, I've learned more about the Lancet study. And the more I learned about it, and the remarkable story of how it was done, the more likely it seems that the 100,000 figure is actually the best estimate and possibly low.

Alex Blumberg

This is Les Roberts, the lead author on the Lancet study. And one of a handful of scientists in the world who could be called an expert in counting war dead.

In the Congo study he found that 1.7 million civilians had died from the war, a figure cited by Colin Powell when he was Secretary of State and Tony Blair on the floor of the British Parliament. Les has also done studies in Burundi, Rwanda and Sierra Leone.

To a guy in Les Robert's line of work, the war in Iraq had a number of unique and interesting things that deserve study. The main thing that distinguished this war was that the military took unprecedented care to avoid civilian casualties. Almost 2/3 of the bombs dropped were precision guided, as compared to just 8% in the first Gulf War and 0% in World War II. They limited daytime strikes and avoided civilian infrastructure like power and sewer plants. Compare that to World War II, where American forces firebombed entire cities as part of the military strategy, killing up to 100,000 people in Tokyo alone. And upwards of half a million civilians in Europe.

And you can see why George Bush called the Iraq War one of the most quote, "humane military campaigns in history." The lesson here that often it's not bombs and bullets that kill people in war, it's the other things that happen when society falls apart. Clean water and medical supplies get scarce. In a lot of the studies he did in Africa, diarrhea killed more people than weapons did. Women can't get to the hospital to deliver their babies so infant mortality rates go up as well.

It took Les a long time to get to Iraq and see if the same things were happening there. First, an Iraqi doctor who he'd planned to work with, died in an auto accident. Another social ill, by the way, that tends to increase during wartime. And then insurgent violence spiked.

It wasn't until August of 2004, five months after he'd originally planned to go, that he finally landed in Amman, Jordan. He had $24,000 in foundation money in his pocket, his passport, and a letter of invitation from the Iraqi Ministry of Education. He found a driver, a retired Iraqi army officer named Wahid, who agreed to take him to Baghdad. Problems started at the very first checkpoint, on the border between Jordan and Iraq.

Alex Blumberg

Fortunately for Les, Wahid he was something of a Han Solo figure. An unenthusiastic but talented smuggler, who didn't look for trouble but didn't run away from it once it found him. He talked his way through that first checkpoint.

Alex Blumberg

Les finally made it to Baghdad, where he met, for the first time in person, his Iraqi co-researcher. The man with whom he'd be working for the next month. His name was Riyadh Lafta, and he was a doctor of community medicine at Al-Mustansiriya University, in Baghdad. Riyadh had hired a team of researchers, mostly doctors from his university. All of them native Iraqis, but fluent in English.

Let's pause here a moment to talk about their methodology. Because when the study came out later, a lot of people wanted to believe that it was flawed or biased. In fact, the survey team used a standard methodology for measuring health and mortality over a geographic area. It's called a cluster sample survey, and it works like this.

Using the most recent census figures available on Iraq, the team made what was essentially a map of the population. They then used a random number generator to pick 33 points on that map. Baghdad was the biggest population center, so it got several points by itself. But the other points were spread all over the country, from the Kurdish north to the Shiite south. From small towns to big cities.

Once they picked a town though, the team still had to figure out who the interview there. Here again, they worked hard to leave everything to chance. Using GPS units, they would drive around the outskirts of the town and store the coordinates, creating a rough outline of the town border. They would then generate a random point within that border, drive to it, and interview the 30 nearest households. It was such a commitment to random sampling that the first few times the team did it, even the researchers Les and Riyadh were working with found it obsessive.

Alex Blumberg

The survey went smoothly, at least for the first couple of days. People, it turned out, were much more willing to answer questions, even to provide death certificates as verification, than the researchers had initially thought they'd be. In fact, the trouble, when it came, came from Les himself.

Alex Blumberg

Les watched from the car as the police took the two researchers, both doctors. One a dignified man in his fifties and the other a single mother, and drove them away. He was terrified that somehow the police would find out that they were working with him, an American. But he could do nothing but sit in the parked car and hope no one discovered him.

Alex Blumberg

Riyadh and Les decided that for everyone's safety he should lay as low as possible. So for the next 16 days straight he didn't leave his hotel. To pass the time, he crunched the numbers that the survey teams were calling in to him every night.

The surveyors were getting basically two pieces of information from each household. How many people in that household had died in the 14 months before the invasion. And of what and when. And how many had died in the 17 months after the invasion. And of what and when.

By the time the teams had completed their 32nd out of 33 clusters, over 900 households and over 7,000 people, the results were pretty shocking. The death rate itself had gone up about 60%. A large increase, but one that Les had expected from his other surveys. The shocker was how people were dying. For the first time, in any of his surveys, the leading cause of death wasn't disease. It was bombs and bullets.

In the 32 of the 33 clusters sampled, 21 people died of violence. That's compared to just one violent death in the period before the war. Of those 21, two people died in firefights where it was unclear where the bullet came from. Three were killed by insurgents, or Saddam loyalists. Seven died from criminal violence-- carjackings, revenge killings, that sort of thing. And the biggest number, nine, were killed by the American-led coalition.

Alex Blumberg

At the end of three weeks, there was only one more cluster to survey. The team had saved it for the end, because it was the most dangerous one, Fallujah. Remember, this is September 2004. Insurgents control the city. And it's basically under siege from the Coalition. They're shelling it regularly.

Alex Blumberg

Only one other interviewer agreed to go to Fallujah with Riyadh. A doctor who had relatives there he wanted to check up on. Their car was stopped three times on the way into the city. Heading to the random spot, they saw devastation everywhere. Houses were bombed. Rubble lay in the streets. The block they stopped on was no different.

They had to visit 52 households to get the requisite number of interviews. 23 homes were either temporarily or permanently abandoned. Neighbors said that in the abandoned houses most people had died, but this data couldn't be substantiated. So it wasn't even included in the survey results. In the 30 households they did survey, there were 53 deaths. 52 of these were violent deaths. All but one caused by Coalition weapons. 24 of the people killed by Coalition bombs and bullets (SUBJECT were children under 12 years old. And with that the survey was over.

Alex Blumberg

This is Fox Evening News, October 28th, 2004, on the day of the results of Les's survey, that just shy of 100,000 Iraqis had died as a result of the war, were released. Les had not even considered the Fallujah data in coming up with this number. Fallujah had so many deaths, it was too much of a statistical outlier to even include. Fox never mentioned the study. Neither did ABC or CBS. The only national network that carried the story was NBC for 21 seconds.

Alex Blumberg

Morning Edition and All Things Considered on NPR devoted 45 seconds to the story. It didn't make the front page of the New York Times, the Washington Post or any national newspaper.

The Iraq study had provided information about the war that up until that point no one had been able to provide. The number it gave was much higher than anyone would have expected. It was just as accurate as Les's previous studies in Africa that he had done using the exact same methods. And which were widely reported in the press and quoted by lawmakers. His Congo study was page one in the New York Times. The only differences with this study where that he'd risked his life to do it and it was about Iraq, which, if anything, should have made it more interesting to the media.

So why didn't it get any press? Partly, it was the timing. The study came out five days before the US election, and so the media was pretty preoccupied. Plus, there was a suspicion that the team had timed the release of the survey specifically to influence the election. A suspicion that Les didn't really help dispel.

He said to an AP reporter about the study, quote, "I emailed it on September 30th, under the condition that it come out before the election. My motive in doing that was not to skew the election. My motive was that if this came out during the campaign, both candidates would be forced to pledge to protect civilian lives in Iraq. I was opposed to the war, and I still think the war was a bad idea. But I think that our science has transcended our perspectives. As an American, I'm really, really sorry to be reporting this."

One desk editor at a national news organization told me that when the study came out, he sent an email to one of his colleagues saying, The Lancet had, in the past, published some studies with a political slant. But that this study seemed sound, and maybe they should report on it. Then he saw Les's comments, and he didn't follow up. This is exactly the type of story that those who believe the media has a liberal bias love to pounce on. And so in essence, if the research turns out to be flawed, this desk editor's organization gets the heat for it.

He had a very small window of time during a very busy news cycle, to decide whether the study was legit or just an angry and easily debunked researcher pushing an agenda. And Les's comments seemed to be all the evidence he needed.

And there was one other thing that made it easy for the media to dismiss the report. A researcher at Human Rights Watch, who himself had done studies of civilian casualties during wartime, said he didn't believe the study. The researcher's name was Marc Garlasco, and he told a reporter for The Washington Post, quote, "The number seems high to me." And quote, "It seems like a stretch."

Alex Blumberg

This is Marc Garlasco. He said he told the reporter from The Post that he hadn't read the study. But the reporter said he really needed a quote, and could he just respond to the number. Garlasco's quote was cited elsewhere and he appeared on CNN, although none of the study's authors were interviewed on CNN or any of the major networks. Here's what Marc Garlasco says now.

Alex Blumberg

OK, let's just stop here for one minute. You heard what he said. He'd taken part in the targeting for the war. Get ready, because this story's about to take a turn. Marc Garlasco isn't your typical human rights advocate.

Alex Blumberg

Marc doesn't see moving from the Pentagon to a human rights nonprofit as the 180 degree flip most people might. He says all he's ever wanted to do is fight bad guys and both organizations do that, just in different ways. He'd been thinking about leaving the military before the war began, and he hadn't supported the war himself. But he stayed through the fall of Baghdad, because he knew the target set better than anyone else. And he figured if there was going to be a war anyway, it might as well be him targeting the bombs rather than someone else who might not know or care as much as he did.

The thing that finally prompted him to leave, he says, didn't have anything to do with the war. His wife got a great job offer at the Bronx zoo, and they'd always wanted to move back to New York. When Marc went with Human Rights Watch to Iraq, it wasn't to get a comprehensive count of civilian casualties. His mission was to look at specific attacks and see which kinds of attacks caused high civilian death tolls. Because Marc had planned many of the strikes he was now going to investigate, it was a little complicated.

Alex Blumberg

The attack had hit the intended buildings, but it also destroyed the two neighboring buildings. That's where the man's family had died. Also, Chemical Ali hadn't been in the targeted building anyway. It's unclear who died there.

Marc went to lots of places in Iraq he'd studied on maps and aerial photographs and heard about from defectors. And there's no way around this. After all those years of imagining these places, what they must be like, it was exciting to actually be there.

Alex Blumberg

The military denied my request to talk on the record about civilian casualties. But Marc Garlasco says that civilian casualties are one of the primary factors he and his colleagues considered when planning the war. Say he had a target he wanted to take out. The headquarters of the Iraqi Secret Service maybe or one of Saddam's palaces. He'd work with the weaponeering guys to figure out how many and which bombs to use, and then--

Alex Blumberg

A lot of times, when the collateral damage assessment came back too high, they'd try to get it lowered. For example, a strike Marc planned early on in the invasion. An Iraqi division was holed up in a big multi-building convention center in Baghdad, which unfortunately was right across the street from a hospital.

Alex Blumberg

What got Marc thinking about civilian casualties in the first place, was a battle damage assessment he did after the war in Kosovo. He targeted the bombs for that war. And then afterwards, the military sent him over to see how well he'd done. He measured how often the bombs hit their targets. Whether they destroyed what they were supposed to destroy. Pretty much the only thing he didn't check the accuracy of were the collateral damage assessments.

Alex Blumberg

In talking to people in the military, off the record, I heard a couple of arguments against counting civilian deaths. First they say, it's not the military's job. What you're trying to do is win a battle. It could be a dangerous, and in the long run, counterproductive distraction to worry about counting all the civilians you accidentally kill along the way. Second, and perhaps more persuasively, they say no one would believe them anyway.

Just ask Les Roberts. Even though Les's study didn't get much mainstream attention, it did provoke, like so many things these days, a bitter debate on the internet. The attacks came mainly, though not exclusively, from right-wing blogs. Several charges leveled at the study were simply untrue, and seemed designed to willfully muddy the waters.

For example, there was a claim repeatedly made, both in the press and online, that the data weren't random because the researchers had been blocked from going certain places. Or had decided against certain places because it was too dangerous. This is simply false. A couple of people suggested that the researchers had gone to Fallujah, on purpose, to boost their numbers, even though exactly the opposite was true. Les had wanted to skip Fallujah altogether, and they hadn't even included the data in their final casualty estimate.

Several objections had merit though. First of all, the study makes no distinction between combatants and civilians. Les actually acknowledged this in the study itself, and went to great lengths not to claim, as others did on his behalf, that the study was a measure of civilian mortality. Certainly some of the people the Coalition killed, they intended to kill. But half of all the casualties were women and children, so even in the unlikely event that 50% of the men who died were actually fighting us, it's still a large number of innocents.

The critique that got most traction on the internet though has to do with something called the confidence interval. Let's take an election poll as an example. Candidate X is projected to receive 55% of the vote. What that really means is that he's projected to receive some percentage within two numbers, let's say 52% and 58%. That range is called the confidence interval.

The confidence interval in Les's survey was very wide, between 8,000 and 194,000. It was this wide for a lot of reasons, but mainly because the sample is relatively small relative to the population. And because violent death, unlike death due to malaria or diarrhea, isn't very uniformly distributed.

So you have Kurdish areas where mortality actually went down during the war, versus Fallujah, which averaged almost two violent deaths per household. Such a wide confidence interval means that, statistically speaking, Les's estimate of 100,000 dead isn't very precise. The number could be thousands or tens of thousands smaller or equally likely, bigger.

But a lot of people made wrong conclusions from the wide confidence interval. They interpreted it to mean it was just as likely that 8,000 people had died as it was that 100,000 had. The online magazine Slate wrote, This isn't an estimate. It's a dart board. In fact, the likelihood follows a bell curve, with 98,000 being at the top of the bell, the most likely number. So actually, there's only a 2.5% chance that the number is 8,000 or below. But a 90% chance that it's 44,000 or above. Here's Les.

Alex Blumberg

Clearly, the people on talk radio weren't attacking the study out of a commitment to experimental rigor. They were attacking it for the same reason that the news media was hesitant to report it. Because the very act of counting civilian casualties is political.

The moral logic of war is this. We're willing to undergo x number of costs in lives, money, resources to accomplish some goal. The goal, we hope, will be worth it in the end. So assuming the goal in Iraq is good, is it wrong to kill 100,000 civilians? Saddam himself probably killed 230,000 of his own people. A number, by the way, no one seems to go out of their way to dispute. If you add the million or so lives he lost in the futile war he launched against Iran, 100,000 seems like a bargain in comparison.

Maybe he would have gone on another killing spree, and this 100,000 is insurance against the later, far worse death toll. Or maybe 100,000 lives is worth it, if in the end democracy does blossom through the Middle East. After all, we killed far more Japanese with just two atomic bombs than, according to Les, we did in a year and a half in Iraq.

If we don't count civilian casualties, we don't have to get into this kind of horrible math. And most of us don't want to. So instead, we leave it to the professionals. The military are the only ones who even try to come up with a formula, the collateral damage assessment. 30 dead civilians for one bad guy. For Les, he doesn't really care who counts, just so long as someone does.

Alex Blumberg

One of the most surprising things Les discovered in Iraq is that, despite what everyone says about the difficulty of counting civilian casualties during wartime, it's actually not that hard. The survey teams got participation rates that most American pollsters would kill for. Only 5 of the 988 households the team surveyed refused to answer the questions. And people were able to provide death certificates over 80% of the time.

That confidence interval, Les is sure that based on the results of the first survey and with a little more money-- remember, this whole thing cost only $40,000-- he could design a follow-up survey that would narrow that interval way down. We can count civilian casualties in wartime. We just have to want to.

Announcer

Alex Blumberg. Coming up, so a military that's trying as hard as it can not to kill civilians makes a mistake. What do they do then. Well, we have a recording of what they do then. Also, we take a look at the new Lancet study. The sequel to the one you just heard about. It just came out two weeks ago. That's in a minute. From Chicago Public Radio and Public Radio International, when our program continues.

Ira Glass

Well it's This American Life, I'm Ira Glass. Each week on our program we choose some theme, bring you a variety of different kinds of stories on that theme. Today's show, What's in a Number. In Act One of our show, we heard about the number of Iraqi deaths that occurred in the first year and a half after the US invasion. And this is no longer true in Iraq, but at the time, according to this Johns Hopkins Study we were talking about, the majority of deaths were caused by the US-led forces. Despite everything they did to avoid Iraqi deaths.

Well, now we're going to move to Act Two, where we hear US forces trying to cope in the aftermath of some of those deaths.

Ira Glass

This is Act Two, Not Just a Number. The civilian deaths, in this particular story came at a unusually bad time for Captain Ryan Gist. He was the American officer in charge of the US Army presence in a section of the Nineveh and Salahuddin provinces in Iraq, a Sunni area. And this meant that he dealt with everything from seeing that the region had water and fuel to dislodging insurgents and winning over the local population. He faced all sorts of problems.

On November 10th, 2004, for example, insurgents blew up most of the police stations in this area. And the next day, Iraqi Army and police simply stopped showing up for their jobs. One town called Atha was especially troublesome. The US had not made many allies there.

Ira Glass

So the guy gives him names and locations of people that he identifies as insurgents. And the army launches an operation to get those people. And this is where things went wrong. As part of this operation, the US forces were supposed to drop a bomb into a nearby field, just as a show of force.

But instead, the bomb was dropped onto the house that the army was just about to raid, killing 12 people inside, including children. And nearly killing the US troops who were about to go in. That's according to an American photojournalist, Sheryl Mendez, who was there. In addition to the human tragedy of these deaths, for Captain Gist this could not have been worse.

Ira Glass

The photographer, Sheryl Mendez, tape-recorded him as he went to the sheiks, who were the local leaders and to the police, as he went around trying to make things right. Here's one of those recordings. He's in a police station.

Ira Glass

This is pretty much the standard way that he'd operated in these villages. He says, sure, there were some villages and local leaders who were sympathetic with the insurgents. But in other places, they were being bossed around by them.

Ira Glass

In Atha, he says, people waited to see who was going to win, the US-led forces or the insurgents. Until there came a turning point.

Ira Glass

Captain Ryan Gist. He left the military in the Fall of 2006. He's now a law student at the University of Wisconsin. Though he still stays in touch, by email, with the Iraqis he knew in Atha.

Ira Glass

Act Three, The War This Time. So a couple of weeks ago, the Lancet published another study of deaths in Iraq by the same research team who did the earlier study. This time, the number of deaths that they came up with was seen as even more astounding than the first study's number. Over 650,000 extra deaths because of the war. 601,000 of those due to violence.

This second study got slightly more coverage than the first study. A reporter even asked President Bush about it. Our producer, Alex Blumberg, examined the debate over this new study. And he called various statisticians and epidemiologists who looked at study.

Ira Glass

Alex Blumberg. Special thanks today to Jerome McDonnell and WBEZ's show Worldview, where you first heard Les Roberts. Thanks Ben Shapiro and Beth Osborne Daponte, Fred Caplan and Michael White, Nancy Yousef. Laura Poitras, who has a new documentary about Iraq coming out called, My Country, My Country. Jamie Tarabay, Adam Davidson, and people at War News Radio, including Wren Elhai. This American Life is distributed by Public Radio International. This American Life is made possible by Volkswagen of America and the Volkswagen Jetta, encouraging listeners to stop stereotyping. Learn more at thejettareport.com.

Support for our podcast, our free podcast, comes from audible.com, where you can download audio books, magazines, newspapers, and radio shows, including archives from the last ten years of This American Life. Available at audible.com/thisamericanlife or at the iTunes store.

WBEZ management oversight for our program by Mr. Torey Malatia. Whenever he finishes a budget or correctly orders a lunch, he calls me into his office, leans back in his chair--

Ira Glass

I'm Ira Glass. Back next week with more stories of This American Life.

Announcer

PRI, Public Radio International.