The Worm Wars

Give me two seconds.

I need to close the windows.

I was like, it's so fucking hot in here.

And I was like, I'll try to do it with the windows open, but there's like shitloads of bird sounds.

Yeah, totally.

All right, no birds.

This is now a no birds podcast.

You have never sounded more like Jerry Stiller.

No birds.

No birds.

What do you got?

What have you got for us?

Hi, everybody, and welcome to Maintenance Phase, the podcast that's crawling out onto the sidewalk after a nice long rain.

Oh, it's a worm.

It's a worm opener.

All I had was worms.

You had two words, worm wars.

Sure, man.

I'll take my inspiration from my morning walk the other day.

I'm Michael Huggs.

I'm Aubrey Gordon.

If you would like to support the show, you can do that at patreon.com/slash maintenancephase.

And if you want to just keep listening, just keep listening.

We're so happy to have you.

You're here with us.

Today we're talking about worms.

Yes.

So, okay, I'm so excited because I like telling you stories and I like fucking with you.

And so

I told you nothing other than the name of this episode, Worm Wars.

This is the controversy that tore methodology Twitter asunder in 2015.

And I want to tell you all about it.

Tell me.

So the hardest part of this episode is going to be getting through the first five minutes.

Oh.

There's just some like gross stuff like body horror that we have to get through.

Okay.

Are you familiar with intestinal worms?

Uh, I mean, not really.

Okay.

So I assume they're like parasites, yeah?

Yes.

Okay.

So somewhere around 2 billion people in the world have intestinal worms, which are like exactly what they sound like.

They are little tiny worms that live inside of your digestive system.

Some intestinal worms have lifespans of like one to two years, and they'll literally just like attach themselves to the inside of your intestine and just like be there for years and then they die.

Most people have light to moderate infections, which don't really have any symptoms, but sometimes you'll get like much more heavy infections.

And so this causes various symptoms.

It usually happens through giving you anemia.

It makes you tired.

It gives you brain fog.

You have to sleep all the time.

It can stunt kids' growth in the developing world.

It can make them too skinny.

I have never had a worm infection or parasite or any of that kind of stuff, but I have been anemic before.

And boy, oh boy, it doesn't seem like it's going to be that rough.

And then it's pretty rough.

Dude, a friend of mine passed out in a Starbucks because she got anemia.

You just can't get your energy up like at all.

Yes.

So, okay, so here's the really gross part.

So the way that worms spread is you get these worms in your intestine lining and they lay eggs.

They lay like hundreds of thousands of like little microscopic eggs.

And these eggs end up in your poo, and then you poo out the eggs.

And so, the reason why worm infections are endemic throughout the developing world is because oftentimes there isn't running water.

So, in some areas, people will kind of use soil to wash their hands after they go to the bathroom.

They'll just kind of grab some like dry soil from the ground.

Sometimes they use poo as like fertilizers in gardens and then it ends up in their food.

Or it's like they're walking around and they have a cut on their foot and there's like a little tiny egg that like ends up getting into the cut and then like swims its way to their intestinal tract.

Sure, sure, sure.

So it's just like the poop-human interface is like how people end up getting worms.

And it's like this rolling snowball where like the more worms there are in the soil, the more likely kids are to get it, the more likely kids are to spread it, et cetera.

It used to actually be a really big problem in America.

Here is what I appreciate about this whole episode:

I didn't have to bring up anything related to poop.

I know.

You just fully picked a topic, and you were like, you know what, I could go for it right now is a little poop research.

The thing is,

I saw a methodology fight and I was like, oh, it's going to drive me into the poop world, isn't it?

I have to go inward.

So, the protagonist for this episode is a guy named Michael Kramer, who is currently a Harvard professor.

Okay.

Before Before all of that, he had spent some time in Kenya as a kid, and he was really fascinated by why are people poor in this country?

Like, what are we doing to alleviate global poverty?

He eventually gets his degree in economics, and so he decides to do his graduate work in Kenya.

So he moves to Kenya, and he starts befriending people in the development sector there.

He meets like World Bank people, and NGO people, and charity people.

And what he finds is that the field of international development is like in kind of what other people will later call a statistical dark ages.

This is like the early 1990s.

And the way that people do development, it's kind of on like the live aid model, right?

Where it's like a celebrity does something and there's like this big fanfare and then like no one ever follows up afterwards.

Yeah.

And if there's any like statistical rigor or like studies being done, they're oftentimes before and after studies.

So they're like, we gave everyone a goat and then a year later they were doing better.

And it's like, well, was just just everyone doing better in the country that year because there was like a really good harvest or something?

Like you're not, you're not comparing them against a control group.

Or was there another policy intervention?

Right.

There's like so many things that it could be.

Right.

Think about the last year in your own life and think about if someone from the outside tried to determine what was the cause of any changes in your life, right?

Like that would be kind of a fool's errand, right?

Right.

And a lot of the projects are done in this sort of like common sense way.

This is oftentimes how you get bad development projects where people are like, oh, it's just common sense to, you know, give everyone a bicycle or whatever.

And it's like, well, did you ask them first if they needed a bicycle?

Like, if you give everybody in the village a bicycle, is that going to like change the dynamics of the village?

Like, it's just being done in this like really unsophisticated way.

So Michael Kramer.

has a friend who works at a Dutch NGO and they're about to implement a program where they give kids in Kenya textbooks.

So at the time, there's like 30 kids in every classroom, and there's one textbook per classroom.

So again, common sense.

It's like, well, if everybody had a textbook, this is obviously going to make like their grades better.

And like school performance, attendance, everything's going to get better if everybody has their own textbook.

Obvious, right?

So Michael Kramer suggests to his friend, he's like, no, no, no, don't do a before and after study.

Instead of just giving this to seven schools, take 14 schools and give seven of them textbooks and the other don't give them textbooks, and then you can actually compare the effect of the textbooks.

So they do this, the project goes for a couple years, and lo and behold, there's essentially no effect of the textbooks.

So after the textbooks don't work, there's this period of kind of testing stuff.

Because it's like, well, that's weird that the textbooks don't work.

So then they try like chalkboards, like a lot of the schools don't have chalkboards.

So again, they split into two groups.

Turns out that doesn't work.

They then do a thing where they hire more teachers at half of the schools.

That doesn't work.

So it's like they're kind of going over these sort of obvious, quote-unquote, education interventions.

And it turns out that like education interventions are not working to improve education.

They're like, hmm.

So one night, Michael Kramer is talking to a friend of his who works at the World Bank, and he explains the symptoms of worm infection to Michael Kramer.

And he tells him that like there are treatments available for worms.

Like there are pills for basically every kind of species that's endemic in that part of Kenya.

And like they're really cheap.

And diagnosing the kids is actually like really expensive because you literally, like, you literally have have to go through their poop and measure how many eggs there are.

And so it's such a pain to diagnose this.

And it's so easy to give the kids these medical treatments.

Like they don't really have any major side effects.

They're pretty cheap.

They're already available in that part of Africa.

And he's like, I wonder what would happen if we just gave all the kids these deworming tablets.

Because some of them are probably infected with worms and just don't know it.

So they do the same thing.

They find a district with 35,000 students, 75 schools, and they do the same thing.

They separate it into control groups and treatment groups.

They roll out this mass deworming program where they just give every kid deworming pills, right?

Okay.

So I'm going to send you an excerpt.

This is from a book called Doing Good Better, How Effective Altruism Can Help You Make a Difference by William McCaskill.

Quote, Kramer did an experiment to see whether treating children for these intestinal worms had an impact on education.

The results were striking.

Absenteeism is a chronic chronic problem in schools in Kenya, and deworming reduced it by 25%.

In fact, every child treated spent an extra two weeks in school, and every $100 spent on the program provided a total of 10 years of additional school attendance among all students.

Enabling a child to spend an extra day in school therefore cost just 5 cents.

It wasn't merely that deworming children, quote-unquote, worked at getting children into school.

It worked incredibly well.

It worked incredibly well.

I mean, fucking yeah.

Yeah.

If you have something uncomfortable to painful to debilitating going on with your body, when you feel better, I bet you do better in school.

Yeah.

So, I mean, it's, yeah, it's a very appealing story also because they've tried all of these educational interventions, and it turns out that there's this sort of like zigzag solution.

That maybe it's not an education intervention, maybe it's like there's something broader and bigger at play that is just invisible to people that are in the school and also development aid workers.

Yep, totally.

So the main headline from this study is that they reduced absenteeism by 25%.

That's the thing that ends up in William McCaskill's book.

That's like the kind of the main thing, it's in the abstract.

But there's also some other effects.

So one of them is like there's health benefits.

So the kids who got the deworming were slightly taller, they're slightly heavier.

It was actually nice to read a study where it's like, the kids are fat.

Yeah.

Yay.

It's like just once.

What a win.

They also had lower rates of anemia.

And of the really interesting effects of the study is these spillover effects.

So because worm infections are an infectious disease, right, one person gives it to another person, once you blast hundreds of kids in these schools with deworming pills, they then don't spread it to other people.

So what they find is that there's actually attendance effects in schools like three kilometers away and six kilometers away.

There's all these like other benefits that come along with basically just like breaking the chain of infection.

And they also find long-term effects.

This is from McCaskill's book.

When Kramer's colleagues followed up with the children 10 years later, those who had been dewormed were working an extra 3.4 hours per week and earning an extra 20% of income compared to those who had not been dewormed.

In fact, deworming was such a powerful program that it paid for itself through increased tax revenue.

Wow.

So it's like you can then even like fast forward 10 years and you find that like the dewormed kids are doing better.

Big if true.

Big if true.

Aubrey, it's the first five minutes of the the episode.

You know everything I tell you at this point is true.

Everything goes according to plan.

The nice white guy from Harvard is definitely right, and no harm is going to come from his actions.

Stop remembering the way that I tell you stories.

Stop remembering the premise of our show.

The premise of our show.

So over the next decade, basically, deworming becomes like the hottest idea in international development.

I was working in international development at this time.

I remember all of like the deworming national plans that were coming out.

It paralleled the rise of this effective altruism thinking, where in Michael Kramer's original paper, which was published in 2004, a lot of the paper is actually about like the cost benefits of this, right?

That, you know, when it comes to something like hiring more teachers, well, like every single teacher costs a lot of money and there's, you know, there's payroll and there's all these other costs associated with these like educational interventions.

Whereas this one, like the deworming pills, cost 30 cents each.

The benefit of this versus the cost of doing it is just like unprecedented in international development.

So in 2007, Michael Kramer founds a NGO called Deworm the World, which like helps governments implement their national deworming plans.

There's also this charity GiveWell, which basically has this model of like, we're going to look at all the evidence and we're just going to tell you that like if you have a hundred bucks to give away, here's probably the most effective thing to give it to.

You know, if you if you want to help people, this is what you should do.

And that's like one of the few charities that is ranked by GiveWell is this Deworm the World initiative and other kind of deworming charities.

There was like a deworming strategy by the WHO.

The World Bank was really big on deworming.

It was like deworming was like the hotness, right?

Kenya has a national plan on the basis of this 2004 paper.

India declares February 10th National Deworming Day

and has a program to deworm 140 million kids a year.

Whoa.

This is huge.

According to the Gates Foundation, at one point, they put out a paper calling this the largest public health program ever attempted.

Wow.

Huge, full court, like decade-long full-court press on deworming.

There's part of this that feels very like TED Talky, life hacky.

This one simple trick

fixes everything.

Aubrey.

Am I getting ahead of myself?

I love that you're at the point with the show where you're like, wait a minute, there's a good story here.

It's probably fake.

I don't trust it.

So, okay, I'm about to send you a couple paragraphs

from

a Guardian article in 2015 that kicks off the worm wars.

This is one of the longest excerpts that we've ever read on the show, but I wanted to get, I wanted you to get like the flavor of this.

Great.

The headline says, New research debunks merits of global deworming programs.

And I say, fucking called it.

Twistity twist.

Quote, deworming children, once ranked by Nobel laureates as the fourth most effective intervention to solve the health problems of the whole world, offers very little benefit despite the millions of dollars spent on it, according to a reanalysis of the evidence.

The World Bank and the UK government are among the big funders of deworming programs that were claimed to improve not only children's health and growth, but also their educational attainment and ultimately their country's economic prosperity.

However, much of that optimism rests on one particular study carried out in Kenya in 1998 to 1999, which reported that deworming improved attendance not only in schools where children were treated and taught about avoiding worms, but also in other schools because of an apparent reduction in transmission.

Researchers from the London School of Hygiene and Tropical Medicine have now reanalyzed the trial data and published two papers in the International Journal of Epidemiology.

Researchers used methods common to health research rather than those used in economic research, as the original paper had.

God damn it.

Good sign.

The epidemiologists finally look into it 10 years later.

There had been no increased attendance in the schools where children had not been treated, as the original study said.

That was down to calculation errors and was wrong.

The researcher also found missing data and some evidence of bias in in the finding that school attendance improved.

Shocking.

Who could have seen that coming?

What is your overall impression?

What is the plain language paraphrase of what you've just read?

The fucking epidemiologists never looked at an epidemiology issue, and it was studied by fucking economists.

What are we doing?

And then, like, whole fucking nations built their interventions around what a fucking economist thought was good for public health.

At this point in 2015, I've seen various numbers going around.

One of the numbers is that 1 billion deworming pills are being distributed annually.

What?

Under programs like this.

I assume you're going to walk me through this.

Yes.

It sounds like what they have debunked here is the idea that school attendance did not improve in schools where kids were not treated.

That's a very good catch.

Yes.

It's not the whole thing is garbage.

It's just this idea of school attendance going up in other schools is not true.

That's what I'm taking away from this paragraph.

That was a very good read that like a lot of people did not notice at the time.

You just notice now.

Oh, fun.

So now we get misunderstandings.

So this is the thing.

So we had to talk about what they actually did and what they actually found.

So as you know, there's like in psychology and other sciences, there's now this like this thing called the replication crisis.

Essentially, the idea is that like there's all these canonical studies that have affected policy and have affected public understandings.

And they're basically rerunning them.

And in a lot of cases, they don't pan out.

So things like the infamous like marshmallow test, what happened here is that people looked at this study from 2004 in Kenya being used to justify deworming 140 million children in India and were like, wait a minute, like a lot is resting on this paper.

Let's just look into the findings.

So like they didn't redo any studies.

This wasn't a replication.

What they did was they went to the original author, so it was Michael Kramer and his co-authors on the original paper, and they said, give us all your data.

Like, give us the software that you ran, give us the original, like the questionnaires you used in the school, like just give us everything, and all we're going to do is just run it again.

So to their great credit, Kramer and his co-authors, like, give them everything.

Okay.

So these reanalysis researchers spend months going over everything and basically publish like a reanalysis of this paper.

So the biggest thing that they find is there's data missing and there's data that's miscalculated.

You know that like spreadsheets have all these conditionalities where like one cell refers to another cell,

yeah.

It's like they basically got one of these little codes wrong, and that ends up cascading down to all of the other codes in this way that was like invisible to the original researchers.

So it's essentially like the level of like a transcription error.

Exactly.

So as a result of all of these errors, everything of kind of recalculating the data, all of the health effects of deworming disappear.

What?

The anemia link is gone, the kids' difference in heights gone, the difference in weights gone.

Wow.

And then essentially all these spillover effects.

So like all this stuff about like the indirect three kilometers, six kilometers, those effects, the six kilometer effects are gone and the three kilometer effects are much, much, much weaker.

To me, the main thing that comes out of this statistical reanalysis, I mean, this is something very well known in studies of school attendance in sub-Saharan Africa and really throughout the developing world, that like if you rely on school registers, they're not very accurate.

Teachers write kids down as being here when they're not really here.

So, if you want to do a study that actually measures school attendance, you have to send people to the school to like physically count the kids, which to their credit they did in this study.

Like, they sent people to like, okay, one, two, three, four, five in every school when they wanted to do it.

And they did it at random times, right?

Because you want to get an actual, like, an accurate count of the kids.

What they found in this reanalysis was that the deworming schools got way more visits from the the counters, from the auditors,

and the schools that got visited more had better attendance rates.

It could be just the fact that someone was checking that's doing this and not actually deworming.

So

this is what turned me into the fucking joker on this.

Once there's all this like statistical stuff going on and this extra scrutiny on the original paper, people also go back and just like read the original paper.

The first thing that jumps out at you is that this wasn't a deworming program.

It was a deworming and education program.

So the school that got the deworming pills also got like classes about like why it's important to wash your hands, like better sanitation techniques, how to prepare food.

The kids and their parents in some cases were getting information along with the deworming pills that was related to not getting reinfected with worms.

Nah, but it's definitely those pills.

But it's definitely, I mean, this is the thing, is like,

the researchers say that when you survey the kids about like how important it is to wash your hands and stuff, there's actually no difference between the kids that got the education stuff and the kids that didn't.

Wow.

As we talk about on this show all the fucking time, like public education campaigns don't really work.

Knowing that washing your hands is good doesn't give you running water in your house.

The researchers are like, oh, well, it doesn't matter that we also gave them education programs because the education programs didn't work.

So we can scale up the pills without scaling up the education.

But the thing is, they didn't check the effect of the education on the school attendance.

There's literally no way to separate those things.

You're doing two things at once, and then school attendance gets better.

I am just really breathing through, feeling my blood pressure rising.

You don't have to be like the world's greatest fucking scientist to figure out that if you give somebody deworming pills and also like a bunch of like educational resources and support for their lives, you can't then say everything that's changed is just because of those pills.

We've been through this so many times as a field.

This is as a former development aid guy.

This is what drives me nuts is that what you have over and over again is you have a technical intervention, something like a medical intervention or like better lighting in schools or whatever, something that's easy to scale, right?

But then you also have all these social things that go around it, right?

Like getting to know the kids, having them attend these educational seminars, having probably just like more adults in the school generally if they're participating in a study, right?

And then it's like when it's time to upscale, we're like, oh, this is a proven development intervention.

So we're going to upscale the technical parts.

But you're not going to upscale all the fucking social stuff that made the technical stuff work.

God damn it.

That's not...

the purpose of a randomized controlled trial.

The entire point of all of this was to be like, we're going to do it like pharmaceuticals do it, right?

We're going to do it in a way that allows us to say with certainty that, like, this is the intervention that mattered.

Yeah.

And then they do this like 101 shit where it's like, oh, well, we're going to do two things at once and then upscale one of them.

Yeah, totally.

And also, like, this does feel sort of related to because this is all happening at the site of education as a field.

It also feels totally fucking wackadoo and deeply predictable to have a bunch of economists roll in and be like, teachers, you've been fucking it up.

You're welcome.

We're here.

We figured it out.

But then, okay,

are you ready to explode?

Oh, no, I guess.

So the other thing that people start like bringing up about the original paper is that that number of that it reduced absenteeism by 25% is a little bit misleading.

Okay.

If you imagine a school with like a 98% attendance rate, and then you switch the vending machines from like Coke to Pepsi, and then they have a 99% attendance rate.

You could say that they reduced absenteeism by 50%.

Oh, god damn it.

That sounds really impressive, but what you've done is you've gone from 98% to 99%, which is not that impressive.

No, this is the like fucking diet marketing of like, you're twice as likely to lose weight on our diet than on this other diet.

And you're like, well, okay, so you lose one pound instead of half a pound in a year.

Great.

You're twice as likely to win the lottery when you buy two lottery tickets.

That doesn't mean you're going to win the lottery.

The actual attendance rates at the schools that didn't get deworming pills, the attendance rates are 75%.

At the schools that did get deworming, it's 82%.

Oh, God.

Is that nothing?

No.

Like, it's a 7% bump in attendance rates.

But also, a 7% increase in attendance.

Much less like we must scale this to 140 million children immediately.

Another thing that, again, is in the paper that was published in 2004 is that the deworming pills did not have any effect on cognitive ability or school performance.

So nothing.

Nothing.

So exam scores, they did some like IQ testy pseudoscience bullshit on the kids.

Nothing.

So it only affected attendance.

I feel like I'm resorting to fucking Lama's breathing to get through this shit.

Like, I feel like I'm this close to like...

to like deal with my fucking rage about this shit.

Well,

the whole narrative of this intervention doesn't really make sense.

Yeah.

The theory of change that deworming is supposed to do, right?

It's like you give kids the deworming pills, their health improves, their school attendance improves, their test scores improve, and then their life gets better, right?

That's like the chain of events, right?

Yep.

But what we have now is we have you give the kids deworming pills, it doesn't seem to affect their health, it does affect their school attendance, but doesn't affect their school performance.

Yeah.

It's just kind of a mess.

I don't know, man.

This feels like an indicator of the depths of two major appetites.

One is how fucking hungry we are at all times for a silver bullet thing

and how much more fun it gets to people and exciting when it's like an unexpected thing.

Like, oh, if you get this one pill, you get better at school.

Imagine.

Totally.

And appetite number two that is sort of shaping this is the unending appetite of white people to be like, we found the thing, we solved it, you're welcome.

You couldn't do it yourself, oh, we did it for you.

Exactly.

Not a bad impulse to want to help, a terrible fucking impulse to want to like save people from themselves.

I actually got radicalized on this, I think, a decade ago now, where I was writing an article about like how big ideas were ruining international development.

I called up one of the charities that was doing one of these projects in India.

They were expanding the deworming, mass deworming programs to India.

In one of the most Kafka-esque interviews I've ever had, I was asking them, like, okay, how's the project going so far?

And they're like, it's great.

We've distributed 10 million pills or whatever it was.

And I was like, okay, great.

Like, what are the results on school attendance?

And they were like, oh, we're not measuring that.

What?

Oh, because it's proven.

Like, it's proven that it affects school attendance.

So we don't really have to measure that anymore.

And I was like, well, it's proven in 75 schools in Kenya 10 years ago.

You're doing this in India, which is a completely different educational context of schooling and like the government's role in school, everything else.

And like, so you're just not testing the effects anymore.

And they're like, but we are testing the effects.

I was like, well, then what are the effects?

They're like, well, we've distributed 10 million pills.

That's okay.

Okay.

So you and I get into this shit often.

That's a fucking output.

It's not a fucking outcome.

Exactly.

That's step one to do what you set out to do.

Like, it's truly fucking wild how often you and I get into data on this show and the results of a given study or a clinical trial or a whatever are just, here's what we did.

And then that's celebrated like something that has like an impact on people's health out in the world.

But Aubrey,

save your rage.

I'm getting too angry too soon, aren't I?

This is a three-part debunking, and that was only part one.

What?

That is, frankly, nothing short of a personal attack.

So, at the same time that this reanalysis of the 2004 paper is going on, I think this is a coincidence.

There's also a meta-analysis of every previous study on deworming that's ever been done.

Oh, one thing that drives me absolutely batshit about this story and about the way that it's presented in this, like, effective altruism book is everyone pretends like no one thought of fucking deworming before.

There have been deworming projects since the 1970s.

Yeah.

So in 2015, there's a Cochrane review of every previous decent study on this.

It covers 45 studies going back to the late 1970s.

It includes 20 randomized control trials in Africa.

It finds, the review says, there is quite substantial evidence that deworming programs do not show benefit in terms of average nutritional status, hemoglobin, cognition, school performance, or mortality.

Nothing.

That's the good stuff right there.

there.

The world's largest randomized control trial, or at least according to the study itself, was performed in India in 2013 on deworming.

It finds no effect.

There's a huge randomized control trial in China, which also does not find anything.

There have been 10 years of various deworming projects in Myanmar, and the rates of worm infection are now higher than they were 10 years ago.

What?

Which I don't blame the, I don't think that has anything to do with the studies itself.

I think, again, it's like, I think there's something else going on there.

Sure.

What drives me nuts about this is like, why would we still be talking about this 2004 study in Kenya when there's like 50 other fucking tests of this by now?

Yeah.

People have tested this, and it has been found wanting.

Time and time and time and time again.

It doesn't have any of the effects that people said that it had.

And that's on any measure, right?

We're no longer in school attendance world.

I mean, they mentioned school attendance.

This is actually like one of the main things of like the worm wars, like the big methodological debate that comes out is because Michael Kramer and others say that like you can't test whether it affects school attendance in like a meta-analysis because there's not enough studies that do school attendance.

It's only based on this one.

And they all like, they throw accusations back and forth, but.

But it's like, there are other studies that look at school attendance and it doesn't seem to have any impact on it.

And also, even if it did, it's like, why are we talking about school attendance?

When like it doesn't affect school performance.

Right.

And it doesn't affect health.

Right.

Then what?

Then what?

You got them in the room, then what?

And also, there's also some like interesting sort of second order effects that oftentimes when school attendance rates go up in the developing world, the teachers are then overcrowded and like lots of the kids end up doing worse on tests because they're getting less attention from their teachers.

Not to say that like school attendance is bad or whatever, but it's like anytime you're doing these development projects, there's always these weird second order effects.

And so it's like it's not, there are actually places where even if it did increase school attendance, if it increases it significantly and they're not able to hire teachers, it might not actually have the long-term effects that you're going for.

Every study on this seems to find that it doesn't really do what they set out to do.

In a lot of these studies, you can tell they don't want to admit that it didn't have any effect.

You and me are like trained to say this.

Now you'll read the abstract and they'll be like, we distributed 50 million pills and then like we surveyed a bunch of kids and the kids had heard of the program.

Yeah.

And you're like, wait,

that's not like no one would look at a deworming program and be like, have the kids heard of this or not?

Yeah.

It's like you're clearly picking something you can publish.

We know that calorie labeling works because some people saw a billboard about it one time.

Exactly.

I was sub-tweeting this when we were recording that episode.

I was sub-tweeting the deworming people.

I have to say, I really appreciate how often in the researching of an episode, you or I end up like cryptically tweeting about something so that we don't spoil it only for the other person.

The thing is, I was going to block you on Twitter because I was like about to tweet something salty about deworming and I was like, I don't want to.

Are you fully going to block me?

I had a whole plan.

I was going to be like, Aubrey, I just want you to know I blocked you on Twitter.

It's not you.

I tweeted about deworming, but then I was like, oh, I don't want her to know.

So then I just, I walked away from my computer ultimately.

Here's the hilarious thing.

As you know, I have been very off Twitter for the last like, I don't know, four months or so.

So the thing that is hilarious to me is I think the rest of the internet would be more likely to figure that out than I would.

Yeah, exactly.

I know.

And then, did you know Mike blocked you on Twitter?

Yeah, Ombre is like,

Mike blocked Ombre on Twitter.

Trouble in paradise.

So, okay, final stage of the debunking.

Okay.

Okay, we've talked about the 2004 paper problems.

We've talked about the field of deworming programs in general, meta-analysis problem.

We're now talking about the problems of upscaling.

Okay.

Because this is another thing that as a former development guy, I know that what happens in 75 schools in Kenya doesn't happen the same way once you start rolling this out on like tens of millions of kids in like in entire countries.

Sure.

So this is something that happens in 2008 in a region of Tanzania called Morogoro.

Tanzania adopted like a national deworming strategy and was rolling this out across the country.

And so the idea was that they were going to give kids a bunch of deworming pills and they had a town hall meeting a couple days before this happened to like tell the parents about it.

But according to the parents later, they didn't like really get the message that this was happening or like they just had other things happening that day, like agricultural duties, things at night that they had to do.

So very few parents ended up attending this like one town hall meeting being like, we're going to give your kids deworming treatments.

And according to the parents who actually went, the information that they gave the parents there was not even very good because they didn't really explain to the parents that they were doing prophylactic deworming.

You know, deworming pills are like pretty widely available in sub-Saharan Africa.

And like when you get symptoms of a worm infection, like you go to the pharmacy and if you can afford it, you buy the pills and you take the pills and the worms are gone.

These are known, these are like known drug treatments, but the idea of we're going to give every kid deworming pills like just in case, that hadn't really been done before.

It's a little bit like some teacher coming coming to you and saying, like, well, you know, your kid takes Tylenol for headaches, right?

Well, we're just gonna give every kid Tylenol at 3 p.m.

every day.

You'd be like, what?

Right?

Like, it just doesn't really fit with your understanding of how that particular medication works.

Yeah.

So after this town hall meeting that like very few people attend and doesn't give very useful information anyway, they start doing this deworming with the kids.

But another change that they make is that the deworming pills that people in Tanzania are familiar with, I think it's called albendazole.

It's like a little tiny pill and apparently it tastes kind of sweet.

And the kids are used to taking it.

You know how it's like kind of difficult to get kids to take pills in the first place?

Sure, absolutely.

But like because it's small, because it's easy to swallow, the kids are kind of used to it, so it's not that big of a deal to get kids to take this.

But there's different species of intestinal worms.

So they're also having these kids take this other pill that is like big and bitter and they have to test the kids height before they give it to them to get the dose.

And so some of the kids are taking three of this.

Some of the kids are taking two of this.

It's just like big and like really unpleasant to swallow.

And the other thing, I cannot fucking believe this, but this second pill that they're giving people, you should take it on a full stomach.

Otherwise, it starts to give you these side effects.

But like we are in a remote, very poor region of Africa.

Like most of these kids are only eating one meal a day.

So it's like they're not taking it on a full stomach.

Like whether you told the parents that or not, they're not taking it on a full stomach.

What are the side effects?

It's like us, it's like stomachache, nausea.

It's nothing serious.

But the kids start coming home from school and they say, oh, they gave us all these pills today.

They gave us this normal pill, but they gave us this new pill and they measured our height.

And Tanzanians are used to their weight being used to determine doses of like various anti-malarials and stuff.

Sure.

Measuring our height instead of our weight.

They gave us this new pill instead of this old pill.

And also, like, I have a stomachache.

And like, maybe I've been vomiting.

Maybe I have a headache.

I had these like weird symptoms.

Yeah.

So the parents obviously are like, what the fuck?

Like, a lot of them have not heard of this at all.

Yeah, totally.

And another thing that happens is fairly shortly thereafter, a little girl dies.

She dies of malaria.

She dies in the hospital.

She was not given the pills.

But it starts to look to people like these two things are connected, right?

All of a sudden, these kids are taking these weird pills in school.

They're getting weird symptoms.

And then this girl dies.

And so people on the local talk radio stations start talking about, you know, three or four kids have died because of the pills.

Yeah.

And so what starts happening is the parents form these kind of mobs and start showing up at the schools.

So this is an excerpt from a PhD dissertation that's written about this.

It says, In some schools, the situation turned violent as parents and guardians argued with teachers who had been administering drugs.

In one primary school in Shemwino, a classroom wall was pulled down, school property was destroyed, the national flag removed, rocks were thrown at the teachers, and one was severely beaten.

Holy shit!

The list of injuries suffered by teachers included a broken arm, a broken finger, and a serious head injury.

In one school, a health worker had the drug stuffed down her throat, causing her to choke and vomit.

Holy hell!

I know.

The riot police were mobilized and dispersed angry crowds at a number of schools using tear gas.

Jeez.

The Ministry of Health and Social Welfare officially suspended the program in Moroccoro and throughout mainland Tanzania the same day.

So like, I want to be very clear that like there's a lot of really fucking ugly tropes about like savage Africans and like Africans blowing.

Like there's nothing unique to African parents here.

We are talking about a program that is giving a medical treatment, a novel medical treatment to children without the knowledge or consent of their parents.

People anywhere would have, I think very rightfully like freaked out about this.

Right.

And also, like, white people in the U.S.

are losing their shit about their kids have to talk about race sometimes.

Well, yeah, exactly.

It is, to me, a deeply understandable reaction that, like, you find out from your kid that they've been given a new pill at school.

You don't know what that was.

You don't feel like you were notified about it.

And now your kid is, like, getting sick as a result of that pill, right?

I don't know, man.

Like, this is like every episode of of like Law and Order SVU.

If anybody ever tried that with my kid, they'd have held a pet.

Yeah.

You know what I mean?

Like all of that kind of shit, right?

Like this is a way that we talk about parenting and it's a way that a lot of parents feel, which is like, do not fuck with my kids.

Also, like, as soon as we start talking about whether a development intervention works or doesn't work, we're already in the wrong place.

Something can work and not.

be rolled out well.

Yeah.

It can work in one place and not work in another place based on like really, really obvious like social and political conditions.

Yes.

So like one of the things that happens after this in Tanzania in 2008 is the ruling party starts blaming the opposition party and says, oh, they're the ones that started the rumors.

And then they use it as an excuse for a crackdown.

Yeah.

So it's like, you might want public health not to take place in a political environment, but it does.

It just does.

That's right.

This is obviously like an extreme case.

I mean, this is by far the worst case of like deworming being rolled out too fast that I was able to find in the world.

But also like these kinds of problems happen pretty frequently with deworming programs because they're being rolled out so quickly.

So one of the things that they don't really mention in the follow-up literature to this 2004 paper in Kenya is that in a lot of schools, the teachers aren't distributing the pills.

So you can do these mass deworming programs where it's like, ah, 80 million kids or whatever.

You give a thousand pills to the teachers and they just don't give them to the kids because the teachers don't know why the fuck they're doing this.

They're not getting paid extra to do it.

They're not given the context for it.

And then you can go back to your donors and be like, we distributed a thousand pills in this rural village, but it's like, did they get to the kids?

There's various studies that measure like the uptake.

And some of them, it's like 20%.

Jesus.

And like, even in this, this drives me nuts.

Even in the original 2004 study in Kenya, again, in the paper itself, the first, they gave kids a couple rounds of pills.

Like it was like every six months they gave them deworming.

The first time they gave the kids deworming, 70% of the kids that were eligible took it.

By the fourth time they're giving kids the pills, only 44% of the kids are taking them.

And all of the stuff about costs, you know, the pills are only like 15 cents.

Like you hear different numbers.

I've heard two cents.

I've heard 45 cents.

The original 2004 paper says it cost about a buck fifty per student.

Once you add up that like we had to give it to them every six months and then all this distributional stuff.

The low cost of this intervention is based almost entirely on like pharmaceutical companies giving it away for free, a lot of like volunteer labor to roll this out, teachers doing this with like no sensitization.

All of like the low cost justification for these programs depends on the kind of like fast, cheap, universal rollout that caused the problems in Tanzania.

As soon as you start slowing down to actually see if this is the right thing to do, the cost savings often disappear.

Well, and also like this feels like very sort of like life hack kind of adjacent, right?

Yes.

It feels like doing that, but with like entrenched public policy and public health issues.

Right.

Like if someone came along and said that shit about like Brexit, you're like, oh, we got this one cool thing you do.

And then Brexit, we just undid it.

It's just all fixed.

People would rightly be like, that doesn't seem right.

Okay, so we've done the bunking.

and we've done the debunking and now we have to do a little bit of rebunking

What's getting rebunked?

So, okay, so that Guardian article that you read that was like, deworming has been completely debunked and like it's it's gone, right?

Is kind of famous in the sort of worm war spheres for being a little overstated.

Journalists only know how to tell two stories about international development.

It's either this is going to solve everything or this is a debunked fucking lie.

We went through this with microcredit.

Remember, it was like cover of Newsweek, microcredit is going to to solve poverty.

And then it like they expanded it again too quickly.

They expanded it to areas where it didn't work as well.

And it was like, microcredit's a fucking scam.

Yeah.

It's like, no,

neither one of those are true.

Neither one of those are true.

And also, like, you only get those two stories if you keep buying into the first one.

That's like, this is going to fix everything.

Exactly.

And also, I mean, my takeaway from this original 2004 paper that has been dissected so much is that like, after all the sort of statistical reanalysis, reanalysis, the schools that got the deworming treatments have a roughly 6% higher attendance rate than the schools that didn't get deworming.

To me, it's like that is a promising result.

It's not bullshit and it's not going to save the world.

It's promising.

It feels like a grounded result.

too.

Right.

If you think about your life and sort of things that have made differences in your life, my guess is that it's a constellation of a lot of 6% things.

Exactly.

It's the kind of study to me that it's like, okay, it seemed to quote unquote work in 75 schools.

Let's try it in another 75 schools.

Yeah.

It seemed to be fine in this region of Kenya.

Let's try it next door in Uganda.

Yeah.

It never should have been expanded to 140 million kids.

The thing that I wish that I could impress upon people in development aid is that literally nothing is going to be so good and so proven that you can go from 75 schools to fucking 3 million schools on another continent.

You're never going to find it.

I'll say in public policy advocacy world, which is the world that I come from,

there is such a deep scarcity mindset.

Yeah.

Right.

That you start from a place of constraints rather than starting from a place of like,

okay, if we really wanted to get to the bottom of school attendance, what would that take?

Instead, we start from a place of like, we found this one thing and that does it.

And we're just going to get a shit ton of it out there at the end and just hope for the best, right?

right that we get to the passage or non-passage of a policy and we think that's the end of the story and actually that's exactly when the rubber hits the road right like you have to pay just as much attention to implementation the least exciting sounding thing i know but like that's where the promise of the policy comes to life right yeah and in this one it feels like folks just completely skipped over program evaluation and assessment stuff right well i mean there are places where this works There's a big study in Uganda that finds that it does increase kids' heights and weights, and it does increase their test scores, although it doesn't affect their school attendance, which I thought was really funny.

So

it's like the mirror image of the Kenya study.

But it's like, okay, so it...

it works on these metrics in this place.

By far the most interesting study I read on this was there's a meta-analysis of 153 deworming projects in Kenya.

And the whole paper is about like, okay, what makes this work?

Because every development intervention works in some places and not others.

And it's not always obvious where it's going to work.

Like you said, implementation.

This is the hard part.

And what they find in Kenya is that the specifics of the implementation really matter.

So basically, every deworming project that only gives kids one dose of deworming pills doesn't work.

Because the deworming pills, they kill all the worms that are inside of you, just like napalms your entire digestive system, but it doesn't prevent you from getting reinfected.

So, if you only go there and give kids one round of pills and you come back in six months and they all have worms again, and it really only works in areas where there's like really high infection levels, because again, if you have like a light or moderate worm infection, like it sounds really gross to think about, but like it doesn't, it often doesn't cause any symptoms.

Like the worm just like lives in your body for a year and then it dies and you poop it out, and that's it.

So, unless you're dealing with an area where there's like large numbers of heavy infections, like in the original study, it was like a 22% heavy infection rate.

Like, that's exactly the kind of place where you should be doing deworming.

Like, that's that's really debilitating for kids.

But if it's in a place where, like, you know, 10% of people have worm infections, only maybe 1% of people have heavy worm infections, you don't want to give pills to every single kid for that, right?

It's like giving every single kid a Tylenol for one kid having a headache.

Yeah.

Another thing they find in Kenya is that it works much better when it's coupled with other interventions.

So, when you're giving kids like these deworming pills, even if they do have heavy infections, the worms are gone, but then they need extra food to kind of get like catch-up growth because oftentimes they've had like stunted growth and stunted weight.

So it's like they need to eat more for like a couple months after that to like catch up to where they should be.

But if these are really poor kids and they're in a really poor area and they don't have access to food.

Right.

Good luck.

Yeah, like it.

It's kind of like there's no point in deworming them because whatever benefits they would get from deworming, they won't actually get because there's not enough food around for them to eat.

So it's like you also have to do these nutritional interventions.

And in general, deworming works a lot better when it's coupled with this like much more holistic program about like sanitation, like getting people running water.

Also, some education programs are part of this.

I'm skeptical of those in general, but apparently they work here in some circumstances.

So it's like deworming as part of a larger program of like, we're going to work on improved sanitation in this area than like, yeah, they're a really essential component of that because you need to get the fucking worms out of the kids.

But then you're back to an expensive project and you're you're back to a project where, like, you have to actually know the local area and you have to talk to people beforehand.

And you have to build relationships.

And donors don't like that.

And like the TED Talk industrial complex, Davos, they don't like that either.

I mean, I think part of the reason that that stuff feels so frustrating to folks is that it feels so slow compared to, I got a million pills out.

Right.

What I would argue is that building something slowly and right and making sure that it works actually probably takes less time than going on five of these wild goose chase scenarios.

Jeez, of course everybody thinks we've found the silver bullet for whatever.

It's the like public policy and international aid equivalent of a get rich quick scheme.

Oh, it is.

You know what I mean?

I mean, it literally is for the developing world, like get rich quick.

Yes, yes.

The most chilling thing to me about this is the way that this generation of development aid has replicated exactly the mistakes of the previous generation.

Boo, say more about that.

So when Michael Kramer got into development during the statistical dark ages, right, everything was kind of like gut-based, vibe-based, right?

And they've now created a situation where they're basically saying like, we're just gonna ignore all of the data.

The World Health Organization is still doing mass deworming.

The

World Bank still stands behind this, the Copenhagen Consensus.

The GiveWell charity, which is like a group that like I really respect,

has written these weird defenses of deworming where they're like, no, no, no, no, no, ignore all the meta-analyses.

What we're really interested in is like the 10-year follow-up study.

We know that deworming works because of the 10-year follow-up study, but it's now, it's like there's no plausible path to them having better lives.

Like it's completely deranged to say that a program of deworming in which like up to half the kids didn't even participate, like didn't even get the deworming pills, and had no effect on health, no effect on test scores, and an extremely modest effect on attendance, would be affecting people's earnings 10 years later.

I'm sorry.

It isn't plausible to me.

And also, it's very weird for this entire field of development, the whole thing was like the randomistas, that's what they called this, like, this generation of mostly economists who came into development and were like, let's do this in a more rigorous way.

That now, these same randomistas, these like rigorous people, are now now telling me to ignore meta-analyses on the basis of a single study?

Like, that is so chilling to me.

It's like, what was the point of all this?

Yeah, I mean, imagine being like, definitely don't pay attention to a fucking Cochran review.

Right.

This is, okay, this is, this is what Michael Kramer, like during the whole kind of worm war, everybody fighting on the internet period in 2015.

This is in a sort of response to the response to the response to the response.

Like, this whole thing is like fucking Tumblr comments in like 2014.

He's talking about like defending deworming.

He says, the estimated value of benefits exceeds the cost by more than a hundredfold.

While the results of any one study should not be taken in isolation and effects may differ across environments, even if policymakers believe there's a one in 100 chance of experiencing effects of this magnitude, the expected benefits of deworming would exceed the costs.

They're playing our song, Mike.

He's basically saying, even if this doesn't work, look how cheap it is.

Look how cost-effective this is.

This is like the asbestos argument.

GiveWell says the same thing.

GiveWell says, because mass deworming is so cheap, there's a good case for donating to support deworming, even when in substantial doubt about the evidence.

This is very similar to some of the responses to the calorie labeling stuff.

Oh, yeah?

Yeah, absolutely.

That people are like, well, it doesn't work for everybody, but look how cheap and effective.

And you're like, effective measured by what?

Right, effective how?

We're doing something, we're not doing nothing, but also, the something that you're doing has the effect of nothing.

And also, plus maybe negatives.

Like, what?

I also, I forgot to mention,

Michael Kramer won the Nobel Prize in 2019.

Come on, Michael!

It's not just for this.

It's for like other work, and it's for like this whole kind of the systematization of statistics within international development.

And he shared it with some other economists.

And like, I actually think that like using more randomized controlled trials in development is a good thing.

I think statistical rigor is a good thing.

But also like that has come with drawbacks and side effects and a lot of like weird and bad cul-de-sacs.

I also think to sort of like defend the original 2004 paper, there's not going to be a perfect methodology ever.

Sure.

Like you can find things to quibble with in anything, in like a meta-analysis, in any individual study.

And ultimately, every single thing that we do in public health is going to be a judgment call.

In principle, that's not necessarily a bad thing.

I think the idea of a judgment-free science is always going to be a myth.

You're always going to have to have values.

Yeah, and if you don't get explicit about articulating and committing to your own set of values, then values will show up in your work.

Yeah, exactly.

Yeah, that's a good way to put it.

Without your intending them to, right?

So if we're thinking about, for example, pretty much all of the like weight loss research that we have out in the world starts from a place of assuming that fat people's experiences are immaterial.

Right.

If I asked some of those researchers, like, hey, do you think that like the life experiences of people who are trying the things that you're testing out matter?

I imagine quite a few of them would be like, well, yes, right.

But they haven't designed a thing that reflects that.

Right.

And like, that's part of what we're getting into here is this is research that was not designed in a way that reflected an understanding that there was knowledge on the ground that they needed to know about.

There are a bunch of things that, again, if you don't articulate those values and then take the next step of going, and here's how we're going to operationalize them.

Other values will speak on your behalf.

I think the parallel to fat people is really good because it's driven by the same thing of like, this is for their own good and like, I'm helping.

It's driven by paternalism.

Yeah.

And I think people give themselves permission to like not think things through.

To not think things through and to not ask for feedback.

Right.

That's the like toxic good guy narrative that worries me so much in international development and in public health and in lots and lots and lots of places.

So that was uh that was deepworming.

Is that where we're going to end it?

Just me being like, everything's fucked and we're all done and we all lost it.

And you're like,

there we go.

I feel like, okay, so this is kind of a digression.

You posted on the maintenance phase Instagram the other day.

It was like a meme about how every millennial podcast is just one person talking excitedly and the other person going, that's fucked up every five minutes.

I was like, wow, they've really got our number.

I think we both have the same reaction of like owned.

Fair enough.

When you're right, you're right.

I was trying to structure something around, like, how can I get Aubrey to say, like, that's fucked up?

It's fucked up.

It's a millennial podcast.

That's fucked up.

We did it.

We got there.