How to spot a suspicious statistic

This BBC podcast is supported by ads outside the UK.

Suffs!

The new musical has made Tony award-winning history on Broadway.

We demand to be home.

Winner, best score.

We demand to be seen.

Winner, best book.

We demand to be quality.

It's a theatrical masterpiece that's thrilling, inspiring, dazzlingly entertaining, and unquestionably the most emotionally stirring musical this season.

Suffs.

Playing the Orpheum Theater, October 22nd through November 9th.

Tickets at BroadwaySF.com.

Hello and thanks for downloading the More or Less podcast.

We look at the numbers and statistics in the news and in life and I'm Tim Harford.

Untruths sneak into our lives in all kinds of ways.

Sometimes they're outright lies, blatant misinformation.

But today we're going to talk about something else.

Those sneaky numbers and claims that bounce around our society that aren't exactly false, but are leading you down the wrong path.

That's the subject of a book called May Contain Lies by Alex Edmonds, a professor of finance at London Business School.

Let's start with a straightforwardly good piece of advice from Alex, if I do say so myself.

If you see a result that you want to be true, then as you say in your own book, Tim, just ask how it makes you feel.

Because if it's something that you really are chomping at the bit to share with others on social media, then it may well be that why this study has become so popular is because of its appeal, not because of its accuracy.

What kind of thing might this be?

Alex's first example is one we've looked at before on more or less, the relationship between breastfeeding and the health of a baby.

The idea that breast milk is better seems plausible, but is it true?

So, the World Health Organization recommended exclusive breastfeeding for the first six months based on evidence that this leads to better outcomes across a range of criteria.

So, physical health outcomes for the child, mental development for the child, physical recovery for the mother.

All of these were strongly correlated.

But here's the thing: is that whether a child is breastfed is not random.

It depends on family circumstances.

So maybe the mothers who are able to breastfeed have a more supportive home environment because breastfeeding is tough.

And it might be that that home environment, not the breast milk itself, is driving the better outcomes.

And so when you control for home environment, are the factors such as the mother's own IQ, whether the mother smoked, family affluence, you actually find that the link between breastfeeding and many of these outcomes goes away.

So just to stick with topics that aren't going to annoy anybody, let's talk about inequality and the costs of inequality.

You talk in your book about the spirit level, which is

feels like a blast from the past for me because I interviewed Kate Pickett, one of the authors of The Spirit Level, back in, I think, 2009, maybe 2010, when that book

was a huge bestseller.

And The Spirit Level is a book

basically publishing a bunch of statistical evidence which links inequality, economic inequality, to social problems and health problems, crime and so on.

Now I think a lot of people found that very plausible.

You have your doubts.

Yeah, so again, this is an issue where the correlation is there in the data.

So I'm not going to quibble with the correlation that they find.

It's not due to mismeasuring particular variables or doing something dodgy with the statistics.

And the correlation is basically more unequal countries have more problems, basically.

Dramatically oversimplify.

There's not really any doubt that that does seem to be true.

Yes, and this is across a range of outcomes.

And so this is why they interpret this as a strong evidence of inequality.

So the outcomes may well be gross national happiness, it might be obesity and health.

So this range of outcomes suggests there might be a unifying explanation, which is inequality.

But the issue here is again, what they've done is they found a correlation.

They have put on an interpretation which many people might be willing to lap up because many people don't like inequality, myself included.

However, there could be a whole host of other potential variables driving it, and one alternative explanation could well be poverty.

Now, poverty is related to inequality, but it's not exactly the same thing because if poverty is the true cause, then the solution as a government is to bring up those at the bottom, not necessarily to clip the wings of the people at the top, unless that's a way of actually redistributing wealth to help the people at the bottom.

And so when you look at other people who've now control for poverty, they find that it's poverty which is driving the outcomes such as ill health and obesity, not necessarily inequality itself.

Needless to say, that's not the view of the authors of the spirit level, but we're just going to leave that there and move on.

For a final example, Alex has been looking at research into the relationship between how diverse a business is and how well that company performs.

For example, you might look at the percentage of women or ethnic minorities on the board of directors or the executive team.

In this research, working out that first bit, the percentage of different people at the top of a company, is relatively straightforward.

But the issue is that there's lots of different ways to measure financial performance.

You could look at sales, you could look at profits, you could divide profits by the total assets, by the total earnings, you could take out accounting issues like depreciation, or you could look at total shareholder return.

McKinsey, the global management consultancy, has released a series of influential reports on this subject.

Which showed apparently compelling evidence that diversity is linked to performance.

More diverse companies equals better performing companies.

But there was a catch.

The results were just not replicable when you looked at alternative measures of performance.

So there was a replication which tried to look at six other equally plausible measures of performance and found that the results completely went away when you looked at the other five of the six.

And even in the one that they did

report themselves, they found that the correlation was much weaker than was reported.

But let's say, well, that was because they just didn't have the same data and they could have replicated it fully had they have the same data.

The point remains that actually if you look at all of the other plausible measures of performance, then you actually get nothing.

All of these are good things to think about.

The complex statistical judgments sitting behind plausible public pronouncements.

But it still leaves the question, what are people who aren't trained in statistical techniques meant to do about it all?

This is an important question because if the solution is you need to get a PhD in statistics, that's just unrealistic.

People just don't have the time to do that.

They don't have the time to look up every single reference and check every regression coefficient.

But instead what I'm trying to highlight in my book is that the skills to discernment are already within us.

So whenever you see a study published on LinkedIn that people don't like the sound of, there is no shortage of alternative explanations.

Maybe it was correlation but not causation.

Maybe this was just a hand-picked set of numbers.

maybe they looked at performance over one year and they should have looked at it over a larger time period, maybe this is too small a sample.

And so when you see a result that you want to be true, one simple trick I give is imagine you had the opposite result

and see how you would try to explain it away.

All we need to do is to unlock the natural discernment that we have anyway, which we already deploy selectively when we see things that we don't like.

But the idea of imagine the opposite ensures that we also use that discernment even when we want something to be true.

Thanks to Alex Edmonds, author of May Contain Lies.

And that's it for this week.

If you've seen a stat in the news that you think needs a stern look, please do let us know.

Our email is moreorless at bbc.co.uk.

Until next week, goodbye.

Suffs, the new musical has made Tony award-winning history on Broadway.

We the man to be home.

Winner, best score.

We the man to be seen.

Winner, best book.

We the man to be quality.

It's a theatrical masterpiece that's thrilling, inspiring, dazzlingly entertaining, and unquestionably the most emotionally stirring musical this season.

Suffs!

Playing the Orpheum Theater October 22nd through November 9th.

Tickets at BroadwaySF.com.