How Artificial Intelligence Is Already Changing Sports, with Daryl Morey and Sendhil Mullainathan

Welcome to Pablo Torre Finds Out.

I am Pablo Torre and today we're going to find out what this sound is.

And anger is all that drives Sixers fans as far as I can tell.

Right after this ad.

You're listening to DraftKings Network.

If you're looking to add something special to your next celebration, try Ramy Martin 1738 Accord Royale.

This smooth, flavorful cognac is crafted from the finest grapes and aged to perfection, giving you rich notes of oak and caramel with every sip.

Whether you're celebrating a big win or simply enjoying some cocktails with family and friends, Remy Martin 1738 is the perfect spirit to elevate any occasion.

So go ahead, treat yourself to a little luxury, and try Remy Martin 1738 Accord Royale.

Learn more at remymartin.com.

Remy Martin Cognac, Feen Champion, African Alcohol by Volume 40 by Remy Control, USA Incorporated, New York, New York, 1738, Centaur Design.

Please drink responsibly.

So I need to jump in at the top of this episode to give you a warning.

A couple weeks ago, I was asked if I wanted to record a live episode of this show at the Sloan Sports Analytics Conference, which is a conference, if you did not know, that started at a lecture hall at MIT when I first attended it 16 years ago now.

But quickly, it moved into a convention center in Boston on account of the thousands and thousands of people who wanted to actually go.

And many of those people annually include the most powerful people in sports at this point.

We're talking lead commissioners, billionaire owners, Barack Obama one year.

Pretty much every data-driven team executive in American sports and beyond shows up.

But the guy presiding over all of this, the conference's co-founder, is Daryl Maury.

Daryl is now the president of basketball operations for the Philadelphia 76ers, and he is also, it turns out, a listener of this show.

And so I do feel journalistically obligated to point out that the Sixers are now about 20 games under 500, having a truly horrible season, as we'll discuss.

But the thing I really wanted to find out about at Sloan, even more pressingly, was a question I did not have a good answer to, I realized.

Despite how much everybody is always talking about large language models, LLMs, like ChatGPT, and also when the robots are finally going to achieve artificial general intelligence, AGI, and, you know, think like human beings actually do.

All of which is to say, what I wanted to find out about today is what AI is actually doing to sports in particular.

Like, what's the signal here, as in the relevant information worth knowing, and what's just noise that we should discard?

And so the warning I need to give you is the same warning that I gave our live audience in Boston as I was introducing Daryl and my other honored guest today, a man named Sendil Mulaynathan, who is the perfect person to join us and demystify this subject.

This is going to be the nerdiest episode of my show.

I'm glad to have all of you here listening and watching.

This will be on YouTube, I presume, and you will then get to see how nervous Daryl Maury is.

I'm nervous, yes.

You've been complaining about this all weekend.

I just send hill.

I know.

I hope I'm not insulting you by

the

genius grant.

It's very intimidating.

It is legitimately fun to be here alongside an actual certified genius

and also an executive who gets to be sarcastically referred to as one.

Is that the only way you could could be labeled genius?

Is the MacArthur?

I think so.

We've copyrighted it.

Oh, nice.

He even said we.

I know.

But he's on the MacArthur Foundation board.

I mean, Sendle's bio.

We could just read his bio for a while.

But just to catch you up on why Daryl and I are both incredibly excited to be sitting next to Sendle, MIT professor splitting his time between electrical engineering and computer science and economics, focusing on the intersection, I guess, broadly speaking, of algorithms and people.

Co-author of arguably the most highly cited economics paper in the 21st century on differences and differences estimation, which is a thing I definitely understand.

You can subtract, right?

And also, landmark papers on algorithmic bias and poverty in the criminal justice system, and now training his brain, his human brain, on artificial intelligence.

And also, Daryl's here.

So, Daryl is killing me.

But the resume for Daryl is

hasn't won yet.

Well,

historically, right, over a 60%

win rate as a GM.

This is, though, barring, I don't know, some act of God or the devil, your first losing season that you're grappling with.

It is my first losing season in, I think, 17 years, yeah.

So feel vulnerable being here on stage?

I do.

I feel very vulnerable, yes, yes.

Plus, you're a good interlocutor,

whatever they say.

I do want to interlock

to with you

um if that's how i betray my own diploma how long you've been using large language models i want to start with this first and i want to also establish just a very basic premise of how long you've been using it and also how are you using ai in your job as president of basketball operations currently how does that look

So I've used them a little longer than the open AI moment, which was what, November two years ago, three years ago now.

with the GPT

I believe three you could get some early access I wasn't really special there except I knew about it I knew people who knew about it and it seemed very cool I even I didn't realize how cool until it went public.

I think that was like sort of a moment that even surprised the AI researchers that I think that they were like, yeah, this is really cool.

But then once the public got a hold of it, it was very cool.

And then in terms of my job, we use,

we've used a version of AI for a long time.

We mentioned this on an earlier panel.

AI is just forecasting.

My job is making good decisions.

It's all about forecasting.

We've used machine learning models for forecasting for a long time.

And primarily, the big areas that would be most relevant would be things like forecasting draft picks.

There are a whole bunch of different forecasting techniques, traditional regression, linear regression,

tree models, you know, graphical models, there's like a whole set that have been tried.

And they're all like a flavor of forecasting of which

even goes into large language models later.

And then

for like pick and roll

identification in computer vision,

turns out even humans can't agree on what a pick and roll is or isn't.

The most basic one everyone can agree on, which is one guy hits another guy, which keeps him from guarding his guy and then something happens that but there's all these like phantom pick and rolls and you act like you're gonna set it and you don't you even humans can't identify it and computers do a pretty good job but not a perfect job at that as well

and then on the LLM side the traditional stuff is helpful coding

you know speeding up writing projects

things like that those are ones we use and then there's some shooting stuff as well so But just to be sort of,

I don't know, top line here, when you're making a decision and insert your favorite Sixers decision here, are you asking an LLM

for their input?

So we, two answers there.

We absolutely use models as

a vote in any decision.

And how much of that decision depends on

how much the assumptions behind those models don't change and their past success rates.

So, if you have a highly successful model for picking draft picks,

and we think that the game, you know, Evan hasn't changed the rules yet, and so the game isn't changing.

Evan Wash, who's in charge,

thank you of defeating you.

Yes, then the models get a lot of weight in those decisions.

It turns out LLMs do fairly well at prediction.

They still are not beating human, like super forecasters, they're called, like in the Human Judgment Project talks about it.

But they do add signal over just scouts and things like that.

So we'll treat them almost like one scout.

Now, over time, if it's shown that they have a better track record than scouts, we might wait the LLMs or use multiple LLMs as one or two, two votes out of the process.

So it turns out I'm building on a lot of things that Sendill knows, and you know that

multi-model approaches can put you more on the decision frontier, the optimal decision frontier.

And when I say that carefully,

you get very similar accuracy with lower risk by using multi-model approaches.

And so that's why we do that.

I would just like NBA Central to aggregate what Daryl just said, which is that he asks ChatGPT for help on trades.

Yeah, let's not, please not aggregate.

I'm just trying to

pay for my kids' tuition, Daryl.

Sendal, look, part of the larger conversation here.

I was in San Francisco for All-Star Weekend.

The word cloud over All-Star Weekend was artificial intelligence, was AGI, artificial general intelligence specifically.

Your view, though, on the ambition of AGI, thinking like a human, it sounds like what to you given your research?

It sounds unambitious.

Yeah.

I think that AGI on the surface feels like an ambitious ambitious goal, but it's a shockingly unambitious goal.

Let me articulate why.

We already have people who can think like people.

People who can think like what?

People.

Okay.

Like we have that, and we have ways to make more of those people.

And they're fun ways, actually.

That's the next panel.

Yeah, that's the next panel.

But

what I think is missing, and you've done a lot of this, what I think we need is algorithms that can do things people could never do.

And to me, that's the fundamental expansion that we're going to get.

We need algorithms that can see things that we can't, that can notice things that we never could.

And the reason I think the AGI part is unambitious is it's also too hard in the sense that we've had algorithms, vectors along which algorithms can see things that we absolutely can't.

And those are actually the easiest paths to improve performance.

To exactly do what we do with very high fidelity, that's actually really hard and not clear there's that much gain.

And so

let me see if I can give a good example.

My favorite example is The Wizard of Oz.

It's by far my favorite example.

How many of you have seen The Wizard of Oz?

Raise your hand.

Okay.

How many of you read the book?

All right.

How many of you read the book?

How many of you read all 30 of the books?

No.

I read one other and it was weird.

I've just found out there are 30 books.

Exactly.

The Wizard of Oz, Cinematic,

and in fact, it gets even weirder.

There are 30 books.

14 of them were written by Frank Baum.

15 of them was written by this woman, Ruth Plumby Johnson.

And the 15th book is the source of a big literary kerfuffle.

Who wrote it?

Frank Baum died while that was being written.

And so amongst literary critics, they just spent a long time close reading that book.

And they couldn't get, they couldn't agree.

And so in 2002, there's a article in the 2002, think of how early this is, in it, an article in this magazine that says, I've done NLP and you read the analysis and it is so clear, like with 100% accuracy, that the algorithm can see that until chapter 12, it's all Baum.

After chapter 12, it's all Ruth Plumley Johnson.

How can it be that an algorithm can notice with so much clarity something that humans have utterly not noticed in our own native domain?

And we see that again and again, which is algorithms are fantastic at picking up signal on the kind of things we didn't even know to notice for, because they notice aggregates.

They don't have prejudgments of where to look for signal.

So to me, I think the ambitious AI goal is actually to find algorithms that can help us know.

Like I loved what you were saying about how you, in scouting, you have another scout.

I think, Pablo, you used this phrase.

I would love it if there was another scout, not of the players, but as you refer to it, self-scouting.

Something that scouted me.

And we do that.

We have a self-scout.

Yeah, just to flag any, like if another team was analyzing us, what would they do?

Wait, so

explain that, yeah.

Can I go back and then come back to that?

Because you mentioned something that we say all the time, which is that on a single game, humans do better than any algorithm.

But similar to your L.

Frank Baum example,

algorithms are very good at seeing these 2 to 5% patterns that

humans miss because we see big things, we see big patterns,

we find big patterns when there aren't any, and there's

all these kinds of things.

So, we have automated tools that say, when I'm about to play this team, here are the kinds of things, you know, that player is very efficient, we should limit this opportunity here.

So, we flip that on its head and self-scout and say, what, when they play us, what are they likely saying?

We don't know for sure.

That's a little different.

So, yeah, what I would love is a model of Daryl that you look at.

No, no, but that you look at, and that says to you, hey, just to you, hey, I think this is the kind of situation where you often do this.

Like something that helps you get candidate biases of yourself.

And in a way, what I like about that is it doesn't need to be perfect at that.

If it was 20% right at that, boy, what value add.

Because it'll notice the type of things we might not notice about you.

So I think once you think of algorithms as where do you deploy them to find things we cannot look for, I just think it's going to open up a huge avenue of.

Can I ask a question?

Because you gave an optimistic take of where we should take AI or even AGI, which left a role for humans, left humans at the center.

And I know that as someone who plays a lot of chess, that was also the belief in chess that there would be this role for humans over time.

And in fact, there was a time where a human machine combo could be a centaur.

A centaur, yes.

One of the great terms of art for anything a human plus a computer being the most dominant combination of any just humans just computers they were the best it is called centaur chess for example as you say but it was in the early 2000s where that went away and actually any human involvement would mess up

would beat a would beat a machine with no human involvement would beat any human involvement and so that that I worry that you know we may face that as well so I love the chess example because it so clarifies many things.

There's this term, though a bit technical, I think it's very helpful, called closed world versus open world problems.

Closed world problems are ones where everything is specified.

Let's say chess satisfies that.

But I'm going to come back and how even chess may not satisfy that in a second.

But let's say chess satisfies that.

All the rules are known.

All the objectives are known.

Everything is there.

We will never beat an algorithm in a closed world problem.

The idea that we're going to beat it, conditional on rules, conditional on things, it's crazy.

I mean, you can just compute your weight of stuff.

And there's an analog to closed world problems where closed world problems, where maybe we have enough data.

Again, we're not going to beat algorithms.

But most phenomena in the world are not really closed world problems.

They're open world problems.

What that means is what are the things that you can do?

The moves in chess are prescribed.

The moves in life are not prescribed.

And some of the most genius insights of someone realizing, hey, I can actually make this move.

This is a feasible thing to do in this situation.

Similarly, the objectives are not specified.

And it may seem like the objectives are specified, but if you think of some of the great innovations that people have had, it has been realizing, oh, that's not the problem we're trying to solve.

This is the problem that we're trying to solve.

Like, I am not in the burger business.

I am in the customer satisfaction business.

I'm not in this business.

I'm in this business.

And so specifying objectives, specifying moves, specifying the rules of the game, once you've done that, the algorithm will clearly win.

So you're basically saying

if we're playing Centaur chess, the computer doesn't know we can still flip the board.

Right.

And another example of chess is, is chess about winning?

For example, suppose you're playing with a friend.

In one sense, the goal is winning.

But in one sense, the goal is to really have like a fun game.

Can you build an algorithm that will have a fun game with you?

Now, suddenly that gets interesting, and there are people who do chess algorithms, have been trying to figure out how do we build a fun game algorithm.

And so that suddenly that's no longer in the realm of just computing.

You've now got to actually get into it and people get involved.

But that brings us right back to sports.

Yeah.

So to me,

the problem, the tension of our time that Daryl has had a hand through this conference actually in shaping is this tension between entertainment and efficiency.

And I always think about when I mentioned the league office trying to keep up with people who are trying to innovate, there is this push and pull of how do we stay ahead of the people people who are the most clever when solving these wicked problems, these open questions.

And so when it comes to how we're figuring this all out, right, currently we're all set up so that Daryl's over there, the players are trying to catch up and figure out, okay, how do we fit into this?

And then the league office is somewhere else, right?

Is there a way, Sendo, to actually acknowledge that a

truly optimal goal here is for everybody to be aligned on what the best outcome is, which involves entertainment.

Yeah,

and I'd be curious what you say about this.

I do think part as a fan, I'm like a big super fan.

I'm a super fan of YouTube.

You know, had an account on RealGM

on the Sixers message board just to out you immediately.

Yeah, yeah, more than an account.

I was so.

What's the account ID?

Well, no, it gets worse there.

No, no, it gets much worse.

This is when I was trying to get tenure at MIT.

I was so obsessed with Real GM that I then became a moderator for the Sixers Message Board.

Just like genuine Sicko behavior.

Yeah, this is

a little bit different.

Was this when Sam was there?

No, this is 2000.

This is when I was 2000.

Oh, this is.

Yeah, I should not have been doing this.

It was really quite.

You seem to have done fine.

Yeah, yeah.

I hope.

I won't give away my user ID.

But I do think as a fan, one of the things that I've been doing.

We can just put NLP on it and figure it out.

Exactly.

I think you should be.

We're going to find out, Daryl.

Don't worry.

Natural language processing.

Yeah, this is a Frank Baum situation all over again.

What I do think, and I'm really curious your guys' opinion on this, I think as a fan, what I've noticed is that wins matter, but it's shocking to me how much of my pleasure comes from a bunch of other things.

The underdog story, the person who just shouldn't be scoring this much, but scores this much.

Like I was happiest as a Sixers fan during the process years, which is a weird thing to say.

I would watch every game, and I don't know what that reveals about sport, but it is.

It reveals that this season for the Sixers is going to be a really enjoyable one for you.

No, no, no offense, Daryl, but it has not been.

And I'll tell you why it is not.

Me neither.

Yeah,

we're in the same boat.

It's not been because the storylines have not been captivating.

And if I think of sports as entertainment, like I would love to know, even, I would love to turn an algorithm on what makes a captivating storyline.

There are just some storylines that just keep you hooked.

You want to go back for the next game.

you want to know what's going on.

And so, is that part of the objective function, or is that an illusion?

Like, look, you guys are thinking to yourself, storylines matter.

We've looked at the data, there's only one thing that matters, it's wins.

I don't know.

I'd love to know.

I don't focus on wins, I focus on championship.

And I would say, the one the Philly fans surprised me in this manner.

So, I guess it's two seasons ago now.

We were up 3-2 on the Celtics.

Objectively, in Vegas, we were at the highest odds to win the championship that year because they felt that the team out of the East would win, I guess.

It wasn't like above 50.

I've scraped 50 when we were up on the Warriors in 18.

And we were up 3-2.

Obviously,

we then lost a close game at home, which was very frustrating.

And then we had a very bad loss in game seven.

And it's a little bit similar to the LeBron-Michael Jordan argument.

Objectively, people should feel better

about that.

Then the next season, we had a very entertaining Knicks series, like almost famously entertaining.

I think it was voted best series that year.

And for me, it was just stressful.

But for everyone else, I'm glad you enjoyed it.

It was very stressful.

And we lost in six games, you know, to a Knicks team that was not as good as the Celtics team we lost to the year before.

And I was very down after the loss.

And I was like, man, people are going to be so angry.

And they were angry.

Obviously, everyone wants us to win.

But they were less angry than the year before.

And anger is all that drives Sixers fans, as far as I can tell.

They were less angry.

Anger and irrational anger.

Yes, yes.

The aggregators were going to have a field day with this, Pablo.

I know you're excited.

My kid's going to go to private kindergarten.

But they were less angry, was my perception.

I don't know for sure.

It's not like you surveyed everyone, how angry are you?

And that really threw me.

Because to me, I'm just trying to win the title.

And so the year before, for our, the way we measure it was objectively better, even though still a zero on the Boolean scale I'm judged on.

So it should all be the same.

And that surprised me.

Can I put on my behavioral scientist hat for a second?

I was going to say, I was going to purely speculate.

Well, you're a Sixers fan, so you can be direct.

No, which is that I even think negative emotions are fine.

I think the thing that is dangerous is disinterest.

And if you think about sports,

it's easy to dismiss sports as entertainment.

But what else in our culture can have people of very different backgrounds, different generations, talk to each other and feel they've meaningfully touched, you know, communicated, you know, had a meaningful meeting of the minds?

You know, parents and children, grandparents.

It's quite a unique place as a cultural artifact.

And the main thing that it does, if I think of its core contribution to society, is it provides conversation.

And I think the worst thing is anger is good.

You got something to talk about.

Like during that, after that Celtics thing, you meet a Sixers fan, like it was easy.

You knew what you would do.

You would complain.

And complaining with another person is its own form of like connection.

It's those periods when you don't really have anything to talk about.

Like it's this interest,

this feeling that the story is old.

It's like kind of stale.

It's in that, that were always interesting, yeah.

Right, right, the curse

is that, and you, it is always interesting, and so I wonder to what degree one should just be maximizing engagement.

I know that we've had bad experiences with that, but it's Pablo's department, that's right.

Uh, Sendal, if you want to trash any other geniuses, feel free

If you're looking to add something special to your next celebration, try Remy Martin 1738 Accord Royale.

This smooth, flavorful cognac is crafted from the finest grapes and aged to perfection, giving you rich notes of oak and caramel with every sip.

Whether you're celebrating a big win or simply enjoying some cocktails with family and friends, Remy Martin 1738 is the perfect spirit to elevate any occasion.

So go ahead, treat yourself to a little luxury, and try Remy Martin 1738 Accord Royale.

Learn more at remymartin.com.

Remy Martin Cognac, Feen Champion, 14 Alcoholic by Volume 40 by Remy Control, USA Incorporated, New York, New York, 1738, Centaur Design.

Please drink responsibly.

But I actually want to bring it back around.

So like the larger picture here that we're sort of saying without explicitly saying it is that the nerds won, okay?

Right?

Like the whole debate about like, oh, when are the Lakers going to show up at Sloan, which is a headline for a long time, the last holdout of this conference from the basketball ops department used to be the Lakers a dozen years ago.

Then everybody, everybody shows up.

And Jason Bus came, actually, is the first Laker.

Yes.

But I say that because when it comes to the distribution, the wide, broad access to LLMs, and I want to get to beyond LLMs in a second here, but when it comes to that, every team, even the most caveman old school team having access to ostensibly an army of robot Daryls.

Isn't the person who should be worried the most is the guy who founded the conference, who had as his competitive advantage, this natural curiosity and skill and career premised on doing things that you couldn't just automate?

Should Daryl be worried, I guess, Sendal?

Is that the subtext here of everybody having access to these devices?

The short answer is definitely not.

And it's worth talking through which experts should be worried and which experts shouldn't.

Please.

Some experts absolutely should be worried.

There are experts where the nature of their expertise is the reliable, repeated application of some tacit knowledge.

So there are, you know, a very good doctor is a reliable, repeated application of tacit knowledge about diagnostics.

Now, it's not that those algorithms today can do that.

I think we're far, but it is also totally plausible that we will get to the point where...

You mean like reading an x-ray or something.

Reading an x-ray.

Yeah.

It's like it's easy to say that's going to happen next year, but that's not going to happen.

But it is definitely like it is.

Do not tell your kids to go into radiology.

So

you'll be happy since we both had moms who told us to go into medicine.

We will.

The nerds will win over the doctor.

Cindy and I bonded over the trauma of not becoming a doctor in his office yesterday.

Yeah.

It's real.

But so that repeated application of testing knowledge.

On the other hand, I think

the genius of someone like Daryl and experts like that are people who

can take

an ill-formed problem

and actually formulate it in a way that it becomes tangible and actionable.

It's easy to say, oh, the three-point, but it was recognizing in this ocean of stuff, let's focus on this part of the problem, is actually very, very, very hard.

The reason algorithms are complementary to that, it goes back to the chess thing.

Once you formulated the problem, we can now use algorithms to bulldoze through it, which then raises the enormous returns to problem formulation.

And I think people who can bridge the inchoate, underspecified, who knows what's even going on here, and bridge it back into the, I've now formulated into a data set or a specific type of activity, that skill is going to go way up.

And so I think the data science skills that are going to be worth less are the people who, once it's formulated, do stuff with it.

But that's not your genius.

And that then, does that make sense either?

Yeah.

And even just like, you know, Daryl, I had the opportunity, I was at the Sixers Eltics game with you.

And I was very curious, for those who remember Larry Kuhn.

Larry Kuhn, a retired computer scientist who had written the seminal text for understanding the NBA's salary cap and collective bargaining agreement, the NBA salary cap FAQ.

Larry was such a scarce resource.

That body of knowledge used to be be so scarce that everybody would consult this like blog spot page.

And it was Larry and just a very few people at the legal office and a few teams at the time.

Yeah.

But now ostensibly,

why can't an LLM analyze the CBA and take the jobs of the many humans who are currently tasked in the way that lawyers are tasked with analyzing documents?

In fact, it can, and so he had me give it what I thought was a medium-hard problem, and it answered it flawlessly.

This is on 4.5, the latest, from OpenAI.

Then we gave it a medium-plus hard problem.

It also answered flawlessly.

And then I was with Monty McNair from the Kings.

He formulated one that he thought was hard, and it did fail the hard one.

So I think we, yes, we have the same problem though, law firms have, which is even though it gets the answer 98.9% right,

you can't hand it to a client who's paying you that much money unless it's, well, nothing's 100, but you probably need a higher sigma on your, or at least to say there was a human who looked at it, right?

Because you're like, oh, you just took the algorithm.

You're like, yeah, that doesn't work.

That will work, though, eventually.

Or,

you know, where people will be like, that's a high enough sigma.

I'm going to take the answer from the algorithm.

And then I'd add one thing to that.

It's the problem as LLMs as they are today is not.

just their error rate, but the inscrutability of their errors.

Explain that.

Yeah, explain that.

We have almost no way of knowing the form factor and where their errors are going to come from.

That is, you tried some hard ones and got it to fail.

I bet if we had an hour, we could find some easy ones that it fails on.

And it's unclear to you why it failed on this easy one, but got this easy one?

So this has been my test since the LLMs came out.

And only 4.5 is the first one to have solved this.

I would give it, I've since the open eye first one, I've given an artificial Monty Hall problem to the LLMs, and it's perfect because they're trained.

There's been so many articles written on it.

It's trained on a billion articles, and the answer is always switch in these articles.

Well, with some tiny exceptions.

So I've been giving it a Monty Howell problem where,

and just so folks know that, three doors.

You pick a door, I had to pick door one.

The host then opens another door and says, okay, you can either keep the door or do you want to switch?

And it turns out if you switch, you get two out of three probability.

If not, most of our audience probably knows this.

But the key to the whole Monty Hall problem is Monty Hall knows which door to open.

So I formulated, I have to change it.

I put it in sacks.

And then I have a person just walk by and say there's nothing in a sack, but without any knowledge of the others and say, do you want to switch?

In that formulation, it's no advantage to switch.

And 4.5 was the first one that got it right.

All the other ones were like switch.

Yeah, yeah, fantastic.

And so that's been my little test for how advanced they're getting.

But those are the kinds of problems you're talking about where like change a little assumption and it changes a major part of the answer.

They will miss it because they've been trained heavily on the correct answer based on what they knew.

Yeah, it's tempting to think that they're reasoning their way to an answer.

When there's something else they're doing, we have this paper we're finishing called Potemkin Understanding, where it's very easy to generate these little Potemkins, where it seems to get this right, then why is it getting

this other thing utterly wrong?

And you can start algorithmically generating, and you see it's kind of like swiss cheese their knowledge this thing and this is the new york driving oh this is another paper we're just finishing now so it's like so it's like an example of this is like you can you can just give it a bunch of examples and say

what's an a b a b rhyme scheme great and then you write a little thing take out a word and say can you fill this in to rhyme it

can or cannot sometimes it gets it right sometimes it doesn't and when it doesn't get it right you take that word and you say to another instance does this word rhyme with this word and it says no.

So it's like

there's incoherence underneath the object.

Well, that's where the inference time comes out.

That's where you.

They just repetitively ask themselves questions before they answer the first time, right?

So you're hoping that those type of tricks kind of get this stuff out of the way.

But it is a kludge.

The more I go into the inference time stuff,

it's basically a kludge on maybe a fundamental problem, but I'm not quite sure.

No, it's definitely a fundamental problem.

It really is.

I think what's happening in this space is that when you play with these LLMs, it's just so amazing.

They feel like they've understood.

It's hard not to get pulled in and feel like, oh my God, this is a thinking creature.

What you're saying is that the fact that it's a language model talking to us has fooled us into thinking that there is a depth that is not actually present.

That's exactly right.

But that's probably true of most humans, too.

Fortunately, that's

like in interviews.

This is my favorite interview tactic is they, I don't care what they bring up.

I sadly might know a little bit about everything and I'll just like start asking and if they can't get past like two layers and you're like, okay, they don't really understand that.

But I do want to get to just the job, right?

So, okay, so there are versions of what I imagine that CBA problem is, where it's like,

could you have an LLM

call a play and then you consult in the huddle, okay, this is what our robot assistant coach is proposing.

You have a person check it.

I can understand they're at the table in every decision.

But just the job, Daryl of what it means to be a president of basketball operations such that Centill has faith in your ability to persist past the broad access of everybody having these robot assistant GMs coaches and everything else how much of your job is dealing with incoming

like bad ideas frankly

yeah we often when I say we I mean like me and some other GMs across sports.

We feel often our job is sort of bad idea whack-a-mole.

Because Because ideas like form in these little narratives, and they can be good or bad ideas, but part of our job is to smother the bad ideas in the crib before they

be, you know.

I will say, I won't say the

assistant, but when I had Kyle Lowry

way back early in his career, what made him a great player was

his tenacity and everything, but it also made his personality somewhat challenging just to

assistant coaches.

And

basically,

we weren't winning.

We were winning about half our games.

And then

Kyle went out and we won six in a row.

And a narrative started forming,

we might be better without Kyle Lowry, which was so absurd in the data that it wouldn't even be something anyone who looks at data would consider.

But I could see how, you know, us, you know, we find patterns anywhere, could see that.

And I had to like work hard to like, to like be like, no, like this guy, like we had an easier schedule.

We had all the different things that would go into that.

And so that's, but that happens like multiple times a week, at least, if not more, where you're just trying to keep things on, like, let's get the big things right and not let these narratives form that might harm us.

Sendal, I want to move beyond LLMs because I know so much of your research is not merely these chatbots that are, again, the talk of every conference you'll go to at this point, but it's also supervised learning and how to train these models once you now account for a more realistic version of what being an executive in sports would be like.

Can you explain supervised learning and how your your fandom of sports has led you to see sports through that lens?

Yeah, so I think supervised learning differs.

I'll do one thing might be helpful to people.

Supervised learning is I collect a bunch of data of various kinds, and I ask the algorithm, what in this data predicts this particular outcome?

And as long as you can specify an outcome and some variables to predict with it, these models can sort of really start finding signal that we never even imagined.

And so, I think the playbook that emerges from that is to be very creative about finding things you'd like to predict.

And so, across disciplines, the best work has been getting clever about that.

That's why I love the Frank Baum story, which is the person who did that was super clever.

He's like, let's get some data and predict whether this was written by Baum or not.

And I've got some data that tells me ones that were, ones that were written by Ruth Plumley.

And so, I think for me, what makes sports so interesting is that there are is so many unknowns that you could imagine collecting data on.

And I'm sure you guys do a lot of this already, but even something like injury risk.

What is a good predictor of whether someone's going to get a certain kind of injury?

And if I just play that forward without knowing what you all do in that space, if I play that forward, it's kind of a nice toy example to start thinking about stuff.

Because first, you have to...

be creative to say we're not just about wins and what happens like injuries are big so why can't we turn the gun on that?

Then you have to be creative about deciding what input variables are you going to get to do that.

And that's where I think people haven't fully appreciated the biggest change that these algorithms

have rendered.

So that if I said that these algorithms can find a lot of signal in data, then a logical implication of that is we ought to invest in new data collection.

And there's huge returns to new data collection.

So what I would do is say, oh, what would I want to predict injury risk?

I don't know, what are some low-cost devices I could put on players?

What are some low-cost devices I could put, not even during the game, but while people are taking free throws?

Maybe if I had video of people taking free throws, I would start noticing small changes in form that indicate things that are wrong at a biomoskeletal levels.

So you kind of, the creativity becomes not in finding patterns in data, but in deciding what data to give the algorithm, either as an objective or an input.

And that's just a very different kind of open-world creativity.

And I think these out that path is, I think, going to take off in the next 10 years.

We're starting to see it in medicine.

Well, you tell me.

I think we'll see it in sports.

We probably already have.

For sure, yeah.

And

I don't know if you read it.

It sounds like you didn't.

I wrote a paper years ago.

on

you need better data, not better analysts.

And I actually met a lot of people

mad at the conference.

But yeah, the data is going to be the unique thing, not the analysts.

How much parity is there now in terms of the data teams have?

I think it's getting pretty high.

I mean, we're following baseball almost in a five-year delay, and

it's pretty homogeneous in terms of how the, I mean, I think the quality varies, obviously, but we're in a tighter band, much, much, much tighter band now, which is not good.

Yeah, not good.

Yeah.

question, though, of, you know, if someone were to approach Sendal and say,

we vetted your real GM posts and yet still,

and yet still, we think you should be the chief intelligence officer for our NBA team.

Do you have a sense of what you would want to do?

given your field of study now?

Yeah, I think, and I'd love to hear what you think about this.

I think what I would focus on is turn the camera, since so much of it has already been focused on the game, I would turn the camera elsewhere, for example, on the decision makers.

I would say, what are decisions that we think are being made all the time?

And how do I start creating what you call the self-like, how do I start creating algorithms that can help scout me and tell me where are my biases, where are the things where I screw up?

So that would be an example where I would start.

No, could you explain?

You had this landmark study of judges and bail.

I just would like everybody to appreciate what it's like when you turn the camera on a decision maker.

Yeah,

was, this study was basically pretrial judges have to decide who to release.

And they're making a prediction of risk.

Will the person come back for trial?

Will they commit a crime?

So the first part of the study was we had an algorithm predict a defendant risk.

And it turns out that does very well and could add a lot of value to the judge.

When I say a lot of value, Sometimes in sports, it's hard to know what the magnitudes are, but in this case, you could close Rikers in July and have the same crime rate.

So you could cut prison population, jail populations by 40%.

So there's a lot of

welfare left on the table.

But then we did two things that were surprising even to me.

So we didn't predict what the defendant would do.

We turned the camera and predicted what the judge would do.

And the first thing we found was that a predictor of the judge beats that judge.

So that if an algorithm just does whatever the judge would typically do, that is better than what the judge actually does.

And this is sort of a crisp way to see noise.

And so many of our decisions, we found this in domain again, domain, something that could just tell you, you seem to be doing something different than you usually do, like in many domains, medicine, judicial, so the predicted you does better than you is quite a

quite a scary thing.

I find that very disturbing at some level, and it's true across all experts.

That must mean you're just cutting off some outliers and tails.

I think that's regressing to the mean more.

Regressing to your own mean.

Yes, yeah.

It's sort of because what happens is we

use variables we really ought not to.

It's exactly to your point.

It's the Kyle Lowry point all over again.

I'm keying in on things in the moment that, like, hey, that's not what you normally do.

Why are you doing that?

Use variables that you probably want.

You have some other reason to be driving an outcome

in this case.

But can you mention which variables were driving the outcome in this case?

So then we expanded the prediction to not just be the things we thought mattered, but include a bunch of stuff.

And you identify about half of the variation in what judges do that they ought not to be doing comes from one variable in the data set, which is the mugshot.

So you can build a predictor of who the judge will release just by using the mugshot.

And it does extremely well.

More than any other characteristic.

More than whether they've committed a crime, what they're being charged for, how many times they failed to appear.

Just the mugshot.

And in the mugshot, it's not just the variables you would imagine.

It reminded me of your Marcassa thing, actually.

It was variables that even judges and public defenders didn't realize mattered.

So, for example, one of the biggest things that has about half the size of having committed a violent crime is if the person has a full or fat face.

Now, what's interesting about that one is when I asked.

The good or bad, that's how it is.

No, but isn't that weird?

We don't know.

You don't know.

No, no, no, we know.

Oh, but our intuition doesn't tell us, even though in the data, it's very.

I just, I'm looking at our mugs.

Yeah, I'm looking at that too.

It's not a good sign, is it?

Yeah, yeah, yeah.

Word for Daryl guilty, Sendal, is what we're asking.

So the good news is I think Daryl will not be going to jail.

So

you have a slightly full.

So I can get away with one.

Yeah, yeah, yeah.

But you're not well-groomed.

That's the other thing.

You need to.

Yeah, yeah.

Halfway there.

Well-groomed.

Well-groomed is good.

Slightly rounder face is very good.

But that's sad, though.

It should not be a fact.

Absolutely.

Not the same thing.

It's frankly, it's mind-blowing.

It's mind-blowing.

That they were not aware of this.

We would not intuit that based on our predisposition for bias.

Or you might predict the opposite.

You might have predicted the opposite.

And so I think these ideas that when the camera turn on us and help us understand what we do and what we're doing,

that would be probably the first angle I would take because a decision aid like that could help me understand myself.

So eliminate mugshots might be yours.

Well, they see the person.

Oh.

Right.

And so in some sense, but yes, if we could do everything behind a curtain, do like orchestras.

Yes, orchestras behind a curtain would be fantastic.

But to me, it's also just I think these algorithms can start ferreting things out about us that we did not know about ourselves.

And that's that's again a very optimistic thing for me.

I think it's part of the path that we've been growing on of becoming better stewards of ourselves.

At the end of every episode of Pablo Torre finds out, a show about finding stuff out.

We say what we found out today.

Daryl, would you like to begin?

Can we start with Sendle?

Sendal, would you like to begin?

Oh my God, I've been in the conversation so much.

I'm going to need you to say something, Daryl.

We're just playing hot potato here.

Yeah, I'm always picking up something.

I'm trying to pick.

So I did find out that Daryl is very attuned to whether fans are

interested, excited, not excited.

I think a lot of fans would have foolishly thought Daryl doesn't care about that, but he is shockingly attuned to that thing, which

ex posts makes sense, but I think fans would have been miscalibrated on that.

I think everyone who says they don't read it, don't read the comments, don't whatever, it's all not true.

We're all humans.

LeBron James reads the comments, as does Daryl Morris.

We learn that this week, absolutely.

I think probably one of the things I took away is

judge, the decision maker, and

yeah, look for times,

maybe even create a decision-making unit.

We wouldn't be able to make a Daryl decision algorithm because it's a federated across a head coach and multiple.

We could create a decision-making model, and then whenever it spits out a different answer, you should start asking questions.

Yeah, so that would be my takeaway.

Yeah, I found out.

What did you find out?

Yeah,

I found out that there's a non-zero chance that Sendle has trashed you online,

and that at the same time, he also thinks we'll probably be talking about this in 15 years.

So,

Sendle, Daryl, you guys for listening.

Thank you for doing this.

This is great.

Thank you, Pablo.

Okay, thanks.

That was fun.

Appreciate you coming over.

Yeah,

this has been Pablo Torre finds out a Meadowlark media production,

and I'll talk to you next time.