Holden Karnofsky - Transformative AI & Most Important Century

If we had AI systems that could do everything humans do to advance science and technology, that would be insane. We live in a weird time.
Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world.
You just imagine a universe where there actually are some people who live in an especially important time. And then there's a of other people who like tell stories to themselves about how what you know whether they do how would you want all those people to behave and it's like to me the worst possible rule is all those people should just be like now this is crazy and forget about it all right today i have the pleasure of speaking with holden karnofsky who is the co-ceo of open philanthropy in my opinion holden and is one of the most interesting intellectuals alive, well, given your role.
So Holden, welcome to the Lunar Society. Thanks for having me.
Okay, so let's start off by talking about the most important century thesis. Do you want to explain what this is for the audience? You know, my story is I originally co-founded an organization called GiveWell that helps people decide where to give as effectively as possible.
I'm no longer there, but I'm on the board, and it's a website called givewell.org that I think makes good recommendations where to give to charity to help a lot of people. And as we were working at GiveWell, we met Carrie Tuna and Dustin Moskovitz.
Dustin is the co-founder of Facebook and Asana and started a project that became Open Philanthropy to try to help them give away their large fortune, again, to help as many people as possible. And so I've kind of spent my career looking for ways to do as much good as possible with a dollar or with an hour, with whatever resources you have, and especially with money.
And so I've kind of developed this professional specialization in looking for ideas that are underappreciated, underrated, tremendously important, because a lot of the time that's where I think you can find just kind of outsized, what you might call outsized return on investment. Opportunities to spend some money and just get an enormous impact because you're doing something very important that is being ignored by others.
And so it's through that kind of professional specialization that I've actively looked for interesting ideas that are not getting enough attention. And then I encountered the effect of altruist community, which is a community of people basically built around the idea of doing as much good as you can.
And so it's through that community that I encountered the idea of the most important century. It's not my idea at all.
I got got to it from a lot of people. And the basic idea is that if we developed the right kind of AI systems this century, and that looks reasonably likely, that could make this century the most important of all time for humanity.
So the basic mechanics of why that might be or how you might think about that. So one thing is that if you look back at all of economic history, just the rate at which the world economy has grown, you see acceleration.
You see that it's growing a lot faster today than it ever was. And one theory of why that might be or one way of thinking about it through the lens of basic economic growth theory is that in normal circumstances, you can imagine a kind of feedback loop where you have people have ideas and the ideas lead to greater productivity and more resources.
And then when you have more resources, you can also have more people and then those people have more ideas. So you get this feedback loop that goes people, ideas, resources, people, ideas, resources.
And starting a couple hundred years ago, if you run a feedback loop like that, standard economic theory says you'll get accelerating growth. You'll get a rate of economic growth that goes faster and faster.
And basically, if you take the story of our economy to date and you just kind of plot it on a chart and do the kind of simplest thing you can to project it forward, you project that it will go, that our economy will reach like an infinite growth rate this century. And the reason that I currently don't think that's a great thing to expect by default is that one of the steps of that feedback loop broke a couple of hundred years ago.
So it goes more people, more ideas, more resources, more people, more ideas, more resources. A couple hundred years ago, people stopped having more children when they had more resources.
They got richer instead of more populous. And this is all discussed in the Most Important Century page on my blog, Cold Takes.
And so what happens right now is that when we have more ideas and we have more resources, we don't end up with more people as a result. We don't have that same accelerating feedback loop.
And if you had AI systems that could do all the things humans do to advance science and technology, meaning the AI systems could fill in that more ideas part of the loop, then you could get that feedback loop back. And then you could get sort of this unbounded, heavily accelerating, explosive growth in science and technology.
So that's the basic dynamic at the heart of it. So that's kind of a way of putting it that's trying to use familiar concepts from economic growth theory.
Another way of putting it might just be, gosh, if we had AI systems that could do everything humans do to advance science and technology, that would be insane. You know, what if we were to take the things that humans do to create new technologies that have transformed the planet so radically, and we were able to completely automate them so that every computer we have is potentially another mind working on advancing technology.
So either way you think about it, you could imagine the world changing incredibly quickly and incredibly dramatically. And so I argue in the most important century series that it looks reasonably likely, in my opinion, more than 50-50, that this century will see AI systems that can do all of the key tasks that humans do to advance science and technology.
That if that happens, we'll see explosive progress in science and technology. The world will quickly become extremely different from how it is today.
You might think of it as if there was thousands of years of changes packed into a much shorter time period. And then if that happens, I argue that you could end up in a deeply unfamiliar future.
I give one example of what that might look like using this hypothetical technology idea called digital people. That would be sort of people that live in virtual environments that are kind of simulated, but also realistic and exactly like us.
And when you picture that kind of advanced world, I think there is a decent reason to think that if we did get that rate of scientific and technological advancement, we could basically hit the limits of science and technology. We could basically find most of what there is to find and end up with a civilization that expands well beyond this planet, has a lot of control over the environment, and is very stable for very long periods of time, and basically looks sort of post-human in a lot of relevant ways.
And if you think that, then this is basically our last chance to shape how this happens. So that's the most important century hypothesis in a nutshell, is that if we develop AI that can do all the things humans do to advance science or technology, we could very quickly reach a very futuristic world, very different from today's, could be a very stable, very large world, this is our last chance to shape it.
Gotcha. Okay, so I and many other people are going to find that very wild.
So maybe you can walk us through the process by which you went from doing global development stuff to thinking this way. So in 2014, for example, you had an interview or a conversation, and this is a quote from there.
Maybe you can walk me through how you got from there to where you are today. I have looked at the situation in Africa, have understanding of the situation in Africa, and see a path of doing a lot of good in Africa.
I don't know how to look into the far future situation, don't understand the far future situation, and don't see a path to doing good on that front I feel good about. Yeah, first, first, I think I just I went on for a while, but I want to come back and connect this back up to the to the how this relates to the work I was even doing at GiveWell, why this is all kind of one theme.
If we are kind of on the cusp for this century of creating these advanced AI systems, then we could be looking at a future that's like very good or very bad very bad. And I think there are decent arguments that if we move forward without caution and we develop kind of sloppily designed AI systems, they can end up with goals of their own.
And basically, we'd end up with a universe that contains very little that humans value or a galaxy that does. We could also imaginably end up with a world in which very powerful technologies are used by just not very well-meaning governments to create a world that isn't very good or a world where we kind of eliminate a lot of forms of material scarcity and have a world that's much better than today's.
And so a lot of what I ask, a lot of what I ask to give all is how can we help the most people possible per dollar spent?

And if you ask how can we help the most people possible per dollar spent, then if you think that funding some work to help shape that transition, to help make sure that we don't move forward too incautiously, to help make sure that we do increase the odds that we do get a good future world instead of a bad future one, that's helping a huge number of people per dollar spent. So that's the motivation.
And now you're quoting a a discussion and argument I was having, where we posted a transcript back in 2014. And at that time was was that was part of my journey of getting here.
So I was talking to people who are saying, Holden, you want to help a lot of people with your resources, you should be focused on this massive event that could be coming this century that that very few people are paying attention to that there might be a chance to make this go well or poorly for humanity. And I was saying, gosh, like that sure is interesting.
And I did think it was interesting. That's why I was spending the time and doing the conversation.
But I said, you know, when I look at global poverty and global health, I see what I can do. I see the evidence.
I see the actions I can take. And I'm not seeing that with this stuff.
So what changed? I would say a good chunk of what changed is maybe like the most boring answer possible is I just kept at it. So I think I was sitting there in 2014 saying, gosh, this is really interesting.
But it's all a bit overwhelming. It's all a bit crazy.
I don't know how I would even think about this. I don't know how I would come up with a risk from AI that I actually believed was a risk and could do something about today.
And now I've just been thinking about this for a much longer time period. And I do believe, you know, most things you could say about the far future are very unreliable and not worth taking action on.
But I think there are a few things one might say about what a transition to very powerful AI systems could look like. There are some things I'm willing to say.
I'm willing to say it would be bad if AI systems were poorly designed, had goals of their own, and ended up kind of running the world instead of humans. That seems bad, and I do, and I am more familiar today than I was then with the research and the work people can do to make that less likely and the actions people can take to make that less likely.
So that's probably more than half the answer. But another thing that would be close to half the answer is I think the world has changed.
And I think that basically there's been big changes in the world of AI since then. So I think basically in 2014, that was the beginning of what's sometimes called the deep learning revolution.
And since then, we've basically seen these very computation intensive, but fundamentally simple AI systems and achieve a lot of progress on a lot of different unrelated tasks. And it's looking to me like not totally crazy to imagine that the current way people are developing AI systems, cutting edge AI systems could take us all the way to the kind of extremely powerful AI systems that automate roughly everything humans do to advance science and technology.
It's not so wild to imagine that we could just keep on going with these systems, make them bigger, put more work into them, but basically stay on the same path and you could get there. And if you imagine doing that, it becomes a little bit less daunting to imagine the risks that might come up and the things we could do about them.
So I don't think it's necessarily the leading possibility, but it's enough to sort of start thinking concretely about the problem. Another quote from the interview that today I find appealing because I haven't done the work you have yet.
Does even the upper crust of humanity have a track record of being able to figure out the kinds of things Miri claims to have figured out? And by the way, for context for the viewers, Miri is the organization Eliezer was leading, which is who we were talking to at the time. Yeah, well, I don't remember exactly what kinds of things Miri was trying to figure out.
And I'm not sure that I even understood what they were that well. So I definitely think it is true that it is hard to predict the future no matter who you are, no matter how hard you think, no matter how much you've studied.

I think that is true. I think parts of our kind of world or memeplex or whatever you want to call it overblow this at least a little bit.
And I think I was I was kind of buying into that a little bit more than I could. So I think, you know, probably in 2014, I would have said something like, gosh, you know, really no one's ever done something like making smart statements about what the, you know, several decades out future could look like, or making smart statements about what we'd be doing today to prepare for it.
Since then, I think a bunch of people have looked into this and looked for like historical examples of people kind of making long term predictions and long term interventions. And I don't think it's amazing, but I think I wrote a recent blog post entitled, The Track Record of Future as Seems...fine.
Fine is how I'd put it, where I don't think there's anyone who has demonstrated a real ability to predict the future with precision and know exactly what we should do. But I also don't think humans track record of this is so bad and so devastating that we shouldn't think we are capable of at least giving it a shot.
And I think if you enter into this endeavor with self-awareness about the fact that everything is less reliable than it appears and feels at first glance, and you look for the few things that you would really bet on, I think it's worth doing. I think it's worth the bet.
My job is to find things that we might do 10 things and have nine of them fail embarrassingly,

and one of them be such a big hit that it makes up for everything else.

And a lot of my job is to find stuff like that.

So I don't think it's totally crazy to think we could make meaningful statements

about how things we do today could make these future events go better,

especially if the future events are crazily far away,

especially if they're within the next few decades. So that is something I've changed my mind on at least to some degree.
Gotcha. Okay, so we'll get to the forecasting stuff in a second, but let's continue on the object level conversation about most important centuries.
So I want to make sure I have the thesis right. So is the argument that because we're living in a weird time, we shouldn't be surprised if something like transformative AI happens this century? Or is the argument that since transformative AI could happen this century, it's a weird time? Yeah.
So something we haven't covered yet but I think is worth throwing in now is that a significant part of the most important century series is kind of just making the case that even if you ignore AI, there's a lot of things that are very strange about the time that our generation lives in. The reason I spent so much effort on this is because my number one, back in 2014 and before that, my number one objection to these stories about transformative AI is, you know, it's not anything about whether the specific claims about AI or economic models or, you know, alignment research makes sense.
It's just this whole thing sounds crazy. And it's just suspicious.
It's just suspicious if someone says to you, you know, this could be the most important century of all time for humanity. I titled the series that way because I wanted people to know that I was saying something kind of crazy and that I should have to defend it.
I didn't want to be backpedaling or soft pedaling or hiding what a big claim I was making. And so I think my biggest source of skepticism has been just like, I don't have any specific objection.
It just sounds kind of crazy and suspicious to say that we might live in one of the most significant times or the most significant time for humanity ever. And so a lot of my series is just kind of saying it is weird.
It is weird to think that, but we already have a lot of evidence that we live in an extraordinarily weird time that would be on the shortlist of contenders for most important time ever before you get into anything about AI, just using like completely commonly accepted facts about the world. For example, if you chart the history of economic growth, you see that the last couple hundred years have seen faster growth by a lot than like anything else in the history of humanity or the world.
And if you chart anything about like scientific and technological developments, you can see that everything significant is packed together in the recent past. And there's almost no way to cut it.
You know, I've looked at many different cuts of this. There's almost no way to cut it that won't give you that conclusion.
One way to put it is that the, you know, the universe is something like 11 or 12 billion years old. Life on Earth is like 3 billion years old.
And then humanity is just a blink of an eye compared to that. You could call it 300,000, 3 million years old.
Human civilization is a blink of an eye compared to that. And we're in this really, really tiny sliver of time, couple hundred years, when we know, all or like just a huge amount of the technological advancement and economic growth.
So that's weird. You know, I also talk about the fact that the current rate of economic growth seems high enough that we can't keep it going for that much longer.
If it went for another 10,000 years, that's another blink of an eye on galactic timescales. It looks to me, and we can get to this, like we would run out of atoms in the galaxy, wouldn't have anywhere to go.
And so I think there are a lot of signs that we just live in a really strange time. I think one more I'll just throw in there, then we can move on, is that, you know, I think a lot of people who disagree with my take would say, look, I do believe eventually we will develop space colonization abilities, go to the stars,

fill up the galaxy with life, you know, maybe have artificial general intelligence. I just,

to say it's the century is crazy. I think it might be 500 years.
I think it might be a thousand years.

I think it might be 5,000 years. And a big point I make in the series is I say, well,

even if it's a hundred thousand years, that's still an extremely crazy time to be in,

in the scheme of things. If you just make, you know, if you make a graphic timeline and you kind

I'm going to be a, in the scheme of things. If you just make, you know, if you make a graphic timeline and you kind of show my view versus yours, they look exactly the same down to the pixel.
And so there's already a lot of reasons to think we live in a very weird time. We're on this planet where there's no other sign of life anywhere in the galaxy.
We believe that we could fill up the galaxy with life. That alone would make us among the earliest life that has ever existed, or would make us the earliest life that has ever existed in the galaxy, a tiny fraction of it.
So that is a lot of what the series is about. And I have sort of answered your question, but I'll do it explicitly.
You ask, you know, is the series about transformative AI could come and therefore this century could be weird? Or is it about this century could be weird, therefore transformative AI could come. The central claim is that transformative AI could be developed in this century.
And the stuff about how weird a time we live in is just a response to an objection. It's a response to a point of skepticism.
It's a way of saying there's already a lot of reasons to think we live in a very weird time. And so actually, this thing about AI is only a moderate quantitative update, not a complete revolution in the way you're thinking about things.
There's a famous comedian who has a bit where he's imagining what it must have been like to live in 10 BC. Yeah, people are just numbering.
It's 10 years before zero. It's nine years before zero.
It's gonna happen. Let's say somebody comes with the proof that current deep learning techniques are not scalable for some reason, and that Transformer DBI is very unlikely this century.
I don't know if this is a hypothetical where that would happen, but let's just say that's what it is. Even if this is a weird time in terms of economic growth, does that have any implications other than Transformer DBI of what you should be doing? Yeah, I think in general, having kind of learned about just how strange the time we live in is when you look at it in context.
And here, I just encourage people to go to my series because I have a bunch of charts illustrating this, and it can be a little bit hard to do concisely. But I think the biggest thing I take away from that is we should just, you know, we should really look for the next big thing.
I think if you'd been living 300 years ago, and you'd been talking about the best way to help people, a lot of people might have been talking about, you know, various forms of, you know, helping helping low income people, maybe they probably would have been talking about spreading various religious beliefs. And I think, you know, it would have seemed crazy to think that what you should be thinking about, for example, was the steam engine and how that might change the world.
But I think the Industrial Revolution was actually just an enormous deal and probably was the right thing to be thinking about if there's any way to be thinking about it, if there was any way to be thinking about how that would change the world and what might could what one might do to make that a world that could be better. So that's basically where I'm at on this stuff is I just think, you know, as a world, as a global civilization, we should have a really high priority on just saying we live in a weird time.
Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world.
We should make a list of all the things that might radically transform the world. We should make sure we've done everything we can to think about them, identify the ways we might be able to do something today that would actually help.
And maybe after we're done doing all that and we have a lot of the world's brightest mind doing their best to think of stuff and they can't think of any more, then we can go back to all the other things that we worry about. But I think right now the world invests so little in that kind of speculative, hey, what's the next big thing? Even if it's not super productive to do so, even if there's not that much to learn, I feel the world should be investing more in that than there is because the stakes, than it is because the stakes are extremely high.
And I think it is just like a reasonable guess that we're living in a world that's recently been incredibly transformed by the Industrial Revolution and the future could be incredibly transformed by the next thing.

I just don't think this gets a lot of discussion in basically any circles.

And if I got some, I would feel a lot more comfortable with the whole thing.

I don't think the whole world should just obsess over what the next transformative event is,

but I think right now there's so little attention to it.

I'm glad you brought up the Industrial Revolution because I feel like there's two implicit claims within the most important century thesis that don't seem perfectly compatible. One is that, you know, we live in an extremely wild time, that the transition here is potentially wilder than any other transition there has been before.
And the second is we have some sense of what we can be doing to make sure this transition goes well. I'm curious if you think that somebody at the beginning of the Industrial Revolution, if knowing what they knew and then could have done something significant to make sure that it went as favorably as possible, or do you think that that's a bad analogy for some reason? It's a pretty good analogy for being thought-provoking and for thinking, gosh, if you had seen the Industrial Revolution coming in advance, and this is when economic growth really reached a new level back in the 1700s and 1800s, you know, what could you have done? And I think, you know, part of the answer is it's not that clear.
And I think that is a bit, you know, a bit of an argument that we should maybe not get too carried away today by thinking that we know exactly what we can do. But I don't think the answer is quite nothing.
So I have a, I have kind of a goofy cold takes post I never, that I never published and may never publish because I don't know, I kind of lost track of it. But it's, it's kind of, it's kind of saying, well, what if, what if you'd been sitting in that time and you had known the industrial revolution was coming, or you had thought it might be, you had asked yourself what you could be doing.
One answer you might've given is you might've said, well, gosh, if this happens, whatever country it happens in might be just like disproportionately influential. And you know, what would be great is if I could help transform the thinking and the culture in that country to have a better handle on human rights and more value on human rights and individual liberties and a lot of other stuff.
And gosh, it kind of looks like people were doing that. And it kind of looks like it worked out.
So this is the enlightenment. And I think, you know, I even give this kind of goofy example.
I could look it up and it's all kind of a trollish post. But the example is it's like someone's kind of thinking through, hey, you know, I'm thinking about this sort of esoteric question about what a government owes to its citizens or, you know, when a citizen has a right to overthrow a government or when it's acceptable to kind of enforce certain beliefs and not.
And it's like the other person, the dialogue is just like, this is the weirdest, most esoteric question. Why does this matter? Why aren't you helping poor people? But, you know, these are the questions that the Enlightenment thinkers were thinking about.
And I think there is a good case that they came up with a lot of stuff that really shaped the whole world since then because because of the fact that the UK became so influential, really laid the groundwork for a lot of stuff about the rights of the governed and free speech and individual rights and human rights. And then I go to the next analogy and I'm like, okay, now we're sitting here today and someone is saying, well, instead of working on global poverty, I'm studying this kind of esoteric question about how you get an AI system to do what you wanted it to do instead of doing its own thing.
And it's, you know, you could, you could, yeah, it's not, I think it's not completely crazy to see them as analogous. Now, I don't, I don't think this is what the Enlightenment thinkers were actually doing.
I don't think they were saying this could be the most important millennium. So let's do this.
But it is interesting that you, it doesn't look like there was nothing to be had there. It doesn't look like there's nothing you could have come up with.
And in many ways, it looks like what the Enlightenment thinkers were up to had the same kind of esoteric, strange, overly cerebral feel at the time and ended up mattering a huge amount. So it doesn't feel like there's zero precedent either.
Yeah. Maybe I'm a bit more pessimistic about that because people like – the people who are working on individual rights frameworks like Log, I don't think they were like anticipating an industrial revolution.
Yeah. I mean, I feel like the person actually did anticipate the industrial revolution probably his political philosophy was actually probably a negative given, you know, I'm talking about Karl Marx.
Karl Marx. Okay.
Yeah. Yeah.
So it's not obvious to me that even if you saw something like this happening that you're – Oh, it's totally not obvious. I mean I think my basic position here is we – I'm sitting here like highly confident i'm not saying you know there's tons of precedent and we know exactly what to do that's not what i believe i believe we should be giving it a shot i think we should be trying and i don't think we should be totally defeatist and saying well it's it's so obvious that there's never anything you could have come up with throughout history and humans have been helpless to predict the future i don't think that is true and so yeah i think i think that's like enough of an example to kind of illustrate that.
And I mean, gosh, like you could make the same statement today as you could say, look, doing research on how to get AI systems to behave as intended is a perfectly fine thing to do at any period in time. It's not like a bad thing to do.
And I think John Locke was doing his stuff because he felt it was a good thing to do at any period in time. But the thing is that if we are at this crucial period of time, it becomes an even better thing to do and it becomes magnified to the point where it could be more important than other things.
Now, the one reason I might be skeptical of this theory is that I could say, oh gosh, if you look throughout history, people were often convinced they were living in the most important time, or at least an especially important time. And if you go back, I mean, everybody can't be right about living in the most important time.
So maybe I should just have a very low prior than they Sure. like an interesting thing to look at, but just from, you know, from stuff I've read about whatever past works on political philosophy and stuff, I don't exactly see this claim all over the place.

It definitely happens. It's definitely happened.
And I think a way of thinking about it is there,

you know, there's kind of two reasons that you might think you are especially important. One is that you actually are, and you made reasonable observations about it.
And another is that you're,

you know, you want to be, or you want to think you are, you're self-deceiving. And so, you know, over the long sweep of history, a lot of people will come to this conclusion for the second reason.
And most of the people who think they're the most important will be wrong. So that's all true.
And that certainly could apply to me and it certainly could apply to others. But I mean, so, so I think that's like, that's just completely fine and completely true.
And I think we should have some skepticism when we find ourselves making these kinds of observations. And at the same time, I think it would be like a really bad rule or something or a really bad norm that every time you find yourself thinking the stakes are really high or that you're in a really important position that you just decide to ignore the thought.
I think that would be very bad. Like if you just imagine a universe where there actually are some people who live in an especially important time, and then there's a bunch of other people who like tell stories to themselves about how what you know whether they do um how would you want all those people to behave and it's like to me the worst possible rule is all those people should just be like nah this is crazy and forget about it that's like the worst possible rule because the people who are living at the important time will then do the wrong thing um i think another bad rule would be that everyone should take themselves completely seriously and literally and just and just promote their own interests ahead of everyone else's.
And a rule I would propose over either of them is that all these people should take their beliefs reasonably seriously and try to do the best thing according to their beliefs, but should also adhere to common sense standards of ethical conduct and not do too much ends justify the means reasoning. You know, I think whatever whatever people are are trying to do to help the most important century go well, it's totally good and fine to do research on AI alignment, but people shouldn't be like going and, you know, telling lies or breaking the law in order to further their ends.
That would be my proposed rule is that is that we should, you know, when we have these high stakes, crazy thoughts, we should do what we can about them and not go so crazy about them that we break all the rules of society. And that seems like a better rule.
That's a rule I'm trying to follow. Okay.
Can you talk more about that? So if for some reason you can convince that the expected value calculation was immense, that you had to break some law in order to increase the odds that EI goes well, I don't know what hypothetical this would be. Is it just that you're not sure that you would be right? So you just want to err on the side of caution? Pretty much.
I'm really not a fan of ends justify the means reasoning i just think it's like historically looks that's the thing that looks really really bad is is people kind of saying you know it's it's worth doing horrible things and coercing each other and you know using force um to to accomplish these things that you know the the ends we're trying to get to are more important than everything else so i'm i'm against stuff. I think that stuff looks a lot worse historically than people like trying to break the future and do helpful things.
So I see my main role in the world is as trying to break the future and do helpful things. I can do that without, you know, without doing a bunch of harmful or common sense, unethical stuff.
Maybe someday, you know, there will be one of these intense trade-offs. I haven't really felt like I've run into them them yet and if i ever ran into one of those intense trade-offs i'd have to ask myself how confident i really am the current level of information and confidence i have is in my opinion not enough to do to do really unjustified the meansy stuff yeah okay so let's talk about the potential implausibility of continued high growth one thing somebody might think is okay maybe two percent growth can't keep going on forever but you know maybe growth slows down to like 0.5 percent a year or something and as you know small differences in growth rates have like big effects on the end result so by the point that we've exhausted all the possible growth in the galaxy we'll probably be able to expand to other galaxies what what is wrong with that kind of train of logic where you know if there's like 0.5 percent growth that't imply a lock-in or it'd be weird if that implied lock-in.
I think we might want to give a little bit more context here or something. And so one of the key arguments of the most important century is it's just part, it's just one of the arguments that we live in a strange time.
But I'm arguing the current level of economic growth just looks too high to go on for another 10,000 years or so. And one of the points I make, which is a point I got from Robin Hanson, it's not something I own originally, is that if you just take the current level of economic growth and extrapolate it out 10,000 years, you end up having to conclude that we would need multiple stuff that is worth as much as the whole world economy is today, multiple times that per atom in the galaxy.
And if you believe we can't break the speed of light, then we can't get further than that. We can't get outside the galaxy.
So in some sense, we like run out of material. And so you're saying, well, okay, but what if the growth rate falls to 0.5%? And then I'm kind of like, okay, well, so the growth rate now, I ballparked it in the post is around 2%.
That's the growth rate generally in the most developed countries. Let's say it falls to 0.5%.
And then I'm like, okay, so the growth rate falls to 0 0.5 just like for how long for did you calculate how long it would take to get to the same place um yeah i think it was like 25 000 years a 0.5 gets to like one world's as um yeah economy per it's 10 000 versus 25 000 but 25 000 is the amount of uh light years between us and like the next galaxy that doesn't sound right i don't think this galaxy calculation was very close there's also going to be a bunch of dead space like as you get to the outer reach of the galaxy there's not going to be as much there that doesn't sound super right but let's let's just roll with it i mean sure sure let's just let's just say that let's just say that you had two percent today and then growth went down to 0.5 and just like stayed there forever yeah i'm pretty sure that's still too big i'm pretty sure you're still going to hit limits in some reasonable period of time. But also that would still be just like weird on its own.
It would just be like, well, we lived in the 200 year period when we had 2% growth, and then we had 0.5% growth forever. That would still make this like kind of an interesting time would be like the most dynamic fastest changing time in all of human history, not by a ton.
But it's also like you pick the number that's like the closest and the most perfectly optimized here. So, you know, if it went down to 0.1% or even down to 0.01%, then it would take longer to run out of stuff, but it would be even stranger, the 2% versus the 0.01%.
So I don't really think there's any way out of like, gosh, this looks like it's probably gonna end up looking like a very special time or a very weird time. This is probably not getting hung up on, but from that perspective, then the century in which we had 8% growth because of the Industrial Revolution, you would say that maybe that's the most important century.
Oh, sure. Yeah, yeah, yeah.
No, totally. No, the thing about the rapid growth is not supposed to be on its own an argument that this century is the special one.
I mean, by growth standards, this century looks less special than like the last one or two. It's saying this century is one of a handful, or I think when I say like one of 80 of the most significant centuries or something by economic growth standards, and that's only one argument.
And then I look at a lot of other ways in which this century looks unusual. But just to be clear, I mean, to say that something is the most important century of all time sounds totally nuts because there's so many centuries.
There's just so many centuries in the history of humanity even, and especially if you want to think about it on galactic timescales. And even once you narrow it down to 80, it's just way less weird.
It's just like, well, if I've already convinced you using kind of non-controversial reasoning that we're one of the 80 most important centuries, it shouldn't take me nearly as much further evidence to say, actually, this one might be number one out of 80, because your starting odds are more than 1%. So to get you up to 10% or 20% or 30%, it doesn't necessarily require a massive update the way that it would if we're just starting from nowhere.
But I guess I'm still not convinced that just because this is a weird century, that has any implications for why or whether we should see transformative AI this century. So if we have a model about when transformative AI happens, and if one of the variables that goes into that is what is the growth rate in 0 AD and what is the growth rate in 2080, it just feels weird to have as a parameter in when this specific technological development is going to happen.
It's just one argument in the series. It just kind of – I think the way that I would come at it is I would just say, hey, look at AI systems.

Look at what they're doing, look at how fast the rate of progress is, look at these like five different angles on imagining when AI might be able to do all the things humans do to advance science and technology. And just imagine that we get there this century, wouldn't it be crazy to have AI that could do all the things humans do to advance science and technology? Wouldn't that lead to just a lot of crazy stuff happening? Like there's only ever been one species in the history of the universe that we know of that can do the kinds of things humans do.
Wouldn't it be weird if there were two? That would be crazy. And one of them was a new one we built that could be like copied at will, run at different speeds, run on any hardware you have.
That would be crazy. And then you might come back and say, yeah, that would be crazy.
This is too crazy. I'm ruling this out because this is too crazy.
And then I would say, okay, well, we have a bunch of evidence that we live in an unusual, crazy time. And you actually should think that there's a lot of signs that this century is not just a random century picked from a sample of millions of centuries.
So that's the basic structure of the argument.

As far as the growth rate in zero AD, I mean, I think it matters.

I think you're asking the question,

why do the dynamics of growth in zero AD matter at all for this argument?

And I think it's because it just, it's a question of like,

how does economic growth work generally and how has it worked?

And what is the trend that we're on and what happens if that trend continues?

So if around zero AD growth was very low, but accelerating, and if that was also true at 100 AD and 1000 AD and negative 1000, or, you know, 1000 BC, however you want to put it, if all those things are true, then it starts to point to a general pattern, the growth is accelerating, and maybe accelerating for a particular reason, and therefore you might expect more acceleration. Gotcha, I gotcha.
Okay, so let's just talk about transformative AI then. Yeah.
Can you describe what success looks like concretely? Are humans part of the post transformative AI world? Are we hoping that these AIs become enslaved gods that help us create a utopia? What is the concrete success scenario look like? I mean, I think we've talked a lot about the difficulty of predicting the future. And I think I do want to emphasize that I really do believe in that.
So my attitude to the most important century is not at all, hey, I know exactly what's going to happen, and I'm making a plan to get us through it. It's much more like there's a general fuzzy outline of a big thing that might be approaching us.
There's maybe like two or three things we can come up with that seem good to do. Everything else we think about, we're not going to know if it's good to do or bad to do.
And so I'm just trying to find the things that are good to do so that I can make things go a little bit better or help things go a little bit better. That is my general attitude.
So it's, you know, it's, I don't know, it's like if you were on a ship in a storm and you saw some like very large fuzzy object obscured by the clouds, you might want to steer away from it. You might not want to say, well, what I think that is, is it's an island.
And I think there's probably, you know, a tiger on it. And if we go and train the tiger in the right way, blah, blah, blah, blah, blah, you don't want to get into that.
Right. So that, that is the general attitude I'm taking.
So what does success look like to me? I mean, success could look like a lot of things, but one thing success would look like to me would frankly just be that we get something not too different from the trajectory we're already on. So in other words, if we could have AI systems that behaved as intended, acted as tools and amplifiers of humans, did the things they're supposed to do.
And if we could avoid a world where those AI systems got sort of like, I don't know, all controlled by one government or one person, avoid a world where that caused a huge concentration of power. If we could just have a world where AI systems are just another technology, they help us do a lot of stuff, we invent lots of other technologies, and everything is like relatively broadly distributed, and everything works roughly as it's supposed to work, then you might be in a world where we continue the trend we've seen over the last couple hundred years, which is that we're all getting richer.
We're all getting more tools. We all hopefully get increasing ability to kind of understand ourselves, study ourselves, understand what makes us happy, what makes us thrive.
And hopefully the world just gets better over time. And we have more and more new ideas.
The ideas make us hopefully wiser. And, you know, I do think that in most respects, the world of today is just like a heck of a lot better than the world of 200 years ago.
I don't think the only reason for that is wealth and technology, but I think they played a role. And I think that like, yeah, if you'd gone back to 200 years ago and said, Holden, you know, how would you like the world to develop a bunch of new technologies as long as they're like sort of evenly distributed and they behave roughly as intended and people mostly just get richer and discover new stuff, I'd be like, that sounds great.
I don't know exactly where we're going to land. I can't predict in advance whether we're going to decide that we want to treat our technologies as having their own rights.
That's stuff that the world will figure out. But I'd like to avoid massive disasters that are identifiable because I think if we can, we might end up in a world where the future is wiser than we are and is able to do better things okay um the way you put it with ai enabling humans that doesn't sound like something that could last for thousands of years it almost sounds as sure like chim saying you know what we would like us as humans to be our tools at best maybe they could hope we would give them nice zoos but like what is the role for humans in this this future? I mean, a world I could easily imagine, although that doesn't mean it's realistic at all, is a world where we do build these AI systems.
They do what they're supposed to do. And we kind of use them to gain more intelligence and wisdom.
I've talked a little bit about this hypothetical idea of digital people. Maybe we develop something like that.
And then, you know, after 100 years of this, we've been around and people have been having discussions in the public sphere. And people kind of start to talk about whether the AIs themselves do have rights of their own and should be sharing the world with us.
And then maybe they do get rights. And maybe, you know, maybe some AI systems end up voting, or maybe we decide they shouldn't and they don't.
And either way, you have this kind of world where there's a bunch of different beings that all have rights and interests that matter, and they vote on how to set up the world so that we can all hopefully thrive and have a good time. We have less and less material scarcity.
So fewer and fewer trade offs need to be made. That would be great.
I don't know exactly where it ends or what it looks like. But that does I don't know.
I mean, what does anything strike you as like as as as unimaginable about that? Yeah, The fact that you can have beings that can be copied at all, but also there's some method of voting that. Oh, yeah.
Yeah. So, yeah, that's a problem that would have to be solved.
I mean, we have a lot of today we have a lot of attention paid to, you know, how voting system works, who gets to vote and how we avoid things being unfair. And yeah, I mean, it's definitely true that if we had if we decided there was some kind of digital entity that it should have the right to vote, and that digital entity was able to copy itself, well, you could get some havoc right there.
So you'd want to come up with some system that maybe restricts how many copies you can make of yourself or restricts how many of those copies can vote. These are problems that I'm hoping can be handled in a way that while not perfect, could be non-catastrophic by a society that hasn't been derailed by some huge concentration of power or misaligned AI systems.
So that sounds like that might take time, but let's say you didn't have time. So let's say you get a call and somebody says, hold on, next month my company is deploying a model that might plausibly lead to KGI.
What does open philanthropy do? What do you do? Well, I need to distinguish. I mean, you may not have time to avoid some of these like catastrophes, like a huge concentration of power or AI systems that don't behave as intended and have their own goals.
If you can prevent those catastrophes from happening, you might then get more time after you build the AIs to have these tools that help us, you know, help us invent new technologies and help us perhaps figure things out better and ask better questions. And then you could have a lot of time or you could figure out a lot in a little time if you had those things.
But if someone said, how long did you give me? A month. A month.
Let's say three months. So it's a little bit more.
Yeah, I would I would find that extremely scary. And I think I would I think that would do I kind of I kind of feel like that's one of the worlds in which I might not even be able to offer an enormous amount.
I think, so my job is in philanthropy, and a lot of what philanthropists do historically or have done well historically is we help fields grow. We help do things that operate on very long timescales.
So an example of something Open Philanthropy does a lot of right now is we fund people who do research on AI alignment, and we fund people who are thinking about what it would look like to get through the most important century successfully. And a lot of these people right now are very early in their career and just figuring stuff out.
And so a lot of the world I picture is like, it's 10 years from now, it's 20 years from now, it's 50 years from now. And there's this whole field of expertise that got support when traditional institutions wouldn't support it.
And that was because of us. And then you come to me and you say, we've got one week left.
What do we do? And I'm like, I don't know. We did what we could do.
You know, like go back in time and like try to prepare for this better. So that would be a lot of my answer.
I mean, I could say more specific things about what I'd say in the one to three month timeframe, but a lot of it would be like flailing around and freaking out, frankly. Gotcha.
Okay. So maybe we can reverse the question.
And let's say you found out that AI is going to take much longer than you thought. And you have like more than five decades.
Yeah. What changes? What are you able to do that you might not otherwise be able to do? Yeah.
I mean, I think the further out things are, the more I think it's valid to say that humans have trouble making predictions on long timeframes. And the more I'm interested in focusing more on other causes or on very broad things we do, such as trying to grow the set of people who thinks about issues like this, rather than trying to specifically study, for example, how to get AI systems like today's to behave as intended.
So I think that's a general shift. But I but I would say that, you know, yeah, I tend to feel a bit more optimistic on longer timeframes because I do think that the world just isn't ready for this and isn't thinking seriously about this.
And a lot of what we're trying to do at Open Philanthropy is create support that doesn't exist in traditional institutions for people to think about these topics. And, you know, that includes doing the AI alignment research that also includes like, thinking through how we want politically what regulations we might want to prevent disaster.
I think those are a lot of the things. So it's kind of a spectrum, I would say, you know, if it's in, if it's in three months, I would probably be trying to hammer out a reasonable test of whether we can demonstrate that the AI system is either safe or dangerous.
And if we can demonstrate it's dangerous, use that demonstration to really advocate for a broad slowing of AI research to buy more time to figure out how to make it less dangerous. But I don't know that I feel that much optimism.
If AI is like five, you know, this kind of AI is 500 years off, then I'm kind of inclined to just ignore it and just try and make the world better and more robust and wiser. But I think if we've got 10 years, 20 years, 50 years, 80 years, something in that range, I think that is kind of the place where supporting early careers and supporting people who are going to spend their lives thinking about this, then we flash forward to this crucial time and there's a lot more people who spent their lives thinking about it.
And I think that would be big deal let's talk about the question of whether we can expect ais to be smart enough to disempower humanity but you know dumb enough to have that kind of goal but you know when i look out at smart people in the world it seems like a lot of them have very complex nuanced goals that they've thought a lot about what is good and how to do good a lot of them don't yeah okay Yeah. Okay.
Yeah. But does that overall make you more optimistic about AIs? I am not that comforted by that.
I, you know, I pretty much, this is a very, very old debate in the world of AI alignment. Eliezer Yudkowsky has something called the orthogonality thesis.
I don't remember exactly what it says, but it's something like, you could be very intelligent about any goal. You could have the stupidest goal and be very intelligent about how to how to get it.
And in many ways, a lot of human goals are pretty silly, like a lot of the things that make me happy are not things that are, you know, profound or wonderful. They're just things that happen to make me happy.
And you could very intelligently try to get those things. So it doesn't give me a lot of comfort.
I think basically my picture of how modern AI works is that you're basically training

these systems by trial and error.

And so you're basically taking an AI system, you're encouraging some behaviors, discouraging

other behaviors, and you might end up with a system that's like being encouraged to pursue

something that you didn't mean to encourage.

And it does it very intelligently.

And I don't see any contradiction there.

I think that just because you, you know, if you were to design an AI system and you were kind of giving it encouragement every time it was, you know, getting more money into your bank account. If you were encouraging it to do that, you might get something that's very, very good at getting money into your bank account to the point where we're going to disrupt the whole world to do that.
And you will not automatically get something that thinks, gosh, is this a good thing to do? And I think a lot of, you know, a lot of human goals are just like, there's not really a right answer about whether our goals actually make sense. They're just the goals we have.
You've written elsewhere about how moral progress is something that's real, that's historically happened, and it corresponds to like what actually counts as moral progress. I wonder, do you think there's reason to think the same thing might happen with AI? Like whatever the process, whatever the process is that creates is moral progress.
I kind of don't in particular. So I've kind of used the term moral progress as a as just a term to refer to changes in morality that are good.
I think there has been moral progress. But I don't think that means moral progress is something inevitable or something that happens every time you are intelligent or something like that.
So I just, you know, an example I use a lot is just like attitude towards homosexuality. It's a lot more accepted today than it used to be.
I call that moral progress because I think it's good. And some people will say, well, you know, I don't believe that morality is objectively good or bad.
I don't believe there is any such thing as moral progress. I just think things change randomly.
And that will often be an example I'll pull out and I'll say, but do you think that was a neutral change? I just think it was good. I think it was good, but that's not because I believe there's some underlying objective reality.
It's just my way of tagging or using language to talk about moral changes that seem like they were positive to me. I don't particularly expect that an AI system would have the same sort of, would have the same evolution that I've had in reflecting on morality or would come to the same conclusions I've come to or would come up with moralities that seem good to me.
I don't have any reason to think any of that. I do think historically there have been some cases of moral progress.
All right. What do you think is the explanation for that historical progress? Well, I mean, one thing that I would say is that humans have a lot in common with each other.
So I think some of history contains cases of humans just kind of learning more about the world,

learning more about themselves, debating each other. Maybe I think a lot of moral progress has just come from humans, like getting to know other humans who they previously were stereotyping

and judging negatively and afraid of. So I think there's some way in which humans learning about

the world and learning about themselves leads them to have kind of conclusions that are more

or judging negatively and afraid of. So I think there's some way in which humans learning about the world and learning about themselves leads them to have kind of conclusions that are more reflective and more intelligent for their own goals.
But if you brought in something into the picture that was not a human at all, it might be very intelligent and reflective about its goals, and those goals might have zero value from our point of view. Recent developments in AI have made many people think that AI could happen much sooner than they otherwise thought.
Have the release of these new models impacted your timelines? Yeah, I definitely think that recent developments in AI have made me a bit more freaked out. And even since I wrote the Most Important Century series and before that, I mean, there were years when open philanthropy was very interested in AI risk, but it's become more so as we've seen the progress in AI.
So I think, you know, what we're seeing is we're seeing these very simple systems. It's kind of the same type of system over and over again, very general, and is able to do a lot of different tasks.
One of the most interesting, so I think people are interested in this. I think you should maybe Google GPT-3.
There's a lot of compilations of what this very simple language model that, by the way, my wife and brother-in-law both worked on, so that's a disclosure, but this very simple language model that just predicts the next word it's going to see in a stream of text. People have gotten it to tell stories.
People have gotten similar, though not identical models, to analyze and explain jokes. People have gotten it to play kind of role-playing games, write poetry, write lyrics, answer multiple choice questions, answer trivia questions.
And, you know, one of the results that I found most kind of ridiculous and strange and weird was this thing called Minerva, where people took one of these language models and with very little intervention, very little special intervention, they got it to do these like difficult math problems and explain its reasoning and get them right about half the time. And this is just not like it wasn't really trained in a way that was very specialized for these math problems.
And so I think that's like we just see AI systems kind of having all these unpredictable human like abilities just from having this very simple training procedure. And that is something I find kind of wild and kind of scary.
I don't know exactly where it's going or how fast. So if you think transformative AI might happen this entry, what implications does that have for the traditional global health and well-being stuff that Open Philanthropy does? I mean, will that have persistent effects of AI, if it gets aligned, will create a utopia for us anyway? I mean, I don't know about utopia.
I mean, my general take is that anything could happen. And I think my general take on this most important century stuff, the reason it's so important is because it's easy to imagine a world that is really awesome and is free from scarcity and that we end up in a really, you know, we see more of the progress we've seen over the last 200 years and end up in a really great place.
It's also easy to imagine like a horrible dystopia. But I do think it's, you know, my take is that the more likely you think all of this is, the more likely you think transformative AI is, the more you should think that that should be the top priority, that we should be trying to make that go well, instead of trying to solve more direct problems that are more short term.
I'm not an extremist on this. So, you know, Open Philanthropy does both.
Open Philanthropy works on speculative far off future risks. And Open Phil also does a bunch of, you know, more direct work.
You know, again, we do direct and recommend a lot of money to give us top charities, which do things like distributing bed nets in Africa to help prevent malaria, treating children for intestinal parasites. You know, Open Philanthropy does a lot of advocacy for more money going to foreign aid or for better land use policies to, you know, to have a stronger economy.
We do a bunch of like scientific research work that is more aimed at just like direct medical applications, especially in poor countries. So I support all that stuff.
I'm glad we're doing it. And it's just a matter of like how real and how imminent do you think this transformative AI stuff is? The more real and more imminent, the more relatively of our resources should go into it.
Yeah, no, that makes sense to me. But I'm curious, whatever work you do elsewhere, does that still have persistent effects after transformative AI comes? Or do you think that does well wash out in comparison to the really big stuff? I mean, I think in some sense the effects are permanent in that if you cause someone to live a healthier, better life, that's a thing that happened.
And nothing will ever erase that life or make that life unimportant. But I think in terms of the effects on the future, I do expect it mostly to wash out.
I expect mostly whatever we do to make the world better in that way, it will not persist in any kind of systematic, predictable way past these kind of crazy changes. And I think that's probably how things look pre and post-industrial revolution.
There's probably some exceptions, but that's my guess. You've expressed skepticism towards the competition frame around AI, or you try to make capabilities go faster for the countries or companies you favor most.
But elsewhere, you've used the innovation as mining metaphor, and maybe you can explain that when you're giving the answer. But it seems like this frame should imply that actually the second most powerful AI company is probably right on the heels of the first most powerful.
And then so just actually, if you think the first most powerful is going to take safety more seriously, you should try to boost them. How do you think about how these two different frames interact? I think it's common for people who become convinced that AI could be really important to just jump straight to, well, I want to make sure that people I trust build it first.
And that could mean my country, that could mean my friends, people I'm investing in. And I have generally called that the competition frame, want to win a competition to develop AI.
And I've contrasted it with a frame that I also think is important, which is the caution frame, which is that we need to all work together to be careful to not build something that spins out of control and has all these properties and behaves in all these ways we didn't intend. I do think we're likely, if we do develop these very powerful AI systems, I do think we're likely to end up in a world where there's multiple players trying to develop it and they're all hot on each other's heels.
And I am very interested in ways to find ways for us all to work together to avoid disaster as we're doing that. And I am maybe less excited than the average person who first learns about this is about picking the one I like best and helping them race ahead.
Although I am somewhat interested in both. But if you take the innovation as mining metaphor seriously, doesn't that imply that actually the competition is really a big factor here? Oh, so the innovation mining metaphor is from another bit of cold takes.
It's an argument I make that you should think of ideas as being somewhat like natural resources in the sense of once someone discovers a scientific hypothesis or even once, you know, once someone writes a certain great symphony, that's something that can only be done once. That's an innovation that can only be done once.
And so it gets harder and harder over time to have revolutionary ideas because the most revolutionary, easiest to find ideas have already been found. So there's an analogy to mining.
I don't think it applies super importantly to the AI thing, because all I'm saying is that success by person one makes success by person two harder. I'm not saying that it has no impact or that it doesn't speed things up.
So just to use a literal mining metaphor, let's say there's like a bunch of gold in the ground. It is true that if you rush and go get all that gold, it'll be harder for me to now come in and find a bunch of gold.
That is true. What's not true is that it doesn't matter if you do it.
I mean,

you might do it a lot faster than me. You might do it a lot ahead of me.
I totally think that

happens. Yeah.
So maybe one piece of skepticism that somebody could have about transformative AI

is that all this is going to be bottlenecked by the non-automatable steps in the innovation

sequence. So there won't be these feedback loops that speed up.
Well, what is your reaction?

Yeah, I think the single best criticism and my biggest point of skepticism on this most important century stuff is the idea that you could build an AI system that's very impressive, that could do pretty much everything humans can do, but there might be like one step that you still have to have humans do, and that could bottleneck everything. And then you could have the world not speed up that much and science and technology not advance that fast because they, as you're doing almost everything, but humans are still like slowing down this one step or the real world is slowing down one step.
So you have to do, you know, let's say real world experiments to invent new technologies and they just take how long they take. Um, I think this is the best, the best objection to this whole thing.
And the one that I'd most like to look into more, but I ultimately think that like, there's enough, there's enough reason to think that if you had AI systems that had kind of like human, like reasoning and analysis capabilities, I think you shouldn't count on this kind of bottleneck causing everything to go really slow. And a lot of that, I mean, I write about that in this, in thisak Point in the Most Important Century, Full Automation.
Part of this is just like you don't need to automate the entire economy to get this crazy growth loop. You can automate just a part of it that specifically parts that have to do with like very important tech like energy and AI itself.
And those actually seem in many ways just like less bottlenecked than a lot of other parts of the economy.

So you could have you could be like developing better algorithms and AI chips, manufacturing them, mostly using robots, using those to come up with even better designs.

And then you could also be designing like more and more efficient solar panels, using those to collect more and more energy to power your AIs. So you like a lot of the crucial pieces here just actually don't seem all that likely to be bottlenecked.

And then also, I mean, at the point where you have something that has the ability to have kind of like creative new scientific hypotheses the way human does, which is a debate over whether we should ever expect that and when. But once you have that, I think you should figure there's just a lot of ways to get out around all your other bottlenecks because you have this potentially massive population of thinkers looking for them.
And so an example is, you know, you could, with enough firepower, enough energy, enough AI, enough analysis, you could probably find a way to simulate a lot of the experiments you need to run, for example. Gotcha.
Now, it seems like the specific examples you used of energy and AI innovations, it seems like those are probably the hardest things to automate, given the fact that those are the ones that humanity has only gotten around to advancing most recently. Can you talk through the intuition that those might be easier? I think some of the stuff that might be hardest to automate would just be stuff that in some sense doesn't have anything to do with software or capabilities.
So an example of something that might just be like extremely hard to automate might be like trust, like, like, you know, making, making a business deal or providing care for someone who's sick. It might just be that like, even if an AI system has all the same intellectual capabilities as a human, you know, can write poetry just as well, can have just as many ideas, can have just as good a conversation.
It just, it doesn't look like a human. So people don't want that.
Maybe you can create a perfect representation of a human on a screen, but it's still on a screen. And in general, I see the progress in AI as being mostly on the software front, not the hardware front.
So AIs are able to do a lot of incredible things with language, things with math, things with board games.

I wouldn't be surprised if they could write hit music in the next decade or two, but they,

people really are not making the same kind of progress on like robotics.

Like so, so weirdly a task that might be among the hardest,

harder ones to automate, or especially things go fast.

It might be hard to automate the task of like taking this bottle of water

and taking off the cap because I have this, like, you know, I have this hand that is just like well designed for that. Well, it's clearly not designed for that, but it's these like these hands that can do a lot of stuff and we aren't seeing the same kind of progress there.
So I think, I think there are a lot of places where like AI systems might have kind of, their brains can do roughly everything human brains can, but there's some other reason they can't do some key economic tasks. And I think these are not the tasks I see likely to bottleneck the R&D as much.
Gotcha. This is an argument I make in one of my more obscure Cold Takes posts, but I say like, you know, AI that could actually take everyone's job, like every human's job, might be like a lot harder than AI that could radically transform the galaxy via new technology.
Because it might be, in some ways, like it might be easier to take a scientist's job than like a teacher's job might be like a lot harder than AI that could radically transform the galaxy via new technology. Because it might be in some ways, like it might be easier to take a scientist job than like a teacher's job or a doctor's job, because the teachers and the doctors are regulated.
And people might just say, I want human teachers. I don't want an AI teacher.
Whereas you can sit there in your lab with your AI scientists and find new theories that change the world. So some of this stuff I think is very counterintuitive.
But I can imagine worlds where you get, you know stuff before you get self-driving cars out on the road just because of the way the regulation is working. Gotcha.
Okay. Let's talk about another weak point, or the one that you identified as a weak point, lock-in.
What do you think are the odds of lock-in given transformative AI? So lock-in is my term, or I don't know if it's my term, but it's a term I use to talk about the possibility that we could end up with a very stable civilization. And so I talk about that as it's another post, it's called weak point in the most important century lock in I wrote posts about the weakest points in the series.
And the idea is basically like, throughout history so far, let's say someone becomes in charge of a government, and they're very powerful, and they're very bad. Usually, this is like, generally considered to to be temporary in at least some sense, like the kind of thing that's not going to go on forever.
There's a lot of reasons the world is just dynamic and the ways the world is tend to just like not stay that way completely. The world just has changed a lot throughout history.
It's kind of a dumb thing to say, but I'll get to why this might be important. So, you know, if someone is running a country in a really cruel, corrupt way, I mean, for one thing, at some point, they're going to get old and die.
And someone else is going to take over and that person will probably be different from them. For another thing, the world is changing all the time.
There's new technologies, new things are possible, there's new ideas. And so, you know, the most powerful country today might not be the most powerful tomorrow.
The people in power today might not be the ones in power tomorrow. And I think this gets us used to the idea that everything is temporary, everything changes.
And a point I make in the most important century series is that you can imagine a level of technological development where there just aren't new things to find, and there isn't a lot of new growth to have, and people aren't dying because we've, you know, for whatever reason, we've, you know, that seems like it should be medically possible for people not to age or die. And so you can imagine a lot of the sources of dynamism in the world actually going away if we had enough technology.
You could imagine a government that was able to actually surveil everyone, which is not something you can do now, with a dictator who actually doesn't age or die, who knows everything going on, who's able to respond to everything. And then you can imagine that world just being completely stable.
I think this is a very scary thought. And it's something we have to be mindful of that if the rate of technological progress speeds up a lot, we could quickly get to a world that doesn't have a lot more dynamism and is a lot more stable.
What do I think are the odds of this? I don't know. It's very hard to put a probability on it.

I think when you think about,

if you imagine that we're going to get this explosion

in scientific and technological advancement,

I think you have to take pretty seriously the idea

that that could end by hitting a wall,

that there could not be a lot of room for more dynamism

and that we could have these kind of very stable societies.

What does seriously mean?

I don't know, a quarter, a third, a half,

something like that.

I don't know.

I'm making up numbers.

I think it's serious enough to think about it

Thank you. these kind of very stable societies.
What does seriously mean? I don't know, a quarter, a third, a half, something like that. I don't know.
I'm making up numbers. I think it's serious enough to think about it and think about it as something that affects the stakes of what we're talking about.
Gotcha. So I'm curious if you're concerned about lock-in just from the perspective of locking in a negative future, or if you think that might intrinsically be bad to lock in any kind of future.
If you could right now press a button and lock in a reasonably positive future, but that won't have any dynamism, or one where dynamism is guaranteed but net expected positive is not, how would you make that determination? Well, I don't think a lot about what I would do with unrealistic buttons where I have crazy amounts of power that I'll never have and shouldn't have. You know, I think I think of lock in by default is like mostly a bad thing.
I mean, I feel like mostly we want to at least at least kind of preserve optionality and, you know, have a world where it's not just like one person running the show with their values set up the way they want forever. I think of it mostly that way.
I can imagine some future world where the civilization has been around long enough. We've learned what there is to learn and we know what a good world looks like.
And most people feel pretty confident about that and they're right to feel confident. And maybe lock-in is not so bad, but I mostly think of lock-in as a bad thing.
I also, I also imagine that you could sort of like lock in some things about the world in order to avoid locking in others. So I can imagine if you had this enormous amount of power over how the world works.
And again, this comes down to some of this is more explained in my digital people series, how this could work. But if you had this kind of world where you completely control the environment, you might want to lock in the fact that you never have one person with all the power.
That might be a thing you might want to lock in, and that prevents other kinds of lock in. Do Do you worry about AI alignment as being a form of lock-in? So in some sense, if the goal of the research is to prevent drift from human values, then you might just be locking in values that are suboptimal.
Yeah, I mostly think of AI alignment as just trying to avoid a really bad thing happening. So I mostly think of it like the thing we don't want to happen is we have some AI system, we thought we were designing it to help us, we're actually designing it to do something extremely random thing.
It's just like, again, these systems work by trial and error, by encouragement, discouragement, or positive and negative reinforcement. And so we might have not even noticed that through the pattern of reinforcement we were giving, we trained some system to want to put as much money as possible into one bank account or gain as much power as possible or control as much energy as possible or something like that, or just set its own reward number, its own score to the highest possible number.
I think that would be a form of lock-in if we had systems more powerful than humans that had these kind of random goals. That would be like locking in kind of a future that is not related to the things that humans value and care about.
That's an example of a thing I think would be really bad. Now, if we got these systems to behave as intended, we still might have problems because we might have humans doing really stupid things and locking in really bad futures.
I think that's an issue too. I feel reasonably comfortable, though not a hundred percent confident saying that we'd like to avoid that just like slip up.
We'd like to avoid having these systems that have these random goals we gave them by accident, and they're very powerful, and they're better at kind of like setting up the world than we are. And so we get this world that's just like doing this random thing that we did by accident.
I think that's a thing worth avoiding. What is your biggest disagreement with Will McCaskill's new book on long-termism? I like Will's book.
I mean, I think it's worth reading. Will McCaskill's book about how the future could be very large and very important.
I think he presents a picture, like, in my opinion, if you want to talk about the long-run future and how to make the long-run future go well, you're starting from a place of, like, by default, almost nothing I can do will actually make sense. Like, I do really believe it's hard to understand the long run future.
And it's hard to make specific plans about it. So I would say compared to Will, I am very picky about which issues I think are like big enough and serious enough to actually pay attention to.
And so I feel the issue of AI, AI would be transformative enough, It looks likely enough that it'll be soon. If it's soon, that means there might actually be things we can do that have predictable effects.
I think this misaligned AI thing is a real threat. The way people design AI systems today would be really bad.
I am ready to like put some resources into preventing it, but like that's kind of, you know, that's crossing my threshold. Most things don't.
And so if you make a list of ways to make the next million years go well, I'll look at most of them and be like, I don't know, I don't really believe in this, I wouldn't really invest in this. So I think Will is a bit broader than I am, in a sense, he's interested in more things.
And I am pickier than he is, because I think it's so hard to know what's really going to affect the long run future that I'm just looking for like a really short list of things that are worth paying paying special attention to. Gotcha, gotcha.
Is there a specific thing that he points out in the book that you think would be hard to grapple with? I don't remember super well. I mean, the book is a really broad survey of lots of stuff.
And I think, you know, an example I might give is he talks about the risk of stagnation, for example. So the risk that growth growth might just stop or something or growth might slow to very low levels and then that kind of implies that what we should be trying to do is make sure we continue to innovate and continue to

have growth but then there's like other parts of the book that kind of make it sound like we

shouldn't move too fast and we shouldn't innovate too much because we don't want to get to our

future before we've like kind of achieved some sort of civilizational maturity beyond what we

have now to decide what we want that future to look like. Like we don't want to build these powers before we like have a better idea of what to do with them.
So I think these are examples of things where I'm just like, gosh, I don't know. Like it could be, it could be good to have more growth.
It could be bad, good to have less growth. It could be that stagnation is a big threat.
It could be that like building powerful technologies too fast is a big threat. Like.
I just don't really know. I'll tell you what I'm thinking about.
I'm thinking about AI because I think it's a big enough deal and likely enough and that we've got enough traction on some of the major risks. Right, right.
When I look throughout history, it often seems like people who are predicting long-term trends are too pessimistic. So in the 70s, you might have been too pessimistic about the ability to find more oil or feed a growing population.
And because they couldn't have predicted the technological breakthroughs that might have made these things possible. Does this inform some sort of vague optimism about the future for you with regards to AI or no? Yeah, I think historically people have been overly pessimistic about future technologies.

And I think by default, the picture with AI looks just like really scary. It just looks like it would be really easy to get a bad future in a lot of different ways if we just didn't move cautiously.
So these two considerations kind of balance each other out a little bit for me. I know a lot of people who believe that we're in just deep, deep, enormous trouble, that this outcome where you get AI with its own goals,

wiping humanity off the map is just like, almost surely going to happen. I don't believe that.
And this is part of the reason I don't believe it. It's just like, I actually think this situation looks very challenging, very scary by default.
And I think we're tending to overestimate how bad and how dire things are. And so they balance out a little bit for me.
Okay, gotcha. But in many of these cases, it seems like it would be impossible to see the positive scenario come about.
Like, for example, if you were forecasting population in the 70s, is there some reasonable method by which you would have predicted this is not going to lead to some massive famine that kills a billion people? Or would that have been your focus in the 70s if the open fill was a thing back then? I think it's really hard to know how knowable the future was in the past and what that means for today. I do think that when you look back, and again, this is something I've looked at a fair amount, when you look back at people trying to predict the future in the past, it just looks deeply unserious.
You could say that future people will say the same about us. I'm sure they'll think we look less serious than they are, but I think there's a difference.
I really do think that attempts to kind of rigorously make predictions about the future, there haven't been a lot of them historically, and I don't think it's obvious that people were doing the best they can and that we can't do better today. So this population is, this population is an example.
Like I'm just, I'm just, it doesn't seem necessarily true to me that you couldn't have been at that time and said, gosh, you know, population has been going up for a while now and people keep inventing new ways to come up with more resources. Maybe that'll keep happening.
There's like really not convinced you couldn't have said that. I'm definitely not convinced no one did say it.
I think some people did say it. And so I think, yeah, I'm hesitant to get too defeatist just from the fact that some people were wrong about the future in the past.
I think it's hard to know if there was really no way to know or if they just weren't trying very hard. Okay.
Well, one thing you just said a minute ago was that we are better at making predictions that people in the past were. So that alone should make us more optimistic about really to predict the future.
That's just a guess. But I do think that there's been a lot of – there's been a lot of – I mean this is what society is, is we have a lot of progress on all kinds of intellectual fronts.
And I think there has been a lot of progress on like what it looks like to make good, reasonable predictions about the future. I think that's something that's happened.
So I think we should be expecting ourselves to do a bit better than people did in the past, and future people will probably do better than we do. Right.
But I often – I kind of wonder when I look at a report like Biological Anchors, whether Asimov's just shooting the shit about screens and what you're able to do with them, whether he might have had less sources of error than this eight step methodology, where you might not even be aware of there's a ninth or 10th missing step that might make the whole thing invalid. And where many of the inputs have multiple orders of magnitude, wide confidence intervals.
What do you think of that general like skepticism? I mean, biological anchors, I think is a very important input into my thinking. It's not the only input.
So I think it's, you know, I think my views on AI timelines are kind of a mix of just kind of a looking at AI systems today, looking at what they did 50 years ago, looking at what they did 10 years ago, and just kind of being like, well, gosh, it sure looks plausible that these will be able to do all the things humans can do to advance science and technology pretty soon. That's one put into my thinking.
Another input into my thinking is what we call the semi-informative priors analysis, which is, you know, it's a complex report because it looks from a lot of different angles. But I think you can summarize the highlights of the report as just saying most of the effort that has ever in the history of humanity gone into making AI will be in this century, ever.
Because the field of AI is not very old, and the economy and the amount of effort invested have gone up dramatically. And I think that's a data point in favor of not being too skeptical, that we could be on the cusp of transformative AI.
It's like, in some sense, the world has not been trying very hard for very long. So that's an input.
Another input is just like expert surveys. When people just ask AI researchers when they think AI will be able to do everything humans can, they tend to come out saying it's a few decades.
Now that can be biased and unreliable in all kinds of ways. All these things have their problems.
That's a data point. And then there's biological anchors.
And biological anchors is a report that kind of – I would summarize it to summarize it on a podcast. It's a very complex report.
There's a lot of different angles it uses. There's a lot of different questions it asks.
There's a lot of different numbers. But I think you can boil it down to some fairly simple observations.
You can say that in some important sense, which could be debated and analyzed, but seems true most ways you look at it, in some important sense, we've never built AI systems before that do as much computation per second as like a human brain does. So it shouldn't be surprising that we don't have AI systems that can do everything humans do, because humans are actually doing more work in their brains than a normal AI system is doing.
However, it also looks like within this century, we probably will have AI systems that are that big. And if we estimate how much it would take to train them, how much it would cost to build them, that looks like it will probably be affordable this century.
And then you can just talk about all the different ways you could define this and all the different ways you could quantify that and all the different assumptions you could put in. But my bottom line is like almost any way you slice it, it looks pretty, whatever, however you want to think about it, however you want to define what it means for an AI brain to be as big as a human's and what it would mean to get that brain sort of trained, most angles on it suggest that it looks reasonably like it'll happen this century.
That's a data point for me. That matters.
So all those are data points feeding into my view. Okay, so I'm stealing this from Eliezer who asked on Twitter, has there literally ever in the entire history of ai been any case of anybody successfully calling the development timing of literally any novel ai capability he has very complex sentences yeah using a bio anchored or bio inspired calculation uh i saw some discussion of this on his facebook and i i think the answer might be yes but i mostly want to attack the premise i mean i mostly just want to say like there haven't been a lot of this on his Facebook, and I think the answer might be yes, but I mostly want to attack the premise.

I mean, I mostly just want to say, like, there haven't been a lot of cases of people predicting AI milestones with great precision, and that's also not what I'm trying to do. A lot of what I'm trying to say is I'm trying to say, gosh, this century, it looks like more likely than not that we'll get something hugely transformative.
that, you know, he's, he's asking about some history of AI. That's like,

you know, that's like a few decades old and there haven't even been a lot of people trying to make predictions. And a lot of the predictions have been way more narrow and specific than that.
So I'm mostly just like, this isn't a very important or informative question, you know, but if, and I think all the work he's doing, all the things, has there ever been an example of someone using the kind of reasoning Eliezer is using to predict the end of the world or something, which is what he's predicting. So I mostly just want to challenge the premise and just say, look, we're not working with a lot of sample size here.
It's not like this is some big, well-developed field where people have tried to do the exact same thing I'm trying to do 25 times and failed each time. This is like mostly people in academia try to advance AI systems.
They don't try to predict when AI systems can do much and do what? We're not working with a lot here. We have to do the best we can, make our best guess, use our common sense, use our judgment, use the angles we've got.
That's my main answer. But another answer I would give is I would just say that, I don't know, like, like Moravec, Hans Moravec was kind of the original biological anchorist person.
And he, you know, I think he predicted like artificial general intelligence, like around 2020 or something. And I'm kind of like, on Eliezer's own views, it's going to look like he was like, unbelievably close.
Like that was like, maybe, you know, Eliezer believes we'll see it by 2030. So if that's what you and I think that's plausible.
So if that's what you believe, then it's going to be, I don't know. I think we'll look back on that as like the greatest technology prediction ever by a lot.
Yeah, I think the answer is maybe. There was also some discussion in the comments about whether Morvec called like big progress on AI's doing well at vision by examining the retina.
There was some debate about that. I think it's all very muddy.
I don't think this is much of a knockdown argument against thinking about biological anchors. I mean, it is only one input into my thinking.
I do think it looks kind of good for biological anchors that we have seen this deep learning revolution, and we have seen these brute force AI training methods working really well when they didn't used to work well. And it is when AI systems started to be about the size of insect insect or small animal brains, or within range within a few orders of magnitude of human brains.
You can call that a wild coincidence, but it doesn't feel you know, probably these numbers are all off. They're probably all off by 10x 100x 1000x.
But I mean, we're talking about very important things and trying to get our best handle and our best guess. And I think biological anchors looks fine so far.
I don't know, it doesn't look amazing. It looks fine.
Now, as I'm sure aware, many people have proposed that increasing progress in science technology and increasing economic growth are the most compelling things to be doing instead of working on transformative AI. I just want to get your broad reaction to that first.
Sure. So I think we're kind of talking about, you know, maybe the progress studies crowd or, yeah, I wrote a piece about this on Coltix called Rowing, Steering, Anchoring, Equity and Mutiny, where I discuss these like different ways of thinking about what it means to make the world good.
And I do have some sympathy for the idea that, you know, a lot of the way the world has gotten better over the last couple hundred years is just we've gotten richer, we've had more technological capabilities. Maybe we should try and do more of that.
I don't think this is like a nutty thing to think. I think this is somewhat reasonable.
But I overall feel that we, you know, even a couple hundred years is not that big a percentage of the history of humanity. I wrote a series called Has Life Gotten Better that kind of asks what the whole graph of quality of life looks like over the course of humanity.
And there is precedent for like technological development seeming to make things worse. That's kind of what it looks like happened in the agricultural revolution.
So I have some sympathy for saying, hey, this thing has been good for 200 years. Let's do more of it.
But I don't think it's like the tightest, most conclusive argument in the world. And I think we do have some specific reasons to believe that developing some particular new technologies, not only AI, but also potentially like bioweapons, could just be like so catastrophically dangerous.
I think it's something like where you could think of us, I think Toby Ord uses the analogy of humanity being like a child who's becoming an adolescent. And it's like, it's great to become stronger up to a certain point.
That's fun. That feels good.
And then at a certain point, you're like strong enough to really hurt yourself and really get yourself in trouble or maybe strong enough that you don't know your own strength. I think there's a pretty good case that humanity is reaching that point.
I think we're reaching the point where we could have a nuclear war or a bioweapon or AI systems that really changed the world forever and so it might've made sense 300 years ago and we were all struggling to feed ourselves to say, hey, we want more power, we want more technology, we want more abilities. Today, I think we're starting to enter the gray area, we're starting to enter the gray zone, or maybe we should slow down a little bit, be a little bit more careful.
I'm not saying to literally slow down, but I'm talking about priorities. I would rather look at what I think are dramatically neglected issues that might affect all of humanity's future and at least do the best we can to have a handle on what we want to do about them, then put my effort into throwing more juice and more gas behind this ongoing technological progress, which I think is a good thing.
It's just a matter of priority. Okay.
Do you think that the entire vision of increasing progress is kind of doomed if ideas get harder to find? I do think there's eventually limits to how much, like, I think that the rate, like I've said, the atoms of the galaxy argument, but I think just a broader, you know, a broader common sense take would be that the world over the last couple of hundred years has changed incredibly dramatically. We've had just like new exciting technologies and capabilities every year.
And I think just a good guess would be that that hits a wall at some point. I think that just, you know, it might be the atoms of the galaxy.
It might just be something much more boring. I don't, it might just be like a gradually slowing, you know, is actually what we seem to observe when we look at the, when we look at the numbers is that we are seeing a bit of stagnation, a bit of slowing down and probably will keep slowing down by default.
So default. So yeah, I think probably a good guess is that we are, the world is changing at an incredible pace that has not been the case for most of history, and it probably won't be the case for the whole future.
Okay, gotcha. I guess there's several reactions somebody could have to the idea that ideas are getting harder to find, and therefore that this makes Parker Studies less relevant.
One is just if you look at your own blog, right? The entire thing is just you complaining about all this low-hanging fruit that people are not plucking. Nobody's thinking seriously about transformative AI.
Nobody's thinking seriously about utopia. Nobody's thinking seriously about ego governance.
How do you square this with the general thing about ideas getting harder to find? Oh, I think there's just a ton of really important stuff today that not enough people are paying attention to.

I just expect, I mean, that was also true 50 years ago. That was also true 100 years ago.
It was probably more true 50 years ago than it is today. It was probably more true 100 years ago than 50 years ago.
It's gradually the supply of amazingly important ideas that are not getting any attention. It's probably getting harder to do that.
Harder doesn't mean impossible. You know, I do actually think that if people want to do something that's really new and world changing and dramatic and revolutionary, I think maybe the worst way to do that is to kind of go into some well-established scientific field and try to revolutionize that.
I think it's better to just use your common sense and ask an important question about the world that no one's working on because it isn't a scientific field, because it isn't a field of academia, because it doesn't have institutions and work on that. And a lot of my blog does advocate that.
So, you know, for example, AI itself is a very well-established field, but AI alignment is a weird field that doesn't really have like academic departments right now. A lot of the stuff I'm talking about, trying to predict what the future is going to look like, it's kind of a weird low prestige thing that you can't easily explain to your extended family.
And I do think that's probably the best place to look if you want to do something that's going to be, you know, super significant, super revolutionary. That is why I've kind of professionally been drawn to it, looking for potential big wins that philanthropy could get.
But isn't another way to just read what you said, which is that we shouldn't follow and this with us with the greats to be a great person you know have great achievements yourself is another way to think about that that you should probably also ignore the advice that the obstacle advice you're giving or 80 000 hours gives because those specific things are that's not what's going to make you the next Einstein I mean I a fair number of your questions or like a fair number of like dialogue I see in in some of the like skeptical of futurism world it's, it feels to me like it's almost just getting unnecessarily fancy or something. I kind of just want to say who, who's someone who really revolutionized the way the world thinks about stuff.
Well, I don't know, like Darwin. Now, what was Darwin doing? Was Darwin saying like, well, you know, I really don't want to think about this thing because like, I don't believe humans are capable of thinking about that thing.
And I don't want to like, I don't want to think about, you know, this topic because like, I think it's too hard to know the future and blah, blah, blah. Was he, was he doing all that stuff or, or was he just, was he just asking an interesting question? You know, who was he, was he just kind of, was he just kind of saying, Hey, you know, this thing, this thing seems important.
I'm going to use my common sense and judgment to figure out how it works and I'm going to write about it. And I think some of this stuff gets too fancy.
So I think today what's going on is if I just look at the world and I say, what are the most important things that could matter for the world being a good or bad place in the future? I've looked at a lot of possibilities. I think that AI is one of the leading examples, and I don't see a lot of people paying attention to it.
And so that's what I want to work on. And I think a lot of the people who have done, you know, revolutionary work that we now look back on and a lot of people try to imitate, they weren't really doing all this stuff.

They weren't trying to imitate the stuff that usually works and stay away from the stuff that wasn't. They were just asking interesting, important questions and working on them.
And, you know, as far as myself in 80,000 hours, I mean, I just I guess I just don't feel that we're well known enough or influential enough that our advice, that that stuff we're interested in is obvious, is automatically, therefore not neglected. I think the stuff we're talking about is very neglected.
But if you find something that's even more neglected and more important, I mean, more power to you. Let's say the total amount of money given to EA just increased by an order of magnitude or something.
What could be possible at that point that's not possible now? I don't know. I think even then, I think the amount of money we'd be working with would be like really small by the standards of any kind of like government budget.
In general with philanthropy, I'm always looking for things where it's like, can we seed the creation of a field? Can we, you know, fund people to introduce new ideas? But, you know, still we're very small compared to the overall economy and the overall government. I think even multiplying everything by 10, that would still be true.
Um, not sure exactly what we do with 10 X as much money. I'm not even sure what we're going to do with, with the money that already exists.
Yeah. Um, but do you think there will be more yay billionaires in the coming future? I would guess so.
Um, does that imply you should be spending money faster now if you were, yeah, I mean, we're trying to model it, right? I mean, it's like we have, I mean, in theory, we have all these models that say, like, here's our guess at how much money is eventually going to be available. And here's our guess at how many giving opportunities will eventually be there to fund.
And, you know, this is our guess at, like, what's good enough to fund and what's not. And that's a very tentative guess.
And a lot of it is just, like, really, you know, really, really imprecise stuff. But we have to have some view on it.
Anyone who's spending money does. So, I mean, yeah, I do, I do tend to assume that, that Sam Baked and Freed and Dustin Moskovitz are not, and, and Cary Tuna are not the last billionaires who are interested in doing as much good as possible.
But it is really hard to model this stuff. And frankly, a lot of it, you know, we, we have, we have like various rough models we've made over the years.

We also just will sometimes use our intuition and just say we fund the stuff that seems like quite good and exciting and we don't fund stuff that doesn't. And that's an input into our thinking, too.
I got you. And then how do you think about the risk that some of you are giving might have negative impacts? So people have brought this up in the context of your $30 million investment in OpenAI.
but you know in of contexts, especially when you're talking about political advocacy, people might think that the thing you do has negative side effects that counteract the positive effects. How do you think? Is it just a straight EV calculation or how do you think about this? I mean, I think in theory, what we want is we want to make grants that have like more upside than downside or have expected net positive effects.
I think we tend to be just, you know, in a common sense way, a little bit conservative with the negative effects in that the thing we don't want to do is we don't want to like enter some field on a theory that's just like totally messed up and wrong in a way that we could have known if we just done a little bit more homework. I think that there's just like something kind of irresponsible and uncooperative about that.
And so in general, like when we are making big decisions, like big dollar decisions or going into a new cause, we often will try really hard to like do everything we can to understand the downsides. And if after we've done everything, you know, roughly everything we can up to some reasonable diminishing returns.
And once we've done that, you know, if we believe as far as we can tell the upsides outweigh the downsides, we're generally going to go for it. Our goal is not to avoid harm at all costs, although our goal is to kind of operate in a cooperative, high-integrity way, always doing our best, always trying to anticipate the downsides.
But recognize we're going to have unintended side effects sometimes, and that's life, and anything you do has unintended side effects. I don't agree with the specific example you gave as an example of something that was net negative, but I don't know.
Are you talking about OpenAI? Yeah. Okay.
Yeah. Many people on Twitter wanted me to ask if you read it up investing in OpenAI.
I mean, you can look up our $30 million grant to OpenAI. I think it was back in 2016.
And we wrote about some of the thinking behind it. We wrote that part of that grant part of that grant was getting a board seat for open philanthropy for a few years, that we could help with their governance at a crucial early time in their development.
You know, I think that, I think you could have, I mean, I think some people believe that open AI has been negative for the world because of the fact that they have contributed a lot to AI advancing and to AI being sort of hyped. And they think that gives us less time to prepare for it.
And I do think that all else equal AI advancing faster gives us less time to prepare and is a bad thing. But I don't think it's the only consideration.
I think OpenAI has done a number of good things too, has set some important precedents, I think is probably much more interested in a lot of the issues I'm talking about and risks from advanced AI than the company that I would guess would exist if they didn't, that would be doing similar things. I don't really accept that, that even, even the idea that open AI is, is a, you know, is a negative force.
I think it's like highly debatable. We could talk about it all day.
And then if you look at our specific grant, it's just, that's, that's even a completely different thing because, because a lot of that was not just about boosting them, but was about getting to be part of their early decision making. And I think that was something that there were benefits and was important.
And so, yeah, I mean, my overall view is that I look back on that grant as one of the better grants we've made, not one of the worst ones. Oh, wow, okay.
But certainly we've done a lot of things that have not worked out. And I think there are some times, surely when we've done things that have consequences we didn't intend, and, you know, no philanthropist can be free of that.
What we can try and do is be responsible, seriously do our homework to try to understand things beforehand, see the risks that we're able to see and think about how to minimize them. So let's talk about ethics.
I think you have a very interesting series of blog posts about future proof ethics. Sure.
Do you want to explain what this is first? Sure. I mean, I wrote a short blog post series just kind of trying to explain some of the philosophical views and ethical views that are common among people who call themselves effective altruists.
And so one of the ideas I appealed to is I'm like, what I think, and I'm not sure I'm getting this right, but what I think a lot of people I know are trying to do is they're trying to come up with a system of morality and a system of ethics that would survive a lot of moral progress or something that, you know, if they later became a lot wiser and learned a lot more and reflected on their morality, they wouldn't look back on their earlier actions and think they were doing horrible, monstrous mistakes. And a lot of history has just like people doing things they thought were fine and right at the time.
And now we look back and we're horrified. So you could think of yourself as like, what morality can I have that would make it not so likely that, you know, if there is a bunch more moral progress, and if people do learn a lot more, that the future won't look back on me and be horrified at what I did.
So I wrote, I wrote a bit of a series about what it might look like to try to do that, laid out a few principles of it, and kind of tried to use this to explain the moral systems a lot of effective altruists tend to use, which tends to be some flavor of utilitarianism, is often very expansive about whose rights count. So effective altruists are very interested in future generations that don't exist yet.
They're interested in animals being mistreated on factory farms. They're interested in like various populations that a lot of people don't care about today, but that there are large numbers of.
So I tried to kind of explain that. A thing that's important is I laid this view out partly so I could argue against it later, and I haven't done the latter yet.
So I have a lot of reservations too about the ethical systems that are common with effective altruism. Okay, okay.
So let's talk about some of the pillars you laid out in this piece so sentientism that seems pretty reasonable to me uh so so sentientism sentientism is the idea that um well so there's let me let me back up there's there's three principles that i kind of like roughly outline that you might want for a morality that you know that is going to stand up to scrutiny or that you won't yeah be so likely to change your mind about if you learn more and get better so one principle is just just systemization. It's like, it's better to have morality based on simple general principles that you apply everywhere that have a morality that's just always you just deciding what feels right in the moment.
Because the latter could be subject to a lot of the biases of your time, and the former lets you kind of stress test the core ideas. And then two of the core ideas I propose are what I call thin utilitarianism, which is basically the greatest good for the greatest number, and sentientism, which is basically saying that someone counts or someone matters if they're able to suffer or have pleasure.

And I think you just said sentientism seems reasonable to you. I think sentientism might be the weakest part of the picture to me.
I think you if you have a morality where you are insistent on saying that everyone counts equally in proportion to the amount of pain or pleasure they're able to have, you run into a lot of weird dilemmas that you wouldn't have to run into if you didn't have that view. So I think it's very strange, but I think it is actually one of the more questionable parts of the view.
It's kind of saying, when I'm deciding whether I care about someone, it doesn't matter at all if they're way in the future, if they're way far away, if they're totally different from me, if they're not human, if I've never met them, all that matters is if they can have pain or

pleasure. I think it sounds great.
And I completely get why someone listening to be like, how could

you ever disagree with that? But I do think I do think there's various challenges with it,

which I have not had the chance to write about yet. And I doubt I can be very convincing on this

podcast yet, because I haven't thought about it. All right.
Yeah, sounds good. Let's talk about the other two so yeah systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization

systemization systemization systemization systemization systemization which I have not had the chance to write about yet. And I doubt I can be very convincing on this podcast yet because I haven't thought about it.
All right. Yeah.
Sounds good. Let's talk about the other two.
So yeah. Systemization.
Yeah. Doesn't the fact that you have lots of, you know, complex and sometimes contradictory moral intuitions, does that suggest that maybe the whole goal of having some fundamental principles you extrapolate the rest of morality from that's a kind of doom project? I think it does somewhat suggest that.
I mean, i am somewhat partial to that view and that that's something i may be writing in my rebuttal but i also think it's it's possible to just be confused and i think it's it's possible to just have lots of stuff going on in your brain and some of it might be based on like really good really good intentions of treating other people fairly and being good to other people and some of it might be based on just like other weird stuff about like wanting to stand up for people who look like you or help people who look like you and things like that. And so I think it's I think it's I do have some sympathy for the project of trying to say, you know, my intuitions contradict each other, but some of them are coming from a good place.
Some of them are coming from a bad place. If I thought more about it, I would realize which ones are which.
And I want to try and do that and do that yeah yeah let's talk about the utilitarianism so there's this question from an old scott alexander post where he asks would you rather the medieval church spent all of its money helping the poor rather than supporting the arts so that maybe there were fewer poor people back in the medieval times but you wouldn't have any cathedrals or you wouldn't have the sistine chapel i don't know how you would answer that if you were in medieval times. It doesn't sound like the strongest version of this argument to me, to be honest.
Like, I think maybe that would be fine or good. I don't know.
Maybe if I was a bigger fan of, I mean, I don't know. My wife really loves these like old churches.
So maybe if I had more of her attitude, I would be more horrified by this idea. I think that, yeah, I mean, like low-income people had a rough time in the past, and having better lives seems pretty appealing.
So I don't really know if that's the best version of – I don't know if this is the best version of this argument. All right.
So how much of future-proof ethics is basically you're very confident that a future Holden will have a much more developed and better set of ethics? Or how much do you think people in general or humanity in general will get better ethics over time? Yeah, this has been definitely a point of confusion in this series, and partly just something I think I didn't communicate well about and makes the series not that amazing, is just that I use the term moral progress, and I just use it to refer to things getting better. I think sometimes there is such a thing as thinking more about your morality, gaining some insight, and ending up in a better place as a result.
I think that is a thing that is real. There are some people who believe morality is an objective truth, but I'm not one of those people.
But even though I believe morality is not objective, I still think there's a meaningful notion of moral progress. There's such a thing as having more reasonable moral views than I used to.
What I didn't mean to say is that moral progress has any inevitability about it. I didn't mean to say that moral progress necessarily happens just because time goes on.
I don't think that. I just think it's a thing that can happen.
So I do think a future Holden will probably be better at morality just because I'm really interested in the topic and I'm going to keep trying to improve it. And I think that, you know, we have some reason to think that actually just like does help a bit, a really tiny bit.
But I'm not confident in that at all. And I certainly don't think that society is going to have moral progress just necessarily.
But I do think we've had some in the past. OK, but then I don't know.
Then it seems weird to like label the system of ethics, future proof ethics, right? Maybe just be future Holden proof ethics. Yeah, possible.
I mean, I talk about this a bunch in the series and I think I just didn't do a great job with this um yeah i mean i think what i was trying to do is i was trying to use a term that you didn't have to be a moral realist to to get behind and what i was really trying to capture is like can i think now to reduce the odds that if later i improve that i'll be horrified by my early actions that was what i was trying to capture the concept of i'm not sure I really did it successfully. Gotcha.
Okay, so you had a recent post on the EA forum that I thought was really interesting. And a quote from that is, my view is that for the most part, people who identify as EAs tend to have unusually high integrity.
But my guess is that this is more despite utilitarianism than because of it. So what do you think is the explanation for this coincidence where a group of reasonable, non-fanatical, high-integrity people also happen to be a community of utilitarians? You might have a set of people who think of themselves as trying really hard to be the kind of person they should be, or really hard to bring their actions into line with their beliefs and their statements.
And so that drives them to be kind of like honest a lot of the time and follow a lot of our common sense rules of morality. But it also drives them to like really try to get that ethics right and land on ideas like utilitarianism that are very like systematic and pure and like give you sort of this like clear theoretical guidance.
So it could drive both those things. Whereas I believe that if you're a utilitarian, it's really unclear whether utilitarianism actually tells you to do things like avoiding lying.
Some people think it does. Some people think it doesn't.
I think it's very unclear. You've advocated for the moral parliaments approach when you're trying to make decisions.
What is the right level of organization at which to use that approach? Should individuals be making decisions based on having multiple different moral parties inside them? Is that the right approach for entire movements but individuals should be specializing? What is the right level to be applying this approach at? I mean, there's something I hope to write about in the future is this topic of moral uncertainty. And the basic idea is like there might be a bunch of different ways about thinking about what the right thing is to do in the world.
And one, you know, you might look at the world from one angle and say, well, what matters is like the total sum of all the pleasures. So therefore a bigger world would be better.
So therefore I should be like really obsessed with like getting the world to be as big as possible. There might be another perspective that says that really matters to suffering.
We should minimize suffering. We should want the world to be small.
There might be another perspective that says, you know, it doesn't matter what happens to the world, it matters how I act, what matters is that I act with integrity, that I tell the truth, things like that. And, you know, there's these interesting debates about what do you do when you think you have some sympathy for all these views, and then how do you choose an action that some perspectives would say is the best thing you've ever done, and some would say is the worst thing you've ever done.
And, you know, the moral of Parliament idea is an that was laid out by Nick Bostrom and then overcoming bias posts, like something like a decade ago that I like, I think about it as if I'm just multiple people, I just think about it as if I'm just there's multiple people all living inside my head, arguing about what to do. And they all are friends, they all care about each other, and they want to get along.
And so they're trying to reach a deal that all them can feel fairly good about. That is how I tend to think about dealing with different moral views.
So I tend to want to do things that are like really good according to one and not too bad according to the rest and try to have the kind of different parts of myself making deals with each other. So that relates to something I said at the beginning about not being into ends justify the means.
I put a lot of effort into doing things that would be like really really good if this most important century stuff came out true but also not too catastrophic if it didn't like i mean there's there's lines i'm not willing to cross there's behaviors i'm not willing to engage in um to promote the kind of you know goals of people who worry about ai safety so it's it's a it's a moderating approach i think gotcha okay yeah that that makes a lot of sense for somebody who's the CEO of open philanthropy, that you would want the decisions you make to reflect uncertainties about your decisions. But if it's just somebody like me, where I'm not in some sort of leadership position where I have a large amount of resources to dedicate to, should I just specialize in that particular moral view I have? Or should I also be trying to allocate my time and resources according to different moral views? I mean, I think no matter what position I was in in the world, and however many resources I had, I would feel that my decisions were significant in some sense and affected people, and were important in that way that they that they affect those around me.
So I think it's just very natural to me to think, you know, there's a lot of different perspectives on what it means to be a good person. And rather than trying to turn them into a single unifying mathematical equation and take the expected value, which is another approach that I think is interesting.
But I think the approach I do tend to prefer is to imagine the different perspectives as different people trying to get along and make a deal with each other. Let's talk about governance and management.
So in software, as I'm sure you're aware, there's a concept of a 10x engineer. Is there something similar in the kinds of work a research analyst at Open Philanthropy does? Is it meaningful to say that two people doing the same job, one can be orders of magnitude more effective than another? Yeah, I think some people are much, any given thing in Open Philanthropy, some people are much better at it than others.
I don't think that's very surprising. I think it's true in many jobs.
And I don't really know the reasons for it. It could be any combination of talent, interest, how hard someone works at it.
So, yeah. But I certainly think there's a lot of variance.
And hiring people who can do a great job at the work OpenPhil does has been a lifelong challenge. You've written about the Bayesian mindset.
You know many billionaires. Many of them are donors to Open open philanthropy.
In your experience, do these startup founders who end up becoming very successful, do they have a Bayesian mindset or is that the wrong way to characterize there? Yeah, I wrote about this idea of Bayesian mindset, which is basically like being willing to kind of put a probability on anything and use your probabilities and say your probabilities as a way of discovering why it is you think what you think and using expected value calculations. Similarly.
Uh, I think this is like much more common among successful tech founders than it is among like the general population. Um, but there's plenty of tech founders who don't think this way at all.
I don't think, you know, I say in the Bayesian mindset, I don't think it's like a super well-tested, well-proven social technology that does amazing things. I think it's more like an interesting thing to be experimenting with.
Okay, that seems... Well, the general population, I mean, it's practically unheard of.
Right. Basin's mindset, yeah.
I mean, not even just the name. I mean, just the idea, this whole idea of thinking expected value and subjective probabilities all the time.
It's just like almost no one does that. I think tech founders probably do it more than the average person..
No, that makes sense. Do you think that adopting more of a Bayesian mindset would help somebody at the top levels be more successful? It's really TVD and unclear.
I think, I just think the Bayesian mindset is a cool thing to experiment with. I experiment with it a lot.
I feel like it helps me sometimes. Like most things, it's good in moderation and with taste and not using it for every single thing.
And maybe 10 years from now, as it gets more popular, a better sense of where the actual applied strengths and weaknesses are. As I'm sure you're aware, there's many prizes floating around for all kinds of intellectual work in effective altruism.
And some of them even have come from open philanthropy. Yeah.
Are you optimistic about their ability to resurface or surface new ideas? I would say I'm like medium optimistic about the impact of all these prizes. I mean, I've been part of designing some of them.
I've just seen some other ones like people say, you know, hey, we'll pay you X dollars if you can give us a good critique of our, like give well, we'll pay people to give them a good critique of their reasoning about what the best charities are to give to. Open philanthropy as a prize for like showing us a cause we should be looking at that we're not.
I don't know. I think I'm medium optimistic.

I mean, I think it will get some interest and it will get some people to pay attention

who weren't otherwise.

And some of those people might have good ideas.

I don't think it's the only way to solve these problems or that it will automatically

solve them.

And that's generally how the people designing the prizes think about them too.

You have an interesting post about stakeholder management that over time institutions have

to take into account the interests of more and more stakeholders.

Do you expect that this will be something that will be a major factor in how open philanthropy

Thank you. post about stakeholder management that over time institutions have to take into account the interest of more and more stakeholders.
Do you expect that this will be something that will be a major factor in how open philanthropy acts in the future? And do you think, what will be the impact on how open philanthropy runs overall? Yeah, I think in general, the bigger your organization is, the bigger your city is, the bigger society is, there's more people who are, you know, if you want everyone to be happy, there's more people you're going to have to make happy. And I think this does mean, you know, in general, by default, as a company grows, it gets less able to make a lot of disruptive quick changes.
A lot of people would use the term nimble. A lot of people in the tech world like to just use these like very negative terms for big company properties and very positive terms for small company properties.
So small companies are like nimble and quick and practical and adaptive and dynamic and high productivity. And big companies are like bureaucratic and slow and non-adaptive.
I think that's all fair. I also think that big companies often just like at the end of the day, just like produce more stuff than they could if they were small.
I mean, like, you know, I think if Apple were still 10 people, it might be a more exciting place to work, but they wouldn't be able to make all those iPhones. You know, there's a lot of iPhones going out to a lot of people, serving a lot of different people's needs, abiding by a lot of like regulatory requirements.
There's a lot of work to be done. So I don't think, you know, I don't think it's necessarily a bad thing, but I think it's a trade-off for a company to grow.
I do think open philanthropy is, you know, in the business of doing kind of unconventional giving and using a lot of judgment calls to do it. So I tend to think we benefit a lot from staying as small as we can.
And I generally have fought for us to stay as small as we can while doing our work. But we still have to grow from where we are.
Gotcha. Do you mean stay small in terms of funds or do you mean people? Yeah.
But it seems odd to say that the organization you have the most experience with, your inside view is that more stakeholders would be bad. But overall, it's been a net zero or positive.
It's not clear. Like, we're growing.
I mean, we're bigger than we were a year ago. We'll be bigger in a year.
So it's definitely not true that I'm just, like, minimize the size of the company. I mean, we're growing.
But I think we want to, you know, I think we want to watch it. I think we want to treat each hire as something that we only do because we had a really good reason to.
And I think there are some companies that may have more to gain from being 10,000 people. I don't think we'll ever be 10,000 people.
Right, right, right. Now, your written career advice emphasizes building aptitudes and specializing.
But when I look at your career, it's all over the place, right? We were just talking about it at the beginning of the interview, you started off in GiveWell, then you were working at Open Philanthropy and now you're forecasting AI. Like, so how do you think about this kind of thing? Where are you specializing or what's going on here? I don't know if I really forecast AI.
I mostly, I mostly distill and bring together analyses that others have done. And I manage people who work on that sort of thing.
Yeah. I mean, I don't know.
I think it's, I think it's really good. I mean, the career advice I often give is just like, it's really good to just have something you're very focused on, that you're specializing in, that you're trying to be the best in the world at.
You know, the way that my career went is like, first off, the general theme of my career is just taking questions, especially questions about how to give effectively, where just like, no one's really gotten started on this question. So even doing a pretty crappy analysis can be better than what already exists.
Often what I, what I have done in my career, what I consider myself, you know, to have kind of specialized in, in a sense, is I do the first cut crappy analysis of some question that has not been analyzed much and is very important. And then I build a team to do better analysis of that question.
So that's, that's been my general pattern. I think that's the most generalizable skill I've had, but I have switched around because I do think that, you know, that I've, I've kind of had various points in my career just said, Hey, here's something that's getting very little attention and it's very important.
And it's worth the sacrifice of the specialized knowledge I built up in one area to switch into this other area that I think I ought to be working on. What does the logo on the ColdTakes blog mean?

There is no logo.

I think you're talking about the browser icon.

Yeah, yeah.

So that is a stuffed animal named Mora.

And at some point, if I get enough subscribers, I will explain who all these stuffed animals are.

But my wife and I basically use a stuffed animal personality classification system

where we will compare someone to various stuffed animals to explain what their strengths and weaknesses are um and maura is a pink polar bear who's like very creative but also very narcissistic and loves attention so she's kind of the mascot of the blog because it's it's kind of this blog that's just like very crazy very out there and and is just like me writing in public and so it just felt like her spirit gosh Gosh, okay. So let me ask, what is the goal of the blog? Having a second job as a blogger in addition to being the CEO of a big organization? No, I think it fits into my job reasonably well.
I didn't want it to be open philanthropy branded. I just wanted the freedom to write about things the way I wanted and how I wanted.
But I do think that like we make these high stakes decisions based on very unconventional views about the world. And I think it's good for us to be trying to make those views have contact with the rest of the world.
So I think there would be something not ideal about being a large foundation, giving you large amounts of money. And we're just kind of quietly going around believing these enormously important, if true things that no one else believes.
And it is a way of kind of, you know, if we put the views out into the world, A, I think just like all the people seeking money from us just have a better idea of where we're coming from. Just like, why is it that we're interested in funding what we're interested in funding? And I think to the extent people find the arguments compelling, or even just like understand them and helps understand our thinking, I think that can help create more grantees for us.
It can help, you know, cause the world to be a place where there's more good stuff for us to fund because more people see where we're coming from, hopefully agree with it and are trying to work on the things we consider important. And then to the extent that my stuff is actually just like screwed up and wrong, and I've got mistakes in there and I've thought it all through wrong.
This is also like the best way I know of to discover that. It's just, um, you know, I don't know how else I'm, how else I'm going to people to critique it, except by putting it out there and maybe getting some attention for it.
So that's how I

kind of think of it is it's taking views that are very important to the decisions we're making

and trying to express them so that we can either get more people agreeing with us who we're able

to fund and support and work with or learn more about what we're getting wrong. All right.
So let

me actually ask you, has that happened? Like, has there been an important view that expressed on on the blog that because of feedback you change your mind on, or is it mostly about the communication part? Let me think. I mean, there's definitely been like a lot of interesting stuff.
I mean, like an example is I put up this post on the track record of futurists, and then there was a post by, I think, Dan Liu recently that I haven't read yet, but it just has its own analysis of the track record of futurists, and I need to compare them, you know, think about what I really think about how good humans have historically been about predicting the future. But he has, he certainly has a ton of data in there that I was not aware of that, you know, that feels like feels like a bit of a response, may or may not have been prompted by it.
Yeah, there's been there's been a lot of commentary, there's been a lot of like, you wrote some critiques of some of the stuff I've written in the most important century. have been other critiques i mean i don't know like i i think it i think a lot of the stuff i wrote about the biggest weak points of the most important century were based on the public criticism that was coming in so i think it has like yeah i think i have become more aware of a lot of the like the parts of my parts of my thinking that are the least convincing or the most weak or the most in need of argument and i have like paid more attention to those things because of of that.
Yeah. Right.
Okay. This may just be me talking, but it actually does sound like maybe you've learned about how people react to the most important century thesis, but it doesn't seem like something has surfaced, which has made you change your mind on it a lot.
Oh, I mean, I haven't dropped my, I mean, that would be a big, a big change, like to drop my view that, that we could be in the most important century for humanity. Like that's still what I believe.
I'm, you know, I mean, I think I've also heard from people who just think I'm like underselling the whole thing, like crazy, like people who just think that, um, that, that I should be planning on transformative AI much sooner than, than what I say, than what I kind of imply in the series. So, yeah, I mean, I, I wouldn't, I mean, I, I put out this thing most important century and I, I don't believe any of the critiques have been deep enough and strong enough to make me just like drop that whole thing.
But it's a big picture with a lot of moving parts, and I have kind of deepened my understanding of many of the parts. Yeah, yeah, yeah.
One thing I find really interesting about your work is how much it involves the CEO having a deep understanding of all the issues involved. you are the one who has to like deeply understand, for example, we were just at Memorial Parliament or whether it's like specific forecast about AI, like biological anchors or whatever else, right? It seems maybe in other organizations, the CEO just delegates this kind of understanding and just asks for the bullet points.
Is this something you think more leaders should be doing or is there something special about your position?

I know much less about any given topic than the person who specializes in the topic.

And I think what I try to do is I try to know enough about the topic that I can manage them effectively.

And that's like a pretty general corporate best practice.

And I think it just varies a lot. So I think, for example, like something like keeping our books, like keeping our finances, doing the financial audits, all that stuff.

That's something that's like really easy to just judge the outputs without really knowing much about finance at all. You can just say, look, was this compliant? Did we do our audit? Did we pass the audit? You know, do we still have money in the bank? Like what's, how much money do we have in the bank? You don't need to know much about it to judge it effectively, but at any company, I mean, you, you may need, you may need to know a fair amount about the top, about some other topics in order to judge them them very effectively if your company is making computers or phones and design is very important you probably the ceo would be really bad if the ceo just had like no opinions on design it was just like well i'm going to let our design person decide the design i mean it's it's a central thing to the company it matters of the company and they should know some things about it so i do you know things that are really central to open philanthropy what is what does it mean to do? How do we handle uncertainty about how to do good? What are the most important causes? If AI might be one of the most important causes, then when might we see transformative AI? What would that mean? How big is the risk of misaligned AI? I think I need to understand those issues well enough to effectively manage people who know a lot more about them than I do.
Yeah. And I'm just curious, like, what do you, you know, what do you think about this whole most important century stuff? Does this just strike you as like crazy? Or like, you know, what do you think when you read the series? Yeah, I've been obviously like through that entire interview, I've been trying to like nitpick small things.
But when I really think about what is the main claim you're making is that this could be the most important century Transformer DBI could happen in the century. And if it does, it's a really big deal.
It's like, yeah, I don't disagree. That makes a lot of sense.
Throughout throughout like preparing for the interview and trying to come up with objections i've just been like a little bit more and more convinced with thinking about like is there actually something i could do over yeah my early career that matters or is that something that maybe i should just hold off on thinking about glad to hear it do you have any ideas what you might do uh no really really literally no ideas you haven't been like could i work on ai alignment or oh well yeah in sense, I thought a little bit about it. Probably in like two months or three months, I'll think real hard about what I actually want to do for a career.
Hey, thanks for listening. If you enjoyed that episode, I would really appreciate it if you could share it.
This is still a pretty small podcast, so it is a huge help when any one of you shares an episode that you like.

Post it on Twitter, send it to friends who you think might like it, put it in your group chats. Just let the word go for it.
It helps out a ton. Many thanks to my amazing editor, Graham Besalou, for producing this podcast.
and to Mia Ayana for creating the amazing transcripts

that accompany each episode,

which have helpful links. producing this podcast and to Mia Ayana for creating the amazing transcripts that accompany

each episode, which have helpful links. And you can find them at the link in the description below.