Breaking the internet

President Donald Trump has been back in office for one month, and what a year it's been. We've covered a lot of Trump at Today Explained this past month, from pardons to executive orders to Greenland to Guantanamo to tariffs to Maha to Elon and Elon and even more Elon.
But today we're going to talk about the websites. DEI would have ruined our country, and now it's dead.
I think DEI is dead, so if they want to scrub the websites, that's okay with me. Government webpages are disappearing.
Sometimes they come back, sometimes they don't, and it's part of a greater problem we have online. Some call it digital decay, others call it link rot.
Whatever you call it, our internet is disappearing, And we're going to help you understand why it matters and what we can do about it on the show today. At UC San Diego, research isn't just about asking big questions.
It saves lives and fuels innovation, like predicting storms from space, teaching T-cells to attack cancer, and eliminating cybersecurity threats with AI. As one of America's leading research universities, they are putting big ideas to work in new and novel ways.
At UC San Diego, research moves the world forward. Learn more at ucsd.edu slash research.
Support for this show comes from Capital One. With the VentureX Business Card from Capital One, you earn unlimited double miles on every purchase.
Plus, the VentureX business card has no preset spending limit.

So your purchasing power can adapt to meet your business needs.

Capital One. What's in your wallet?

Today Explained Today. Today explained.
This is dead. This is me too.
This is it. Today explained.
Today explained. Sean Ramos from here with Addie Robertson, senior editor at The Verge, here to tell us about the websites.
What is going on with the government's websites? So Trump signed a couple of executive orders, one of which defined officially the idea that there are only two genders, male and female. And another one that ends, quote unquote, diversity, equity and inclusion in the government.
We will forge a society that is colorblind and merit based. And so the result here has been that more or less across the government, in addition to the kind of thing that we saw in the first Trump administration, which included purging information about climate change and some other general climate related issues, we've seen just a massive cut of anything that involves racial equity or transgender people or really anything that is sort of a subject of Republican culture wars.
The CDC is currently scrubbing information from their website right now to be in compliance with a recent executive order. Here are some of the pages that have gone down.
The Trump administration has taken away reproductive rights.gov from the federal website. They also have scrubbed federal websites for any search of abortion.
Within hours of President Trump's inauguration, the Spanish version of the official White House website disappeared. The website now gives users an error 404 message.
A lot of the stuff initially happened very quietly. Reporters noticed it.
People who used the information on these sites, which included data on the CDC or even transportation statistics, they have ended up uncovering a lot of this. And from there, the way that the Trump administration has mostly addressed it is in response to lawsuits, that there were claims that they deleted this data improperly.
There was a court order that required them to put it back up. And they have responded by putting it back up with a big banner that says, we reject this information.
We were forced to keep it online. But it violates something like, say, our dictate that there are only two sexes.
So we find it unscientific or we find it against our policies. Any information on this page promoting gender ideology is extremely inaccurate and disconnected from the immutable biological reality that there are two sexes, male and female.
Is there presidential precedent for something like this happening? Or is Donald Trump and Doge and Elon Musk and the gang like the first administration to come in and just start ripping apart websites? First of all, just for context, every time there is a new presidential administration, there is data that changes their priorities. There are new programs or old programs that get retired.
So it's not necessarily surprising that some things have changed. But we have, as part of this, seen just a massive and really unprecedented deletion of information, including information that is required for people to do their jobs outside of the White House.
And so it's a really huge issue right now. I don't think we've ever seen this kind of scale of data purging, especially of records and scientific research.
Obviously, the first Trump administration deleted some data in ways that seemed very ideological, aimed at suppressing information about climate change. The White House and other federal agencies are also revamping their websites, for instance, scrubbing mentions of climate change.
And Trump is blasted. And obviously there have been pages that just disappeared at the end of terms, but that tended to be more about oversight.
It tended to be more that there was a changing of the guard and they didn't really know where everything was.

So some websites are disappearing. Some websites are disappearing and coming back.
Some websites are still up. Is there anyone who has like a full grasp of what exactly is gone forever? There are nonprofit groups and journalists that are working to preserve this information.

There were already groups before Trump that are working to preserve this

information. There were already groups before Trump took office, like the Environmental Data and Governance Initiative, that we saw a little of this in Trump's last term.
And so there was this effort preemptively to preserve information, which includes not just web pages, but also just collections of data from groups like the CDC. So there are all of these, not necessarily fragmented, but individual and private efforts.
And also one of the really big load-bearing institutions here is the Internet Archive and the Wayback Machine, which has always maintained this project that archived data at the end of every term, but now has become a place where you can go and check and see what's disappeared and has become part of this process of identifying and trying to recover data. Beyond the American people perhaps needing access to some of this information, beyond any number of institutions needing access to this information, it points at a bigger problem we have on our internet right now, right? Something called link rot? Link rot or digital decay, which is a general phenomenon where webpages either disappear or they move in a way that makes them more difficult to find.
And so the internet, which is a series of links that point to information, ends up with all of these little dangling ends and dead links and places where you can no longer find information that someone has referred to or when you can simply no longer find a record of it at all. A 2013 Harvard study, for example, found that half the hyperlinks in Supreme Court cases, today's equivalent of footnotes, are broken, a phenomenon known as link rot.
Why do webpages disappear? The most obvious case is when a page is just taken down, maybe sometimes because the entire website went under, maybe sometimes

because they think that page is no longer valuable. Government agencies remove documents and companies fail and with them the sites they host.
Think of GeoCities, Yahoo Video, and more recently the news site Gawker. There are also incidents where just the URL of it, the link that points to that information, changes.
And so it's harder to find. So if you previously linked to it from another web page, then that's just not going to go there anymore.
The wonder of it is it's very, very simple. Anybody could go and set up a web server on their computer and make it available to the world.
Unfortunately, it's too simple. It's fragile.
That if something happens, that piece of equipment, that website, just blink, is gone. So you've been covering this issue, Addy, for more than 10 years.
Is LinkRot getting worse online, or is it sort of, you know, continuing apace? LinkRot has been an issue that people have been identifying in some ways since really the beginning of the internet. But for definitely at least a decade, a really significant proportion of web pages and links have no longer functioned.
I think the latest research was something like 38% of web pages that existed in 2013 are no longer available. This is, I think, not necessarily an issue that has suddenly snowballed, but I think we're seeing some unique circumstances now that have added to it.
One of them is something like search engine optimization, where Google rewards pages, or at least people think it rewards pages, that regularly refresh or that seem like they are providing new information. And so, for instance, CNET, which is a really venerable tech publication, removed a bunch of its older articles because it wanted to appear in Google search results more highly.
And so there was this sense that, okay, it makes people more likely to find current articles, but also just this trove of information disappears. Right.
I mean, I think we can all, you know, mourn the loss of like our Geocities homepage from 2003. But it's a lot rougher when like, I don't know, some billionaire buys out an alternative newspaper and just decides one day to shut down its website.

Sometimes it's a billionaire that buys something and shuts it down. There are also just more insidious phenomena that I think really kind of speak to the commercialization of the internet and the sort of cannibalization of anything that can be turned toward profit.
So you have old websites that say have a name people recognize and then they get resurrected, but they no longer have the old information. They've been filled with AI generated new articles that can sort of capitalize on this old name as this zombie site.
Or you have issues where there's a link that goes dead and somebody tries to kind of hijack that link and they either they contact the website administrator or they find some other way to get that to point to a new page that will then build their own credibility but doesn't provide the original information. So there are all these cases where archival gives way to profit.
That information was useful sometimes because it provided, say, statistics or it provided evidence. If you're, say, looking at Wikipedia and there's a dead link that no longer provides the information it used to.
And sometimes just because these things are a valuable record of what the internet

used to be and of how people lived, there are a lot of things that at one point would have been

written down on paper or in some other medium that's just a hard document and people can look

back on it. But at this point, a huge amount of our culture takes place on the internet and the internet is a very fragile place.

Addie Robertson, reader at TheVerge.com. When Today Explained returns, we're heading into the

Wayback Machine to hear from the people trying to archive the entire internet one web page at a time.

Fox Creative.

This is advertiser content brought to you by the all-new Nissan Murano. Okay, that email is done.
Next on my to-do list, pick up dress for Friday's fundraiser. Okay, all right, where are my keys? Oh, in my pocket.
Let's go. First, pick up dress, then prepare for that big presentation.
Walk dog. Then, okay.

Inhale.

One, two, three, four.

Exhale.

One, two, three, four.

Ooh, who knew a driver's seat could give such a good massage?

Wow, this is so nice.

Oops, that was my exit. Oh, well, that's fine.
I've got time. After the meeting, I gotta remember to schedule flights for our girls' trip, but that's for later.
Sun on my skin, wind in my hair. I feel good.
Turn the music up. Your all-new Nissan Murano is more than just a tool to get you where you're going.
It's a refuge from life's hustle and bustle. It's a place to relax, to reset, into spaces between items on your to-do lists.
Oh, wait, I got a message. Could you pick up wine for dinner tonight? Yep, I'm on it.
I mean, that's totally fine by me. Play Celebrity Memoir Book Club.
I'm Claire Parker. And I'm Ashley Hamilton.
And this is Celebrity Memoir Book Club. Support for this show comes from Indeed.
The most exciting part of a heist movie is when the heroes bring together a team of guys. The best safecracker, the fastest driver.
With one montage, they're all assembled. Unfortunately, in real life, you can't hire people with one simple jump cut.
Luckily, there's Indeed, where you can find relevant candidates and qualified talent quickly. Indeed's Sponsored Jobs allows your listings to jump to the top of the page for your relevant candidates.
Plus, with Indeed's Sponsored Jobs, there are no monthly subscriptions, no long-term contracts, and you only pay for results. Join the 3.5 million

employers worldwide that used Indeed to hire great talent fast. Speed up your hiring right now with

Indeed, and listeners of the show will get a $75 sponsored job credit to get your jobs more

visibility at Indeed.com slash Vox Business. Just go to Indeed.com slash Vox Business right now and

support our show by saying you heard about Indeed on this podcast. Indeed.com slash Vox Business right now and support our show by saying you heard about Indeed on this podcast.
Indeed.com slash Vox Business. Terms and conditions apply.
Hiring? Indeed is all you need. Hey everyone, Sean here.
I'm really excited to share with you our special series, AI and Us, sponsored by Canva. We'll take an in-depth look at how AI will impact our lives in the future and explore the observations, ambitions, and reasoning behind it all.
In our first episode, I'll interview futurist and tech pioneer Jaron Lanier about the current state of AI, the potential it can unlock if we manage this technology well, and the implications for humanity if we don't.

And then in the second episode, I'll talk to Julia Longoria, host of Vox's Good Robot,

about the beliefs and ideologies of the people building, funding, and influencing artificial

intelligence, and how looking at this AI origin story can provide clues into how this technology

will change the way we live and work. You can find our special series, AI&Us, right here on the Gray Area.
This is Today Explained. So let's just have you start by saying your name and what it is you do.
Sure. Yeah.
Hi. My name is Mark Graham, and I am the director of the Wayback Machine at the Internet Archive.
Which is a not-for-profit that has been preserving the web since 1996. Journalists use it all the time.
But for the uninitiated, I asked Mark to show us around the Internet Archive. Where do I begin? It's like walking into a very large library and saying, show me your favorite book.

Well, for example, last year it was a big news story that MTV News was shut down.

And the founding editor of MTV News wrote about it on LinkedIn.

And there was a lot of other editors talking about it.

It was like, oh, my God, all of our articles are gone.

They're missing.

And I just casually, you know, waded into the conversation and go, hi, check here, Wayback Machine. And they were like, oh my God, you guys like got it all pretty much.
Yeah. And they said, well, you know, people say, well, what did you do? I mean, did you, what did you do when it went down? You must've, I say, we didn't do anything when it went down because we've been doing our job all along.
We've been working to archive the public web as it's published on an ongoing and continuous basis. So if we have to start paying attention to something after it's gone down, that means we screwed up.
So with that example, with MTV News, give us a sense of what you guys were doing in advance of that website going down to make sure that people could find out, you know, I don't know, what Everlast was singing about in 2004. Hello, Jancy Dunn here, and joining me now is former House of Pain member Everlast.
Welcome, Everlast. Thank you.
So for any one of a number of thousands of reasons, we set our web crawlers and archiving software out on a mission every day to identify and to download web pages and related web-based resources. We bring in millions and millions of URLs every day that are signals to us, signals of where new material is being published on the web.
And we make sure that we archive all of those URLs, all the web pages associated with those URLs. And then we look at those pages and we identify links to other pages.
And then we go to those pages and we archive them, etc, etc, etc. That's where you get this metaphor of crawling like a spider throughout this web.
And the net result of it is that we add more than a billion archived URLs to the Wayback Machine every day.

And this material, as it's added to the Wayback Machine, is indexed and is immediately available to people who go to web.archive.org, enter in a URL, and then are able to see a history of archives that we have of the web page that was available from the URL in any given time. I want to talk about government websites now because that's sort of the reason we're having this conversation today.
I think most people probably think the government will take care of archiving government websites, but here we are in a new administration and websites are disappearing, coming back online, and people are worried. When you, an archivist of the internet, see government websites disappearing, coming back online, becoming unreliable, how do you react to that? Is that like better or worse than regular websites that are non-governmental going offline? Well, as an American, my tax dollars help pay for some of this stuff.
And that much of it is maybe a benefit to people. Certainly, my first reaction is, hmm, that might not be such a good thing.
I do want to underscore that there is the National Archives and Records Administration that does do archiving as well. But for whatever reason, we seem to be like one of the main players in the space of trying to archive much of the public web, including, and right now especially, U.S.
government websites, and making those archives available in near real time. Were you caught off guard when you saw the new administration removing web pages, removing websites? This is pretty normal in some respects.
It's normal and expected, and it's what's happened, frankly, for each administration in the time that we've been working on this effort. I mean, look, it's under new management, right? For example, you wouldn't expect the whitehouse.gov website under any new presidential administration to be the same as it was before.

So we go out of our way to try to anticipate the frequency in which web pages should be archived so that we got a pretty good shot at getting those changes. You're saying, you know, the whitehouse.gov site obviously changes administration, administration.
I think to some degree people understand that that joe biden's administration probably wouldn't have been posting trolley valentines about immigration you know a year ago this time to their instagram account but we're seeing what we're seeing here is is websites that people need websites that record public health information going offline, briefly, permanently, what have you. No, that's true.
Is that a different degree of sort of erasing the historical record or messing with the historical record than we've seen? It's different. It's certainly different in terms of the number seemingly.
I mean, we're still in the early stages of this administration. But yeah, I'd say on the face of it, you're right.

historically we haven't seen

major US government websites taken offline

like we did say for example

with regard to USAID

and I'm going to leave that kind of analysis to others and really just focus on trying to archive the material. The Wayback Machine, the Internet Archive, mostly funded through donations, the generosity of people, institutions, even governments.
Is that going to be enough to archive the internet to the extent that future generations will want to see and need? Enough is a very subjective term. Well, as an archivist, for me, it's never enough.
Because you don't know. No one knows what is going to be of use, value, importance in the future.
Maybe even the near future of tomorrow, much less like the very far off future. And since millions of people use our site on a daily basis, we get a lot of feedback from them.
It motivates us, but it also helps direct us and inspires us to continuously try to do a better job at being the best library that we can be. Godspeed.
There you have it. Let me ask you one last question, Mark.
You guys have been at this for nearly three decades. Certainly, you've saved a lot of stuff, and certainly a lot of stuff has fallen through the cracks.
I wonder, is there something that slipped through the cracks that you could tell us about that might suggest to our audience what is lost when we can't archive to the extent we want to or need to?

Okay, so it kind of caught me up with that question.

I'll just say, I don't know right now.

I can't say that thing.

Gosh, I wish.

Okay, I got one.

I mean, this is just in recent history.

Apparently, there was a page up on the CDC website about bird flu last week. It apparently was only up for a few minutes and no one got it.
Huh. And by losing that fleeting webpage, that one, you know, maybe minor, maybe major webpage about bird flu on the CDC's website, what are we losing? Well, we're losing part of the story, right?

We're losing part of our understanding

of the evolution of arguably

a significant health issue, concern.

We don't know where this is going to go.

I don't know.

I guess that's the other point, right?

I mean, you don't know necessarily now

that which is going to be very important in the near or longer term. In the time of Martin Luther, there was raging, raging debates.
And much of that debate took the form of things that were written on pamphlets. The pamphlets at the time were considered of little value.
I mean, people read them and they shared them, but they didn't necessarily save them. So today, a scholar of that time or someone like me was just kind of strangely curious, what I would give for a collection of those pamphlets? Yeah.
I mean, and you are comparing, in a way,

a CDC website to the Protestant Reformation,

but I think you mean it, don't you?

I do, because I don't know.

And one really can't know

without the benefit of the long historical view.

And that's not something that we have access to today. Why? Because we don't have a real-time machine.
Mark Graham, known exclusively to Amanda Llewellyn as WebMG. Check out the Wayback Machine at web.archive.org.
Amanda produced the show today. Laura Bullard helped and wore the hat.
Jolie Myers edited. Andrea Kristen's daughter and Patrick Boyd mixed.
And Andrea even made some original music. Thanks to the Free State of Aftonia for the Wi-Fi.
Oh, and it's Today Explained's seventh birthday today. What did you get us? Maybe show some love in the comments

and the ratings and the reviews.

They say it helps.

And thank you for listening

for however long you've been listening.

If you're new to the show,