Breaking the internet

Speaker 1 President Donald Trump has been back in office for one month, and what a year it's been.

Speaker 1 We've covered a lot of Trump at Today Explained this past month, from pardons to executive orders to Greenland to Guantanamo to tariffs to Maha to Elon and Elon and even more Elon.

Speaker 1 But today we're going to talk about the websites.

Speaker 2 DEI would have ruined our country and now it's dead. I think DEI is dead, so if they want to scrub the websites, that's okay with me.

Speaker 1 Government web pages are disappearing. Sometimes they come back, sometimes they don't, and it's part of a greater problem we have online.
Some call it digital decay, others call it link rots.

Speaker 1 Whatever you call it, our internet is disappearing. And we're going to help you understand why it matters and what we can do about it on the show today.

Speaker 3 Every story you love,

Speaker 3 every invention that moves you,

Speaker 3 every idea you wished was yours, all began as nothing.

Speaker 3 Just a blank page with a blinking cursor

Speaker 3 asking a simple question:

Speaker 6 What do you see?

Speaker 3 Great ideas start on Mac.

Speaker 3 Find out more on apple.com/slash Mac.

Speaker 7 Support for this show comes from OnePassword.

Speaker 8 If you're an IT or security pro, managing devices, identities, and applications can feel overwhelming and risky.

Speaker 7 Trellica by OnePassword helps conquer SaaS sprawl and shadow IT by discovering every app your team uses, managed or not.

Speaker 5 Take the first step to better security for your team.

Speaker 7 Learn more at onepassword.com slash podcast offer. That's password.com slash podcast offer.

Speaker 4 All lowercase.

Speaker 6 This is a day today's flame. This is dead.

Speaker 9 This is day.

Speaker 1 Sean Ramas from here with Addie Robertson, Senior Editor at The Verge, here to tell us about the websites. What is going on with the government's websites?

Speaker 9 So Trump signed a couple of executive orders, one of which defined officially the idea that there are only two genders, male and female, and another one that ends, quote-unquote, diversity, equity, and inclusion in the government.

Speaker 10 We will forge a society that is colorblind and merit-based.

Speaker 9 And so the result here has been that more or less across the government, in addition to the kind of thing that we saw in the first Trump administration, which included purging information about climate change and some other general climate-related issues, we've seen just a massive cut of anything that involves racial equity or transgender people or really anything that is sort of a subject of Republican culture wars.

Speaker 11 The CDC is currently scrubbing information from their website right now to be in compliance with a recent executive order. Here are some of the pages that have gone down.

Speaker 12 The Trump administration has taken away reproductive rights.gov from the federal website. They also have scrubbed federal websites for any search of abortion.

Speaker 13 Within hours of President Trump's inauguration, the Spanish version of the official White House website disappeared. The website now gives users an error 404 message.

Speaker 9 A lot of the stuff initially happened very quietly.

Speaker 9 Reporters noticed it. People who used the information on these sites, which included data on the CDC or even transportation statistics, they have ended up uncovering a lot of this.

Speaker 9 And from there, the way that the Trump administration has mostly addressed it is in response to lawsuits.

Speaker 9 There were claims that they deleted this data improperly. There was a court order that required them to put it back up.

Speaker 9 And they have responded by putting it back up with a big banner that says, we reject this information.

Speaker 9 We were forced to keep it online, but it violates something like, say, say, our dictate that there are only two sexes, so we find it unscientific or we find it against our policies.

Speaker 14 Any information on this page promoting gender ideology is extremely inaccurate and disconnected from the immutable biological reality that there are two sexes, male and female.

Speaker 1 Is there presidential precedent for something like this happening?

Speaker 1 Or is Donald Trump and Doge and Elon Musk and the gang like the first administration to come in and just start ripping apart websites.

Speaker 9 First of all, just for context, every time there's a new presidential administration, there's data that changes, there are priorities, there are new programs or old programs that get retired.

Speaker 9 So it's not necessarily surprising that some things have changed.

Speaker 9 But we have, as part of this, seen just a massive and really unprecedented deletion of information, including information that is required for people to do their jobs outside of the White House.

Speaker 9 And so

Speaker 9 a really huge issue right now.

Speaker 9 I don't think we've ever seen this kind of scale of data purging, especially of records and scientific research.

Speaker 9 Obviously, the first Trump administration deleted some data in ways that seemed very ideological, aimed at suppressing information about climate change.

Speaker 17 The White House and other federal agencies are also revamping their websites, for instance, scrubbing mentions of climate change. And Trump is blasting.

Speaker 9 And obviously, there have been pages that just disappeared at the end of terms, but that tended to be more about oversight.

Speaker 9 It tended to be more that there was a changing of the guard and they didn't really know where everything was.

Speaker 1 So some websites are disappearing. Some websites are disappearing and coming back.
Some websites are still up. Is there anyone who has a full grasp of what exactly is gone forever?

Speaker 9 There are nonprofit groups and journalists that are working to preserve this information.

Speaker 9 There were already groups before Trump took office, like the Environmental Data and Governance Initiative, that we saw a little of this in Trump's last term.

Speaker 9 And so there was this effort preemptively to preserve information, which includes not just web pages, but also just collections of data from groups like the CDC.

Speaker 9 So there are all of these, not necessarily fragmented, but individual and private efforts.

Speaker 9 And also one of the really big load-bearing institutions here is the Internet Archive and the Wayback Machine, which has always maintained this project that archive data at the end of every term, but now has become a place where you can go and check and see what's disappeared and has

Speaker 9 become part of this process of identifying and trying to recover data.

Speaker 1 Beyond the American people perhaps needing access to some of this information, beyond any number of institutions needing access to this information, it points at a bigger problem we have on our internet right now, right?

Speaker 1 Something called link rot.

Speaker 9 Link rot or digital decay.

Speaker 9 Which is a general phenomenon where web pages either disappear or they move in a way that makes them more difficult to find.

Speaker 9 And so the internet, which is a series of links that point to information, ends up with all of these little dangling ends and dead links and places where you can no longer find information that someone has referred to, or when you can simply no longer find a record of it at all.

Speaker 18 A 2013 Harvard study, for example, found that half the hyperlinks in Supreme Court cases, today's equivalent of footnotes, are broken, a phenomenon known as link rot.

Speaker 1 Why do web pages disappear?

Speaker 9 The most obvious case is when a page is just taken down. Maybe sometimes because the entire website went under, maybe sometimes because they think that page is no longer valuable.

Speaker 18 Government agencies remove documents and companies fail and with them the sites they host. Think of GeoCities, Yahoo Video, and more recently the news site Gawker.

Speaker 9 There are also incidents where just the URL of it, the link that

Speaker 9 points to that information changes and so it's harder to find. So if you previously linked to it from another web page, then that's just not going to go there anymore.

Speaker 1 The wonder of it is it's very, very simple.

Speaker 1 Anybody could go and set up a web server on their computer and make it available to the world. Unfortunately, it's too simple.
It's fragile.

Speaker 1 That if something happens to that piece of equipment, that website, just blink is gone.

Speaker 1 So you've been covering this issue, Addie, for more than 10 years.

Speaker 1 Is link rot getting worse online, or is it sort of, continuing apace?

Speaker 9 Link rot has been an issue that people have been identifying in some ways since really the beginning of the Internet, but for definitely at least a decade, a really significant proportion of web pages and links have no longer functioned.

Speaker 9 I think the latest research was something like

Speaker 9 38% of web pages that existed in 2013 are no longer available.

Speaker 9 This is, I think, not necessarily an issue that is suddenly suddenly snowballed, but I think we're seeing some unique circumstances now that have added to it.

Speaker 9 One of them is something like search engine optimization, where Google rewards pages, or at least people think it rewards pages that regularly refresh or that...

Speaker 9 seem like they are providing new information.

Speaker 9 And so, for instance, CNET, which is a really venerable tech publication, removed a bunch of its older articles because it wanted to appear in Google search results more highly.

Speaker 9 And so there was this sense that, okay, it makes people more likely to find current articles, but also just this trove of information disappears. Right.

Speaker 1 I mean, I think we can all, you know, mourn the loss of like our GeoCities homepage from 2003.

Speaker 1 But it's a lot rougher when like, I don't know, some billionaire buys out an

Speaker 1 alternative newspaper newspaper and just decides one day to shut down its website.

Speaker 9 Aaron Ross Powell, sometimes it's a billionaire that buys something and shuts it down.

Speaker 9 There are also just more insidious phenomena that I think really kind of speak to the commercialization of the internet and the sort of cannibalization of anything that can be turned toward profit.

Speaker 9 So you have old websites that say have a name people recognize and then they get resurrected, but they no longer have the old information.

Speaker 9 They've been filled with AI-generated new articles that can sort of capitalize on this old name as this zombie site. Or you have issues where there's a link that goes dead, and somebody

Speaker 9 tries to kind of hijack that link.

Speaker 9 And they either contact the website administrator or they find some other way to get that to point to a new page that will then build their own credibility but doesn't provide the original information.

Speaker 9 So there are all these cases where archival gives way to profit.

Speaker 9 That information was useful sometimes because it provided, say, statistics or it provided evidence.

Speaker 9 If you're, say, looking at Wikipedia and there's a dead link that no longer provides the information it used to.

Speaker 9 And sometimes just because these things are a valuable record of what the internet used to be and of how people lived, there are a lot of things that at one point would have been written down on paper or in some other medium that's just a hard document and people can look look back on it.

Speaker 9 But at this point, a huge amount of our culture takes place on the internet, and the internet is a very fragile place.

Speaker 1 Addie Robertson, reader at the verge.com. When Today Explained returns, we're heading into the Wayback Machine to hear from the people trying to archive the entire internet one web page at a time.

Speaker 16 support for Today Explain comes from Indeed. Indeed, says if you're looking to hire, you can give your job posting the best chance to be seen with Indeed-sponsored jobs.

Speaker 16 With sponsored jobs, Indeed boosts your posts so that it is more visible. Their data shows it's effective.

Speaker 16 According to Indeed, sponsored jobs posted directly on Indeed are 90% more likely to report a higher than those non-sponsored jobs because you reach a bigger pool of people.

Speaker 16 1.6 million companies sponsor jobs with Indeed. You can spend more time interviewing candidates, less stress, less time, more results now with Indeed sponsored jobs.

Speaker 16 Listeners of this show get a $75 sponsored job credit. To help your job get the premium status it deserves at Indeed.com slash today explained, indeed.com slash today explained.

Speaker 16 Support our show by saying you've heard about Indeed on this podcast, TodayExplained, Indeed.com slash Today Explained. Terms and conditions do apply, guys.
Hiring. Do it the right way with Indeed.

Speaker 16 Support for Today Explained comes from ATT. There's nothing worse than needing to make a call and realizing you can't connect, says ATT.

Speaker 16 And of course, every wireless provider will claim that they're the best, but AT ⁇ T says ATT has the goods to back it up.

Speaker 16 According to Root Metrics, AT ⁇ T earned the best overall network performance.

Speaker 16 While the other guys are busy making claims they can't keep, AT ⁇ T says they're making connections on America's fastest and most reliable wireless network.

Speaker 16 No matter if you're at a concert, a huge sporting event, or just out enjoying nature, you can post when you want to post. Don't post when you're enjoying nature, guys.
Keep it in control.

Speaker 16 Call when you want to call and rest easy knowing that no matter where you go, AT ⁇ T has got you covered. When you compare, there's no comparison.
AT ⁇ T.

Speaker 16 Based on Root Metrics United States Root Score Report 1H2025 tested with best commercially available smartphones, smartphones on three national mobile networks across all available network types, your experiences may vary.

Speaker 16 Root Metrics rankings are not an endorsement of AT ⁇ T.

Speaker 16 Support for Today Explained comes from Chime. What's QIIME? QIIME is different.
Chime is a financial technology company that wants you to embrace each and every dollar.

Speaker 16 When you set up direct deposit with QIIME, you can get access to fee-free features like overdraft protection, or they say you can get paid up to two days early and even more.

Speaker 16 Speaking of no fees, QIIME says that when you open a checking account with them, there are no monthly fees and no maintenance fees.

Speaker 16 And with qualifying direct deposits, you can be eligible for free overdraft up to $200 on debit card purchases and cash withdrawals. Not to mention, although I will, 47,000 fee-free ATMs.

Speaker 16 You can work on your financial goals through Chime Today. You can open an account in two minutes at chime.com/slash explain.
That's chime.com/slash explain. Chime feels like progress.

Speaker 15 Chime is a financial technology company, not a bank, banking services and debit card provided by the Bankor Bank NA or Stripe Bank NA, members of the IC.

Speaker 15 Spot me eligibility requirements and overdraft limits apply. Timing depends on submission payment file.

Speaker 15 Fees apply at out of network ATMs, bank ranking, and number of ATMs, according to US News and World Report 2023. Chime checking account required.

Speaker 6 This is Today Explained.

Speaker 1 Explained.

Speaker 1 So let's just have you start by saying your name and what it is you do.

Speaker 6 Sure. Yeah.
Hi.

Speaker 1 My name is Mark Graham, and I am the director of the Wayback Machine at the Internet Archive, which is a not-for-profit that has been preserving the web since 1996. Journalists use it all the time.

Speaker 1 But for the uninitiated, I asked Mark to show us around the Internet Archive.

Speaker 6 Where do I begin? It's like walking into a very large library and saying, show me your favorite book. Well, for example, last year it was a big news story that MTV News was shut down.

Speaker 6 And the founding editor of MT News wrote about it on LinkedIn. And there was a lot of other editors talking about it.
It was like, oh my God, all of our articles are gone. They're missing.

Speaker 6 And I just casually

Speaker 6 waded into the conversation and go, hi,

Speaker 6 check here, Wayback Machine. And they were like, oh, my God, you guys

Speaker 6 got it it all pretty much. Yeah.
And they said, well, you know, people say, well, what did you do?

Speaker 6 What did you do when it went down? You must have. I say, we didn't do anything when it went down because we've been doing our job all along.

Speaker 6 We've been working to archive the public web as it's published on an ongoing and continuous basis.

Speaker 6 So

Speaker 6 if we have to start paying attention to something after it's gone down, that means we screwed up.

Speaker 1 So with that example, with MTV News, give us a sense of what you guys were doing in advance of that website going down to make sure that people could find out, you know, I don't know, what Everlast was singing about in 2004.

Speaker 19 Hello, Jancy Dunn here, and joining me now is former House of Pain member Everlast. Welcome, Everlast.

Speaker 3 Thank you.

Speaker 6 So for any one of a number of thousands of reasons, we set our web crawlers and archiving software out on a mission every day to identify and to download web pages and related web-based resources.

Speaker 6 We bring in millions and millions of URLs every day that are signals to us, signals of where new material is being published on the web.

Speaker 6 And we make sure that we archive all of those URLs, all the web pages associated with those URLs. And then we look at those pages and we identify links to other pages.

Speaker 6 And then we go to those pages and we archive them, etc., etc., etc. That's where you get this metaphor of crawling like a spider throughout this web.

Speaker 6 And the net result of it is that we add more than a billion archived URLs to the Wayback Machine every day.

Speaker 6 And this material, as it's added to the Wayback Machine, is indexed and it's immediately available to people who go to web.archive.org, enter in a URL, and then are able to see a history of archives that we have of the web page that was available from the URL in any given time.

Speaker 1 I want to talk about government websites now because that's sort of the reason we're having this conversation today.

Speaker 1 I think most people probably think the government will take care of archiving government websites, but here we are in a new administration and websites are disappearing, coming back online, and people are worried.

Speaker 1 When you,

Speaker 1 you know, an archivist of the internet, see government websites disappearing, coming back online, becoming unreliable,

Speaker 1 how do you react to that? Is that like better or worse than regular websites that are non-governmental going offline?

Speaker 6 Well, as an American, my tax dollars help pay for for some of this stuff, and then much of it is maybe of benefit to people. Certainly, my first reaction is, hmm, that might not be such a good thing.

Speaker 6 I do want to underscore that there is the National Archives and Records Administration that does do archiving as well.

Speaker 6 But for whatever reason, we seem to be like one of the main players in the space of trying to archive much of the public web, including, and right now especially, U.S.

Speaker 6 government websites, and making those archives available in near real time.

Speaker 1 Were you caught off guard when you saw the new administration removing web pages, removing websites?

Speaker 6 Aaron Powell, this is pretty normal in some respects.

Speaker 6 It's normal and expected, and it's what's happened, frankly, for each administration in the time that we've been working on this effort. I mean, look,

Speaker 6 it's under new management, right? For example, you wouldn't expect the whitehouse.gov website under

Speaker 6 any new presidential administration to be the same as it was before.

Speaker 6 So we go out of our way to try to anticipate the frequency in which web pages should be archived so that we got a pretty good shot at getting those changes.

Speaker 1 You're saying the whitehouse.gov site obviously changes administration, administration.

Speaker 1 I think to some degree people understand that, that Joe Biden's administration probably wouldn't have been posting trolly Valentines about immigration, you know, a year ago this time to their Instagram account.

Speaker 1 But what we're seeing here is

Speaker 1 websites that people need, websites that record public health information going offline, briefly, permanently, what have you.

Speaker 6 No, that's true.

Speaker 1 Is that a different degree of sort of erasing the historical historical record or messing with the historical record than we've seen?

Speaker 6 It's different.

Speaker 6 It's certainly different in terms of the number,

Speaker 6 seemingly. I mean, we're still in the early stages of this administration.
But yeah, I'd say on the face of it, you're right.

Speaker 6 Historically, we haven't seen major U.S. government websites taken offline like we did, say, for example, with regard to USAID.

Speaker 6 And I'm going to leave that kind of analysis to others and

Speaker 6 really just focus on trying to archive the material.

Speaker 1 The Wayback Machine, the Internet Archive, mostly funded through donations, the generosity of people, institutions, even governments.

Speaker 1 Is that going to be enough to archive the internet to the extent that future generations will want to see and need.

Speaker 6 Enough is a very subjective term.

Speaker 6 Well, as an archivist,

Speaker 6 for me, it's never enough

Speaker 6 because

Speaker 6 you don't know. No one knows what is going to be of use, value, importance in the future.
Maybe even the near future of tomorrow, much less like the very far-off future.

Speaker 6 And since millions of people use our site on a daily basis, we get a lot of feedback from them.

Speaker 6 It motivates us, but it also helps direct us and inspires us to continuously try to do a better job at being the best library that we can be.

Speaker 1 Godspeed.

Speaker 6 There you have it.

Speaker 1 Let me ask you one last question, Mark. You guys have been at this for nearly three decades.
Certainly you've saved a lot of stuff and certainly a lot of stuff has fallen through the cracks.

Speaker 1 I wonder, is there something that slipped through the cracks that you could tell us about that might suggest to our audience, you know, what is lost when we can't archive to the extent we want to or need to?

Speaker 6 You know, okay, so it kind of caught me up with that question. I'll just say,

Speaker 6 I don't know right now. I can't say that thing.
Gosh, I wish. Well, okay, I got one.
I mean, this is just in recent history. Apparently, there was a page up on the CDC website about bird flu

Speaker 6 last week. It apparently was only up for a few minutes and no one got it.

Speaker 1 Huh. And by losing that fleeting web page, that one, you know, maybe minor, maybe major web page about bird flu on the CDC's website, what are we losing?

Speaker 6 Well, we're losing part of the story, right? We're losing part of our understanding of the evolution of arguably a significant health issue concern. We don't know where this is going to go.

Speaker 6 I don't know. I guess that's the other point, right?

Speaker 6 I mean, you don't know necessarily now that which is going to be very important in the near or longer term.

Speaker 6 In the time of Martin Luther, there was

Speaker 6 raging debates. And much of that debate took the form of

Speaker 6 things that were written on pamphlets.

Speaker 6 The pamphlets at the time were considered

Speaker 6 of little value. I mean, people read them and they shared them, but they didn't necessarily save them.

Speaker 12 So today,

Speaker 6 a scholar of that time or someone like me was just kind of strangely curious,

Speaker 6 what I would give for a collection of those pamphlets. Yeah.

Speaker 9 I mean, and you are

Speaker 1 comparing, in a way, a CDC website to the Protestant Reformation, but I think you mean it, don't you?

Speaker 6 I do, because I don't know. And one really can't know without the benefit of the long historical view.
And

Speaker 6 that's not something that we have access to today. Why? Because we don't have a real time machine.

Speaker 1 Mark Graham, known exclusively to Amanda Llewellyn as Webb MG. Check out the Wayback Machine at web.archive.org.
Amanda produced the show today. Laura Bullard helped and wore the hat.

Speaker 1 Jolie Myers edited. Andrea Kristen's daughter and Patrick Boyd mixed.
And Andrea even made some original music thanks to the free state of Aftonia for the Wi-Fi.

Speaker 1 Oh, and it's today, Explain's seventh birthday today. What did you get us? Maybe show some love in the comments and the ratings and the reviews.
They say it helps.

Speaker 1 And thank you for listening for however long you've been listening. If you're new to the show, feel free to browse the archive.

Speaker 7 Support for this show comes from one password.

Speaker 8 If you're an IT or security pro, managing devices, identities, and applications can feel overwhelming and risky.

Speaker 7 Trellica by OnePassword helps conquer SaaS sprawl and shadow IT by discovering every app your team uses, managed or not.

Speaker 5 Take the first step to better security for your team.

Speaker 7 Learn more at onepassword.com/slash podcast offer. That's onepassword.com/slash podcast offer.

Speaker 4 All lowercase.

Speaker 20 Support for this show comes from Capital One. With the Spark Cash Plus card from Capital One, you earn an unlimited 2% cash back on every purchase.

Speaker 20 Plus, no preset spending limit helps your purchasing power adapt to meet your business needs. Capital One.
What's in your wallet? Find out more at capitalOne.com/slash Spark Cash Plus.

Speaker 20 Terms and conditions apply.