A Massive Breach Reveals the Truth Behind 'Secret Desires AI'

Speaker 1 Hello, and welcome to the 404 Media podcast, where we bring you unparalleled access to hidden worlds, both online and IRL. 404 Media is a journalist-founded company and needs your support.

Speaker 1 To subscribe, go to 404media.co,

Speaker 1 as well as bonus content every single week. Subscribers also get access to additional episodes where we respond to their best comments.
Gain access to that content at 404media.co.

Speaker 1 I'm your host, Joseph, and with me are 404 Media co-founders Sam Cole.

Speaker 2 Hello.

Speaker 1 And Jason Kebler.

Speaker 3 What's up?

Speaker 1 As we're talking, I'm getting notifications that Jason is telling Emmanuel I don't know how to zoom in on my new camera, but maybe we'll save that for another time.

Speaker 1 Yeah, okay.

Speaker 3 Jose has been setting up his camera for like three weeks now.

Speaker 1 I do it bit by bit as in I'm slowly improving the podcast settings.

Speaker 3 He's reading the instruction manual on how to turn camera on.

Speaker 1 Well, we got that far. It looks really good.
This is not it. This is not it.

Speaker 1 This is not it. No,

Speaker 2 the audio is new.

Speaker 1 This is just my standard webcam.

Speaker 2 You're just looking sharp today.

Speaker 3 He was showing me the inside of his nose moments ago.

Speaker 2 Oh, well, I missed that.

Speaker 1 Well, that's the pivot. We're pivoting.

Speaker 2 Subscribers' content is

Speaker 2 inside of Joe's nose.

Speaker 1 Exactly.

Speaker 1 One bit of housekeeping before we get to the stories. We have a reader/slash

Speaker 1 listener survey there isn't a fancy link like a bitly but I will put a link in the show notes Jason do you just want to explain why we're doing this and sort of why it's important

Speaker 3 uh yeah we are on ghost uh which we've discussed before but is uh the architecture that our website runs on and it has no cookies which is very good it's very privacy uh focused privacy centric uh which is important to us because we care about these sorts of things um and you know, we don't want to track our readers all over the internet.

Speaker 3 That said, we don't know very much, like really anything about the people who subscribe to us and what things you like, what things you don't like.

Speaker 3 So this is going to help us as we decide sort of what to do, which projects to prioritize going into 2026.

Speaker 3 We have questions about like,

Speaker 3 Do you like our email newsletter? Do you want it to be different? Do you want to see us cover different things?

Speaker 3 Would you be interested in a print product, like a magazine, things like this?

Speaker 3 And then, I mean, the other thing, and just like full transparency, to be honest,

Speaker 3 when we try to sell advertisements on this podcast and in our newsletter,

Speaker 3 the prospective advertisers are like, well, what can you tell us about your audience? And right now we're like, well, literally nothing. We have no idea.

Speaker 3 So we're asking people to sort of give us like general information about the types of jobs that you have or like the types of products that you like, things like that.

Speaker 3 So yeah, it's a voluntary survey. It would be very helpful.
It's anonymous, of course, and

Speaker 3 trying to just like get a sense of what types of people listen to this podcast and read our website and subscribe to our newsletter and that sort of thing.

Speaker 3 And then, yeah, on the ad front, it's like if you are a paying subscriber of 404 media, you will not see ads. There's no ads, including on this podcast.

Speaker 1 Yeah. So a link to that survey is in the show notes.

Speaker 1 It'll take literally a minute, I think, and we would really, really appreciate it because it would help us grow sustainably and figure out, well, where exactly do we grow and where should we invest our time and resources?

Speaker 1 So let's get to the first story of the week. This one is written by Sam.
It's a really, really crazy one.

Speaker 1 The headline is, Massive leak shows erotic chatbot users turned women's yearbook pictures into AI porn. Sam, first of all, what is this service exactly? It is called Secret Desires AI.

Speaker 1 What is it?

Speaker 2 So

Speaker 2 it's really similar to a lot of the erotic roleplay

Speaker 2 chatbot slash image generator apps that are out there right now. There are a ton of these.
You've probably seen seen them being advertised on Instagram or YouTube.

Speaker 2 They're pretty popular, all of them. They all have a ton of users.
This one is really similar to like character AI.

Speaker 2 Chubb AI is another one. It's similar to the meta chatbots.
If you recall those from when I wrote about

Speaker 2 like the therapy chatbots that were on meta. Basically, you can go on there and say, I want to customize my own chat bot.

Speaker 2 Usually, I mean, it's called Secret Desires with like a heart. So it's the main thing that they do is like quote unquote spicy AI chatting is what the website says.

Speaker 2 The site says, build your perfect AI partner, customize their looks and personality to bring your fantasies to life. And it's like, it's,

Speaker 2 I would say, 99.9%

Speaker 2 erotic or like sexualized.

Speaker 1 It's a sex chat ball.

Speaker 2 Yeah. I mean, if you go through the homepage, you can sort it by like 18 plus.
It's, it's all like porn categories, basically. It's like you can choose like by

Speaker 2 ethnicity, ages,

Speaker 2 realistic, anime, both. You can search by like kink.

Speaker 2 It's just like

Speaker 2 anime girls and fantasy women are probably 80% of the homepage, I would say. And then there's like

Speaker 2 ripped men.

Speaker 2 Like

Speaker 2 eight packs are also part of the offering, which I do want to, I mean, I do want to say off the bat that women use these services, I would say, not maybe not just as much as men, but there's a, there's a big demographic of women using erotic chatbots and erotic roleplay because a lot of it's like text-based.

Speaker 2 And

Speaker 2 I think it's a lot of the same,

Speaker 2 not the same people, but like the same kind of draw as like a romantic novel, except you're interacting with the character.

Speaker 2 So yeah, it's not, it's not like it's just men doing this, but it is on the homepage, like mostly hypersexualized anime women.

Speaker 1 And what do you see?

Speaker 1 And again, we'll get to the leak and stuff in a minute, but when you use Secret Desires AI, so it's a text interface, you're interacting with this AI persona, but there are generated images as well.

Speaker 1 Is it like, it's all part of the same product or what is?

Speaker 2 Yeah, pretty much. So if you go through, like, the, so you can pick from the stuff on the homepage that I just described.
Um, there's like a bodybuilder lady who's like basically nude. There's like

Speaker 2 a sassy 50-year-old pharmacist who is tender and nurturing and definitely not wearing anything work appropriate.

Speaker 2 So you can pick from those or you can make your own, which I think is what a lot of people do.

Speaker 2 So in the character creation part of it, you can choose gender, you can choose style, like anime or realistic, you can do like

Speaker 2 identity. So you're doing like their

Speaker 2 like their ethnicity basically, which there is not a a lot of options to choose from there.

Speaker 2 You can do appearance, but it goes through all of this and it creates the personality of the chat bot based on what you choose.

Speaker 2 It's like you can choose like their career, their traits, their lifestyle. You can type in your own prompts, but then you get to a point where you can then generate an image of

Speaker 2 your

Speaker 2 AI partner.

Speaker 2 Yeah, you're describing it. You're doing prompting just like you would with like

Speaker 2 Sora Sora or Nana Banana or any of these other um salad fusion, any of those other generative AI

Speaker 2 platforms where people also use, you know, to. We've talked, Emmanuel's talked a ton about Civatai.
Um, so it's all part of the same universe.

Speaker 1 Um, and then you upload your own image as well, right?

Speaker 2 Well, not anymore. I mean, that's that's a feature that they used to have was they called it face swap, but as far as I can tell,

Speaker 2 you were uploading an image of someone

Speaker 2 and it was generate, it was generative AI. It wasn't like deep fakes where it's a picture of someone, and then you're putting them in a scenario that already exists in real life.

Speaker 2 It's generating a whole new fantasy, a whole new person, like who looks like the person that you are the target person, basically.

Speaker 1 Yeah, it's,

Speaker 1 I mean, I feel like obviously readers of 404 and listeners of the podcast will know now, but like we are a million miles away from deep fakes at this point. Like deep fakes are this quaint

Speaker 1 thing at this point where we're not talking about really face swapping. We're talking about using generative AI to make entire new scenarios and that sort of thing.

Speaker 1 Sort of like the Taylor Swift videos, which we might get to in a minute.

Speaker 2 And people still call all that like colloquially deep fakes because it's a person in a scenario that they were never in. I think is now the definition.
The definition changes every year.

Speaker 2 I've given up trying to keep up on

Speaker 2 specifically makes a deep fake, but I think most people are, when they say deep fake, what they mean is like all of the above.

Speaker 2 It's a huge umbrella term of like people generating wholesale new images, like these fantasies that people are making on Secret Desires and other platforms on Civitai, things like that.

Speaker 2 And then also

Speaker 2 the face swapping that you're talking about. So that was with like generative adversarial networks, which is a different sort of

Speaker 2 family of the same

Speaker 2 product or the same ELP, I guess. Without getting too technical about it,

Speaker 2 that's what we're dealing with.

Speaker 1 So that all exists. Secredesires.ai offers that.
You then get this tip, I think, via email. What exactly was this tip? What did it say?

Speaker 2 So, I mean, there are websites out there that track open, like exposed containers.

Speaker 2 And by a container, I mean this is something that like it's it's storage, it's cloud storage being used by a company or by a even an individual, but usually it's companies because this stuff is expensive.

Speaker 2 And then they're drawing from that container to

Speaker 2 the inputs go in, they draw out of it. It's it's cloud storage.

Speaker 1 AWS buckets, that sort of thing.

Speaker 2 Yeah, exactly. And so this specific container was Microsoft's Desert.

Speaker 2 blob storage, which is such a funny name every time I come across it. That is what it is.
It's a blob of data.

Speaker 2 And then the blobs are, they're different blobs and different containers of like related data.

Speaker 2 So someone emailed me and was like, Hey, I came across this on one of those sort of websites that tracks these things.

Speaker 2 He was like, This seems like a big deal because it's,

Speaker 2 well, first of all, it was like 1.8 million images.

Speaker 2 A lot of those are duplicates, but it's individual images, 1.8 million.

Speaker 2 And a lot of them were pictures of people, real people,

Speaker 2 generated like AI

Speaker 2 characters and companions, all coming from secret desires that was there. The container belonged to them.

Speaker 2 So this person flagged it to me and was like, this is something that you cover. This is probably

Speaker 2 not good or at least interesting. I don't think, I even think they said it not good.
And I was like, yeah, that's super interesting. This is like on a Saturday or something.

Speaker 2 I was like, yeah, let me spend like

Speaker 2 the afternoon. dragging these images out of this container to see what's actually in here.

Speaker 1 Yeah. And just to clarify, again, this is publicly exposed information, like it shouldn't be, obviously, but it's basically been put in a database which is publicly accessible from normal internet.

Speaker 1 So, you know, journalistically,

Speaker 1 we are able to see the information and use it. And we've done tons of stories about exposed buckets over the years, not just here, but you know, way back at Vice and Motherboard as well.

Speaker 1 So you go in and you get these images, and it sounds like they're split into two sets, which is sort of the input where people are taking images of people and putting them in and then the output, like this AI generated non-consensual

Speaker 1 intimate imagery or however you wish to describe it, the output. First on the input images, what

Speaker 1 was in those? Is it like porn stars, celebrities, normal people? Like what were you seeing in there?

Speaker 2 So, yeah, and also just to your note that it is, it's public.

Speaker 2 If you were responsibly storing this data, if you were responsibly using Microsoft's blob storage, you would password protect it, you would encrypt it, you would be doing things to make sure people can't just access it from the open web, which is something Secret Desires obviously did not do, which is why I was able to just click on a link that took me straight to like a file full of this stuff.

Speaker 2 So

Speaker 2 what was in it? I mean, so yeah, there were, there were, I guess that's, that's true about there are two different sorts of images and there's the input and then the output.

Speaker 2 That's kind of the broad categories. But then there was also like specifically named containers.
So one of the containers was named face swap.

Speaker 2 And that's where

Speaker 2 a lot of the images that were kind of the crux of the story were being held. So a lot of those were just images of

Speaker 2 from what I could tell, completely like random people.

Speaker 2 People would know like internet internet and social media footprint.

Speaker 2 A lot of them are old images, like images from clearly from like 10 years ago, 20 years ago, even because they were like taken with flip phones, flip phones in mirrors or with like a Blackberry with the sliding keyboard and they looked like deep-fried, low-resolution, like MySpace picture.

Speaker 1 So somebody's gone and got MySpace pictures and fed them into this AI.

Speaker 2 Yeah. And that's the name of the container is why we,

Speaker 2 how we figured that out, because they're called FaceSwap. FaceSwap is a tool that Secret Desires used to have where you could upload these pictures of someone, anyone,

Speaker 2 and then

Speaker 2 it would, I assume, this is how it works, just

Speaker 2 based on what we saw in the container.

Speaker 1 Yeah, how this stuff works, yeah.

Speaker 2 People would then upload the images. It would go to the face swap container, which was a public again, exposed.
And then that's where they were stored.

Speaker 2 And then then Secret Desires was using its generative AI algorithm to turn those into like whatever fantasy prompting that the user described.

Speaker 2 So if they, if someone was like, I want to see my this, and this is something they advertise publicly as of like recently.

Speaker 2 If you want to see like this girl that I saw at the gym,

Speaker 2 that's a fantasy that you can

Speaker 2 fulfill by

Speaker 2 used to be by uploading a picture of this girl at the gym. Maybe you know her like Instagram or like you took a picture of her without her knowing or whatever.

Speaker 2 Maybe she sent you a picture.

Speaker 2 Doing, you know, whatever you want. So like, I think we can infer based on the

Speaker 2 other containers, which were full of porn, AI generated porn, that people were making porn out of these people's images. So yeah, it was like regular people.

Speaker 2 It was also a ton of influencers, a ton of just like screenshots of

Speaker 2 influencers and also like sex workers and like porn influencers and adult content creators, normal, regular, like non-adult, you know, not safe for work influencers, celebrities.

Speaker 2 There was one, probably the one that like really disturbed me the most was, and this is all, it was all pretty disturbing just to go through like these people's images who had no idea clearly that they were in this.

Speaker 2 But

Speaker 2 one of them was like a local

Speaker 2 representative, like a like a lower level politician who was she was the image was from

Speaker 2 where she was speaking at like some

Speaker 2 kind of like town council sort of event, city council thing. Um, and someone uploaded that, and then you could go to the other container, which said, like,

Speaker 2 what were the container names specifically? Anyway, they were, I think they were called like live photo or like um

Speaker 2 character creation or something. But you could go to the next one and see where their faces looked a lot like the people who were in the AI porn.
So I could, I found one of her, looked just like her

Speaker 2 in like a really crazy, like non,

Speaker 2 not humanly possible type form.

Speaker 2 So yeah, it's like,

Speaker 2 and someone actually did end up telling me after this story

Speaker 2 came out that

Speaker 2 they were a tester for the face swap tool. And they said that Secret Desires never let people upload those images from FaceSwap to the public feed.

Speaker 2 But you could make them, you could put them wherever you want.

Speaker 2 Like, just because you can't put them in the same ecosystem in the feed doesn't mean you can't just share them in many other forms that these

Speaker 2 people also share things too. So it's a huge ecosystem.
It doesn't just stop at like, you can't upload it to the same website.

Speaker 1 That point on public, I think, is like the really interesting bit about this piece, which is that, of course,

Speaker 1 we see AI-generated porn of people either on tube sites or it will happen with the Taylor Swift

Speaker 1 videos that go viral on X or whatever. But there are people doing this by themselves, basically, using these tools.

Speaker 1 And of course, those are supposed to be opaque systems where I'm going to go to this website, I'm going to use this app, I'm going to upload imagery, I'm going to make AI porn of this person without them knowing.

Speaker 1 And maybe they don't distribute it, maybe they don't send it to the person, they just keep it to themselves on the hard drive. And of course, nobody gets visibility into that.

Speaker 1 I don't think a lot of these people would are going to want, obviously, visibility into that, even though they're doing this to these other people.

Speaker 1 And this provides a snapshot of, oh, holy shit, this is what people are doing. And yes, there were a lot of responses going, Are you really surprised? Like, no, of course not.

Speaker 2 But we're doing this.

Speaker 1 So we found an actual use case of this happening. And now there is a really significant data point

Speaker 1 to show that.

Speaker 1 Just very, very briefly, before we move on to the next bit, the headline does mention yearbook photos.

Speaker 1 And I think Emmanuel came up with the headline.

Speaker 1 I think it was very, very smart to include yearbook because people instantly go, oh shit, that's a really horrible recontextualization of a photo or a piece of data.

Speaker 1 How did you figure out they were yearbook photos?

Speaker 2 I mean, they were just, they were literally like pictures you would see in a yearbook. It was someone like graduating.
They like had like class of 2000, I think it was like four or five or something.

Speaker 2 Like they were like

Speaker 2 holding their little cap and like it was like the life touch background.

Speaker 2 I don't know how to describe that, but like blue, speckly.

Speaker 1 I think people can visualize it.

Speaker 2 Yeah, it's like you look like you know what a yearbook. And also, I don't know if people know that like most, I think a lot of,

Speaker 2 especially public school yearbooks are publicly available. Right.

Speaker 2 That's all data that's out there anyway, but this is like a specific person that obviously had access to someone's like

Speaker 2 yearbook picture. So maybe it was their classmate, maybe it was just like some random person.

Speaker 2 But yeah, and then there were others that were like, there was a woman standing in front, she was like graduating from university and she was like standing in front of like

Speaker 2 the sign for the school, um, just like a very normal, like wholesome picture. And someone took it and turned it into porn.

Speaker 2 Um, yeah, I don't know.

Speaker 1 Let's let's not get too

Speaker 1 specific on the porn that was made, but I think, can you give people a very general sense of

Speaker 1 how explicit or hardcore was this pornography?

Speaker 1 Yeah,

Speaker 2 um,

Speaker 1 yeah, I mean,

Speaker 2 so and we've written about this before, um,

Speaker 2 and

Speaker 2 it's something that like is not new in AI, but with generative AI, especially, so we're talking again about the difference between like old-fashioned deep fakes, where you're putting someone into a porn scene that already exists, which is already shitty and sucks because you're stealing someone else's content and also someone's face.

Speaker 2 But now it's like you can create scenarios and

Speaker 2 have people do things that just don't exist in real life.

Speaker 2 Um,

Speaker 2 a lot of it was like pretty,

Speaker 2 I don't want to say violent, but like it was pretty violent because it was like not real people. It's like no one in this is like the people are not real.

Speaker 2 And also, the person whose face it is is not consenting to like

Speaker 2 this being out there of them.

Speaker 2 There were quite a few very young generations, like generative AI of like very young people.

Speaker 1 Or what looks like a young person.

Speaker 2 Like clearly the prompt, and a few of these prompts were, you could see the prompts in some of the file names, but people were prompting for like 17-year-old looks like this and that celebrity.

Speaker 2 And it's like, that's a minor, but

Speaker 2 like, I'm not even talking about 17. I'm talking about like child.

Speaker 1 It felt like a real mix because sometimes, like, with Emmanuel's civitai coverage, that has been specifically focused on child sexual abuse imagery, CSAM.

Speaker 1 And then this, it sounds like, had both potentially CSAM, but then also definitely adults. And it was like everything.

Speaker 1 And I was kind of taken aback by how explicit some of the images were.

Speaker 2 Right. And AI-generated child sexual abuse material in many states is treated the same.

Speaker 2 So generating

Speaker 2 porn, like a sexual abuse imagery of a child with AI doesn't mean it's not a crime in many places. And experts have talked about how this is

Speaker 2 not only

Speaker 2 not helpful to like actual investigations and like tracking down the people making this stuff, but also it actually damages investigations that are trying to find real

Speaker 2 like abuse victims out there. So, I don't know.
It's like that, it opens up a whole other can of worms about the conversation about AI and CSAM.

Speaker 2 But yeah, I mean, it's as far as the hardcore porn stuff, it was just like crazy, crazy shit. I don't even know what that is.

Speaker 3 I would say the stuff that I saw when editing this piece was extreme, like extreme, almost all of it.

Speaker 3 And like, I don't know, like, gangbang stuff, like, I don't know, really just like quite extreme, which,

Speaker 3 you know, is, is in the piece, but this is not like a standard Nunify app where it's like taking the clothes off of a shirt, off of someone who's wearing clothes.

Speaker 3 Um, it is like for hardcore porn primarily, um, at least what, what we saw.

Speaker 1 Yeah, and Jason, I was going to bring you in now because you did edit it, and I feel like

Speaker 1 when Sam got the tip, I think you were the first to respond saying something like, Well, this is insane and wild. What did you think of this while you were editing it?

Speaker 1 Like, what's new about this story?

Speaker 3 I mean, we knew that this was the case, but it is another example of the fact that these tools are being used primarily to non-consensually put women into porn and sexual situations.

Speaker 3 And it was not okay when it was deep fakes uh for either the porn performer whose body was being used or for the celebrities who were having their faces swapped onto it um

Speaker 3 it was not okay then but i think it was it's like pretty alarming to see that quite clearly what's happening here is just like random people are being put into these uh situations where they're definitely not public figures and they're definitely not um you know i don't know it's just like it's it's pretty gross to see and I think that we've done a lot of reporting where it's like, this is what these tools are being used for.

Speaker 3 But I think to see the scale at which this occurred and the explicitness and all of that was like quite jarring.

Speaker 3 And then, yeah, again, it's like, this is not even a big one. Like, this is like not.

Speaker 3 I had not heard of this app before. And the fact that there's more than a million images on this app is like, well,

Speaker 3 there's many, many apps like this. And I guess it's just like,

Speaker 3 I hate to be hyperbolic, but it's one of those moments that makes you like lose a little bit of faith in humanity. I think it's just like walking around.
It's like, people are doing this.

Speaker 3 Like people are doing this. That's not good.
And we're not really doing anything to stop it.

Speaker 1 And it could be you or your partner or a friend or something, because apparently people are just harvesting photos that

Speaker 1 they're harvesting any photos they can get a hold of essentially is what it looks like um i guess just to wrap up this section sam what was your takeaway for me it was sort of the scale and it was being done to ordinary people like what was your takeaway after doing this

Speaker 2 um

Speaker 2 i mean not to like keep belaboring the point but like it's it's just something that like we we know people have been doing with the technology because first of all, it's wildly popular.

Speaker 2 We had, you know, we know that erotic roleplay is hugely popular and we know that generating real people is massively popular.

Speaker 2 And we know this from like lots of other reporting that we've done on this. And we know it from like listening to and like talking to the users of these tools.

Speaker 2 Like they say it out loud all the time that this is what they're doing with it. They're in Discord talking about it and making these generations.
And it's like, that's, that's whatever. That's fine.

Speaker 2 um

Speaker 2 you know like fantasy is fantasy porn is fantasy you know i I don't like, I don't love that use of AI, but

Speaker 2 go nuts, it probably shouldn't be illegal to make fantasy AI porn.

Speaker 2 But I think with the face-wapping stuff,

Speaker 2 I think it was really, like you said, very telling that this is like a huge part of, that was a huge part of the site. It was also very telling

Speaker 2 that they took it down and that it was not a tool on the site for very long.

Speaker 2 It went down in, I think, April, and a lot of people were mad. A lot of people were like, I'm canceling my subscription.

Speaker 2 People were posting on Reddit talking about how they don't want to use Secret Desires anymore.

Speaker 2 And they were like, oh, it was my favorite tool, but now I've got to go somewhere else because they don't let me do face swap anymore.

Speaker 2 So, like, there are a lot of things going on there. It's like, first of all, people really want this feature.

Speaker 2 It's in demand. And also, it was in demand, and the company still took it down, which is interesting to me because usually these companies are only driven by

Speaker 1 profit. What did they find? What did they decide?

Speaker 2 what did they decide like what um

Speaker 2 the take it down act went into um effect earlier this year so like maybe they were like oh we have to like preemptively make sure no one's violating federal law um

Speaker 2 so i don't know it's like the company hasn't replied and said anything which shut down the buckets though they closed them they shut down the buckets immediately like in less than an hour they like read my email and then shut down the buckets

Speaker 2 which I sent them all the links because I was like, you need to shut this down because it's violating these people's privacy. Or, you know, I was like,

Speaker 1 you can't cover it. Yeah, you can't cover it before it gets shut.

Speaker 2 Yeah, we can't like highlight something that's like an exposed. It's, this is something that's like, it's with any kind of like security or like breach reporting.

Speaker 2 It's like, you want the company to close the breach before you cover it because you don't want to draw more attention to it. So

Speaker 2 they shut it down, but like they didn't reply to me. And I would think it would be very easy to say, like, we found out that like it wasn't, we weren't into it.
It's not what we're about.

Speaker 2 We don't like that use of it. And we found out people were making like,

Speaker 2 but it is what they're about because that's all their ads are like

Speaker 2 AI women can't say no. You can generate the lady at your gym.
Um, that is what they're very much about. So, I'm very curious.
It's kind of like an open question about why they took it down.

Speaker 2 That's my takeaway is like, uh,

Speaker 2 what's going on there? So, I don't know. I mean, maybe we'll find out down the line.
Maybe someone will come forward and say, I worked in Sierra Desires and

Speaker 2 I told them to take it down or something. But

Speaker 1 yeah,

Speaker 1 if you're listening and you do know more, of course, please reach out to Sam. We'll leave that one there.

Speaker 1 When we come back after the break, we're going to talk about one of Jason's pieces about some very funny shit that happened on X, but you know, there's much broader implications as well.

Speaker 1 We'll be right back after this.

Speaker 3 Are you always searching for that one hoodie you'll actually live in? The one that feels just right on the couch, on a plane, or out on a hike?

Speaker 3 I found mine with PACA, and now it's the only one I reach for. PACA makes performance apparel from Alpaca Fiber, one of the world's most sustainable natural fibers.

Speaker 3 Their best-selling hoodie is softer than cashmere, warmer than wool, and breathable. This hoodie is built for real life.
Thermoregulating, odor-resistant, durable, and made to last.

Speaker 3 It's lightweight but still cozy, doesn't stretch out, doesn't pill, and somehow keeps me warm when it's cold and cool when it's hot. Basically, it adapts to wherever life takes you.

Speaker 3 Over 100,000 people have already picked up the pack of hoodie. What makes it even better?

Speaker 3 Knowing it's made sustainably and ethically from traceable alpaca fiber while supporting the communities and artisans in Peru who bring it to life. Little known fact, it actually gets cold here in LA.

Speaker 3 Most nights, actually. My pack of hoodie has become my go-to around the house at happy hour drinks and running errands.

Speaker 3 The only problem is I have to keep telling people where I got such a great hoodie from. If you've been thinking about leveling up your hoodie game, this is your sign to do it now.

Speaker 3 And right now, during PACA's Black Friday/slash Cyber Monday sale, you can get 30% off with the purchase of three items or more. Head to go.packaapparel.com slash 404 media.

Speaker 3 That's go.paka apparel.com/slash 404 media to get 30% off with the purchase of three

Speaker 3 With Aura Frames, you can relive your holiday traditions every day. We've all got them, those weird family traditions that make the holiday feel magic.

Speaker 3 For my sister and me, it really isn't Christmas until we put our childhood ornaments on the tree. We've got a lot of great pictures at that moment, and Aura Frames is the perfect way to show them off.

Speaker 3 Aura Frames is a digital picture frame you can load with unlimited free photos and video.

Speaker 3 You can preload it with photos before it ships, so you're ready to relive your holiday traditions the moment it's out of the box on Christmas morning.

Speaker 3 And the moment my sister and I put our ornaments on the tree this year, we can grab a picture and send it right to the frame from our phone. That way, when our mom opens it up, she'll remember.

Speaker 3 Memory keeps these traditions alive, and Aura Frames is the perfect place to store those memories. You can't wrap togetherness, but you can frame it.

Speaker 3 For a limited time, visit auraframes.com and get $45 off Aura's best-selling Carver Matte Frames, named number one by Wirecutter, by using promo code 404Media at checkout.

Speaker 3 That's auraframes.com promo code 404media. This exclusive Black Friday slash Cyber Monday deal is their best of the year.
So order now before it ends. Support the show by mentioning us at checkout.

Speaker 3 Terms and conditions apply.

Speaker 3 I remember how shopping for glasses used to be.

Speaker 3 Stumble into a stuffy doctor's office, spend hundreds and hundreds of dollars on glasses I didn't really like, wait a month to get them, then feel dizzy because they weren't fitting right.

Speaker 3 Warby Parker changed all that. Warby Parker uses only premium materials in each frame.

Speaker 3 They design the frames in-house, and their collection includes different silhouettes, colors, and fits to suit every face and every fashion style. I love that it's a one-stop shop.

Speaker 3 They offer everything you need for happier eyes: eyeglasses, sunglasses, contact lenses, and eye exams. You can shop with them online and in stores.

Speaker 3 Warby Parker has over 300 retail locations across the U.S. and Canada.
Their glasses start at just $95 and include prescription lenses with anti-reflective, scratch-resistant coatings.

Speaker 3 Add a pair and save 15% when you purchase two or more prescription pairs of glasses or sunglasses. This offer is available both online and in stores.
Free shipping and 30-day returns.

Speaker 3 I've been going to Warby Parker for years now. I actually don't think about shopping anywhere else for glasses anymore.

Speaker 3 If you don't have time to get to the store, you'll love their virtual try-on system that lets you figure out which frames and what size you need.

Speaker 3 Gone are the days where you'll be disappointed in your glasses. I've been lucky because it always feels like I've lived around the corner from one of Warby Parker's many convenient locations.

Speaker 3 Warby Parker has over 300 locations to help you find your next pair of glasses. You can also head over to warbyparker.com/slash 404media right now to try on any pair virtually.

Speaker 3 That's warbyparker.com/slash 404 media. Warbyparker.com/slash 404media.

Speaker 1 All right, and we are back. Jason, this is when you wrote the headline: America's polarization has become the world's side hustle.
Really good headline.

Speaker 1 Before we get into sort of the meat of your story, what was this change on X over the weekend? Because it, I mean, it was pretty funny.

Speaker 3 Yeah, so X changed,

Speaker 3 they added a feature where basically you can click into someone's profile and look at what country they're based in.

Speaker 3 I'm not sure exactly how it's doing that, like whether it's IP address, whether it's like where it was registered, things like that.

Speaker 3 There's been actually some beef about this because I believe the Department of Homeland Security's... Twitter page was showing that they were based in Israel,

Speaker 1 which

Speaker 3 I don't know if that's real or not. It's like hard to tell what's real.

Speaker 1 And there's also like Hank Green's profile, the famous YouTuber. It says his was in Japan and like he's never been to Japan.

Speaker 1 Some of them look accurate. I presume it's IP and the registration one's easy enough.
Like when they made the account, where were they located? Like that's straightforward.

Speaker 1 Right.

Speaker 3 So anyways, they added this feature and then people started looking at different really popular mega accounts. So there's one called Ivanka News that had over a million followers.

Speaker 3 And that one was based in Nigeria. There's one called Red Pill Nurse that was from Eastern Europe, I believe.
Something called MAGA Nadine, which is in Morocco.

Speaker 3 An account called Native American Soul that was in Bangladesh. And there was many, many, many.
It's like the entirety of my timeline over the weekend was about this.

Speaker 3 And I don't know, I don't even go on X very much at all anymore, but this had like broken containment. I saw it, saw people in Blue Sky talking about it.
I saw people on Threads talking about it.

Speaker 3 And then, so I went and looked. And

Speaker 3 there's been like kind of a bit of meltdowns on both sides of the political spectrum here, being like, well,

Speaker 3 everything is just a psyop and every like, you know, all these accounts are disinformation campaigns, so on and so forth.

Speaker 1 And what do they mean by that? As in, just to hear their arguments.

Speaker 3 Well, they're just saying that like a lot of these accounts are like, oh, like I'm a I'm a single white mother from Oklahoma and I hate woke, like the woke left and things like that.

Speaker 3 And that's like all they tweet about. And then you check, click into their profile and it's like, oh, they're from Bangladesh or whatever.

Speaker 3 And so people are saying, well, this is either foreign governments attempting to,

Speaker 3 you know, like divide Americans. or it is like foreign interests, like big moneyed interests trying to do this, or it's just like troll farms or bot farms, dead internet theory.
Like all this,

Speaker 3 like that, that's sort of like where the narrative

Speaker 3 went.

Speaker 3 And I guess before we get into it, let me just say that after 2016 election on Facebook, where there was like, you know, these Russian bot armies and Cyprus fake news organizations and things like this, Facebook launched a feature that lets you see the country of origin for different pages.

Speaker 3 Now, you have to click through like three different things to get to it.

Speaker 3 But if you do that, you will see like the vast majority of, I mean, many, many of these like spam accounts that are posting like AI slop or like weird stuff say that they're based in Cambodia or Vietnam or just like other countries.

Speaker 3 And they'll say like, I'm a MAGA mom. Like that, that is sort of what they're fronting as.
And so This is not a new feature like for social media.

Speaker 3 This is something that has existed on social media for almost a a decade, first of all.

Speaker 3 Second of all, I would argue it hasn't really done anything to like better our online discourse or to like make people understand that this is

Speaker 3 that not everyone is telling the truth on the internet about who they are.

Speaker 1 You don't think the change on X may have some

Speaker 1 impact where even like, I don't know, very brain dead X users might be like, oh, actually, those big viral accounts I followed are from America. I mean, maybe, I don't know.

Speaker 3 I think it's a good feature, and I think I'm glad that the feature exists. I'm just saying it's like 10 years too late.

Speaker 3 And that YouTube has had a similar feature for a long time for channels where you can see where they're located. And it's like, these things can be gamed,

Speaker 3 you know, with VPNs and with like selling and buying accounts and things like that. But it is good.
And I think it's helpful.

Speaker 3 Like, it's helpful as a journalist so that we can go in and like look at where the country of origin is for some of these things especially when some of them are claiming to be like official accounts or you know the you know verification on Twitter has been just a thing that you buy for since Elon Musk took over but

Speaker 3 like it is a good feature

Speaker 3 it hasn't done that much on Facebook to like

Speaker 3 fix the misinformation problem. And that's largely because

Speaker 3 this stuff is decontextualized when you look at it. Like you're just scrolling your timeline and someone has put it in your feed.

Speaker 3 And like, I don't know, there's, it's only a couple clicks, but how often are people doing the few clicks to sort of get to that on someone's profile?

Speaker 3 That said, it's, it's a good feature. And I think it's led to interesting journalism.

Speaker 3 It's, it's led to interesting, uh, you know, people have sort of woken up to the fact that a lot of these accounts are just

Speaker 3 trolling for engagement and trolling for clout and that sort of thing. But

Speaker 3 I don't know that it is going to like heal our divide. And I guess I would also argue that it's like the divide goes further than a few like weird mega accounts that are promoting things.

Speaker 3 But that said, it's like it's a new feature that people are talking about on X. Yes.
And it was like quite funny to see some of these accounts and like where they're supposedly based.

Speaker 1 I mean, it caused a shitstorm, basically, where

Speaker 1 it looked like a lot of these accounts or people who followed them freaked out essentially.

Speaker 1 And just

Speaker 1 the entire weekend, again, I don't really check X that much either. But then when I heard rumblings of this, I logged in.
It's just like, wow, this is all anyone is talking about.

Speaker 1 And it's really, really, it's really, really good.

Speaker 3 Yeah, I will say before we get into like my specific piece

Speaker 3 that

Speaker 3 when Facebook did some of this stuff and when they've tried to make

Speaker 3 like try to do a little bit of transparency around this, the sort of free speech warriors have said, like, this type of transparency is censorship and that sort of thing, which is really funny.

Speaker 3 But then when Elon Musk does it, it's like, oh, he has

Speaker 3 cracking down on bots and things like that. So to see the dichotomy of how these sorts of features are received has been interesting.

Speaker 1 But, but yeah.

Speaker 1 So but you approach it from a different angle. So that all happens over the weekend.
We come back in on Monday, right?

Speaker 1 And you then write this. And as I said, you approach it from a slightly different way because you've been covering very much the money and the ecosystem, specifically on social media, behind AI slop.

Speaker 1 But not just AI slop, just almost like grifters in general now. It's just they're using a lot of AI right now.

Speaker 1 So what did you do to get this article going? It sounds like you started to search for certain things after you saw the X stuff.

Speaker 3 Yeah, so

Speaker 3 I did a series of posts about where Facebook AI slop comes from. And I did, I found a lot of YouTube tutorial videos about how to make AI slop, how to bypass

Speaker 3 Facebook's pretty shitty filters and how to monetize your account and that sort of thing.

Speaker 3 And that was about a year ago. And we talked about it on the podcast.

Speaker 3 But something that I remember from that, and something that I knew already was that there was a specific interest in targeting American users, like in getting, in making stuff that goes viral in the United States.

Speaker 3 And the reason for that is twofold. One, well, the reason for that is because

Speaker 3 advertisers pay higher ad rates to reach American users. They pay higher CPM, which is cost per millia, or milli, which is cost per 1,000 views.
It's an advertising.

Speaker 3 It's how sort of like the advertising industry does rates.

Speaker 3 And they pay the most to reach American users for two reasons.

Speaker 3 One, because the United States has a relatively wealthy user base, at least definitely higher than like Pakistan, India, Cambodia, Vietnam. It's like more valuable for people based there to

Speaker 3 make content that reaches American users versus people in their own countries.

Speaker 3 And the second reason, which is not in the article, but which Corey Doctorow texted me about afterwards and was like, you should have put this, and I should have, to be clear, is that

Speaker 3 the U.S. has like no

Speaker 3 privacy laws basically whatsoever. And so you can target Americans a lot better.

Speaker 3 You know, we don't have GDPR and things like that. So the ads are usually more effective because you're targeting people more directly.

Speaker 3 And therefore, the platforms can charge more money for them because they're more effective. So, for those two reasons, the United States is the most valuable ad market.

Speaker 3 And so, a lot of the YouTube videos were like,

Speaker 3 you need to make content that is popular with Americans. And so, when I saw this, I was like, I wonder if they're talking about this.

Speaker 3 I wonder how this ecosystem has developed since I last wrote about it. And so, I started searching for a monetized X account

Speaker 3 in Hindi. So I just used Google Translate.
And I started in Hindi because it seems like India has the biggest ecosystem of side hustler podcast bros.

Speaker 1 And that's what you found before with your Facebook stuff.

Speaker 3 It's what I found before. And I mean, this is happening all across the world.

Speaker 3 Like I found some in Cambodian. I found some

Speaker 3 in Hindi and Portuguese and Vietnamese. And then a lot in English, to be clear, like people in the United States are doing this also,

Speaker 3 like using AI to spam social media. But it's particularly popular in India right now.
And I found so many videos from the last few months about channel ideas for what they call a USA channel.

Speaker 3 And it just means like content that Americans are looking at.

Speaker 3 And so these include ideas such as make content about MMA fighting, which this one Pakistani creator said is similar to cricket in Pakistan in terms of Americans like fucking love this shit.

Speaker 3 They love MMA. So make MMA AI videos.

Speaker 3 You know, make videos about American politics. Make videos about cool cars.

Speaker 3 Make videos about pets and dogs because Americans love dogs. Things like that.
And

Speaker 1 it's not like there's AI.

Speaker 1 Yeah, exactly.

Speaker 3 It's not like there's like one or two videos like this. It's like there are dozens and dozens and dozens of videos about how to make USA content, USA channel.

Speaker 3 And a lot of them are pretty basic, but then a lot of them are relatively sophisticated where

Speaker 3 they're like, they'll start off the video being like, CNN is a popular American news source.

Speaker 3 Go to CNN.com and copy paste their articles, put them into ChatGPT, change, you know, have it write you a script, run that script through an AI voice, generate images based on that script and then uh you know use cap cut and make a video and and you can make

Speaker 3 my you can like post this content and americans will love it um

Speaker 3 there is also a lot of like black history content which is kind of wild and it's something that I've seen on YouTube a lot.

Speaker 1 And it's popular or the people doing the grifting think it's popular or both?

Speaker 3 I mean, people are trying everything.

Speaker 3 it's like it's like people are throwing at the wall and so it a lot of the tutorial videos are like go to wikipedia type in like black history or african-american experience copy paste that run it through chat gpt or perplexity or like each each per each like uh influencer is like here is the type of ai tool that you should use and some of them have promo codes and things like that so it's like it is a side hustle grift vibes going on.

Speaker 3 But that's like what they're promoting. And then they'll show you examples of the types of spam that has been created.
And so

Speaker 1 like,

Speaker 3 you know, they'll show like successful YouTube channels. And it's very interesting because we did a story a few months ago about

Speaker 3 like boring history videos on YouTube. And there a lot of these, a lot of these tutorials are like how to make boring history videos for YouTube.

Speaker 3 And so, I mean, there was also some stuff about

Speaker 3 how to monetize an X account, how to monetize a Facebook account, how to monetize a TikTok account.

Speaker 3 So, this is like the type of thing that is happening.

Speaker 3 And this is like why,

Speaker 3 in my opinion, we are seeing so much of this type of content that is

Speaker 3 like

Speaker 3 culturally off

Speaker 3 because it's not based in like, I did deep research about what is occurring in the United States.

Speaker 3 It's very often like I used Google Trends and I copy-pasted something, or I like wrote a script in Hindi and then I translated it into English, and then I use that to generate a script.

Speaker 3 And there's like something lost in translation. And there's also something lost because

Speaker 3 a lot of these people are running a lot of different channels. And so they're not

Speaker 3 or a lot of different social media accounts across different platforms. And so they're not necessarily like doing a deep dive into how I can be the most accurate.

Speaker 3 And so I think, like, my theory of the case is that, yes, there are some

Speaker 3 disinformation campaigns out there.

Speaker 3 There are some like true believers who live in other countries who are, you know, trying to either sow division or who are like actually interested in the MAGA movement.

Speaker 3 There are obviously tons and tons of Americans who hold really polarized views in either direction, who are are making a lot of content about it and are beefing online about it. But then you also have

Speaker 3 the entire rest of the world who have been incentivized by these social media companies to make content that Americans will see.

Speaker 3 And what are Americans talking about on the internet right now?

Speaker 3 Well, they're talking about politics and they're talking about the economy and they're talking about tech news and they're talking about ICE and immigration raids and all this sort of thing.

Speaker 3 So if Americans are talking about it in any way, there's going to be spam content about it.

Speaker 3 And these social media companies, like they incentivize it both through their algorithm, which incentivizes division and engagement and, you know, incentivize, frankly, like spam, a lot of you, you post enough, like you're going to go viral.

Speaker 3 And then it also has these programs where it directly pays the people who post them. And I should have said that up top, but like.

Speaker 1 Well, I was going to ask it now, but you just,

Speaker 3 yeah, it's like the, you know, X has a monetization platform where like if you enroll in it and you go viral, you get direct payments from X.

Speaker 3 And YouTube has, you know, an AdSense thing where you get a fraction of ad revenue.

Speaker 3 Facebook has a creators program where, you know, they pay out and it's not that's not directly ad revenue or like they wouldn't say that it is, but that's where it's coming from.

Speaker 3 But it's like they pay out based on views.

Speaker 3 We actually don't know the full kind of like formula for how they calculate uh but a big part of this youtube influencer economy is for them to show their uh dashboards where they show how much money they make and they they show very clearly like

Speaker 3 this video or this photo that i posted had 60 of its views come from the united states and i made a lot more money than this other one where most of the viewers were in Eastern Europe or whatever.

Speaker 3 And so, you know, they've learned this and they're like well we're going to make content that theoretically americans will care about yeah and you when you have the social media platforms introduce these monetization strategies and programs obviously that creates all of these perverse incentives where people are not posting uh for the sake of accuracy they're not even posting for the love of the game man they're posting to get their money basically they're they're posting to get their money and i mean here's here's one of the slides that i saw and the slide is in english uh which is interesting some of these videos are actually in english a lot of them are are not um but now there's a lot of really good translation tools uh but anyways uh it says youtube earnings pakistan versus usa youtube earnings in pakistan one dollar per 1000 views and then youtube earnings in the usa five dollars to seven dollars per one thousand views and so you know you're making between five and seven times as much according to this video,

Speaker 3 if you're able to reach an American audience. And so

Speaker 3 why would you cater something to a smaller audience

Speaker 3 or like an audience that these companies are not paying as much to reach when it takes the same amount of effort to spam something that is about the NBA or Major League Baseball or something that, or American politics, something that is specifically focused on American,

Speaker 3 like American audience and viewership and politics and news.

Speaker 1 Yeah.

Speaker 1 All right. We'll leave that there.
If you're listening to the free version of the podcast, I'll now play us out.

Speaker 1 But if you are a paying 404 Media subscriber, we're going to talk about how we contributed to the shutdown of a warrantless surveillance program. You can gain access to that content at 404media.co.

Speaker 1 As a reminder, 404 Media is journalist-founded and supported by subscribers. If you do wish to subscribe to 404 Media and directly support our work, please go to 404media.co.

Speaker 1 You'll get unlimited access to our articles and an ad-free version of this podcast. You'll also get to listen to the subscribers-only section where we talk about a bonus story each week.

Speaker 1 This podcast is made in partnership with Kaleidoscope. Another way to support us is by leaving a five-star rating and review for the podcast.
That stuff really does help us out.

Speaker 1 I'll read out more of those soon. This has been 404 Media.
We'll see you again next week.