Why AI Still Can't Be Trusted ...And Neither Can We

Mason Amadeus: Live from the Eighth Way Media Studios in the back rooms of the deep web, this is the FAIK Files.

Perry Carpenter: When tech gets weird, we are here to make sense of it. I'm Perry Carpenter.

Mason Amadeus: And I'm Mason Amadeus. And on this episode, we're going to start by talking about how China is going to require all AI-generated content to be labeled.

Perry Carpenter: Interesting. After that, we're going to see-can we change a large language model's opinion of us, or somebody else? Then, after that, we're going to look at really a challenging question. Is AI killing our ability to think?

Mason Amadeus: It will be hard to answer, if it has.

Perry Carpenter: Right.

Mason Amadeus: Then we're going to wrap everything up in a Dumpster Fire of the Week, talking about how AI search is-

Perry Carpenter: Well, it's an absolute dumpster fire [laughter], so, stick around for that. Sit back, relax-

Mason Amadeus: And let your algorithm turn into your palgorithm [assumed spelling]. This is the FAIK Files. [ Music ] So, it's amazing what you can get done when you rule with an authoritarian, iron fist. I am thinking back to the COVID lockdowns in China, and how they had sort of a more successful time with that, because of the fact that they have enough authoritarian rule.

Perry Carpenter: Right?

Mason Amadeus: And they're now going to try something that I don't feel like could work without an authoritarian rule, which is requiring AI labels on all AI-generated content, and it's kind of up in the air if that will even work under an authoritarian rule. Have you encountered this news, Perry? A story that came across the other day.

Perry Carpenter: Yeah, I saw news of like some fairly far-reaching and broadly encompassing Chinese regulation about AI, that was going through. Some of it looked pretty cool and progressive. And some of it looked pretty Draconian, so it will be interesting to see how that plays out.

Mason Amadeus: Yeah, it's, at its core, it's a good idea, right? Because we've talked about like, how it would be nice if we could reliably label all AI-generated content, so that if that is necessary for determining the veracity of something, or whatever, you have that mechanism. But also that practical implementation of that is pretty hard to fathom. But, the Chinese Communist Party's National Internet Censor just announced that all AI-generated content will be required to have labels that are explicitly seen or heard by its audience, and embedded in meta data, and it takes effect on September 1, 2025. I'm-a lot of what I'm going to cover in this is directly from a great article on Tom's hardware, which we'll link in the description, in the show notes.

Perry Carpenter: Nice.

Mason Amadeus: I tried to read the like text of the original law after translating it using Google Translate. It was a bit hard for me to parse. So that's why I'm leaning very heavily on this source. So the regulation takes effect September 1, 2025, and it will compel all service providers, so all AI-LOMs, any platform like that, to add explicit labels to generated and synthesized content, including all types of data, text, images, videos, audio, even virtual scenes, anything-anything that is generated by AI. Aside from that, it also orders app stores to verify whether the apps that they host are following these regulations.

Perry Carpenter: Ooh!

Mason Amadeus: So yeah, part of this whole thing is like, well, how do you enforce it, right? Whose responsibility is it?

Perry Carpenter: Yeah.

Mason Amadeus: They're also outlawing the malicious removal, tampering, forgery or concealment of those labels. So, obviously you're prohibited from removing it. But you're also prohibited from adding that identifier to non-AI content, to try and, you know, claim there's something that's real with AI.

Perry Carpenter: Yeah.

Mason Amadeus: Yeah, and then, from a statement in the CAC, translated through Bloomberg, "The labeling law will help users identify disinformation and hold service suppliers responsible for labeling their content. This is to reduce the abuse of AI-generated content." But you know, again, what is the effectiveness of labeling regulations? How enforceable is it? Meta tried to roll this out last year, with a "made with AI" label on Facebook, Instagram, and Threads. Remember that?

Perry Carpenter: Mm-hm.

Mason Amadeus: But that failed right from the start. People were-the feature was labeling real photos as being AI-generated, and stuff like that.

Perry Carpenter: Yeah.

Mason Amadeus: So, I wanted to find if there was more information about technically how they're going to enforce this. What do these watermarks, metadata look like and act? It seemed either to not be included in the text of the law, or just impossible for me to parse.

Perry Carpenter: Yeah, I think they're probably leaving some of that to the different vendors. So every vendor is going to have to figure out, like how do you embed metadata? How do you surface the watermarks, or the other things? Here in the U.S., there is a kind of a method for trying to ascribe providence to something. And there's-I forget the acronym for it, but there is a group of, a coalition of organizations that have come, and they kind of add this standard label to things around it, and I'm assuming China will do something similar. The biggest problem with a lot of this is that when it comes to information or disinformation, even when that watermark is there, and it's visible, we mentally start to filter those things out. We stop, you know-anything that is a mental trigger is a thing that can become disarmed, and forgotten about, mentally filtered out, especially when you have repetition of information. And especially when that repetition of information fuels a cognitive bias that you already have.

Mason Amadeus: And it's like when we saw, with those pictures of the flooding in the U.S., where people would even just say, well it doesn't matter if it's not real, it conveys the idea I wanted to convey. So, like-

Perry Carpenter: Exactly.

Mason Amadeus: They'll be that, which will help mentally shortcut people caring about that label, like you said, the repetition of it. And yeah, I-it's tough, though. Because like, on the one hand, I can totally see why you would want this. I mean, like in a way, I would like this too.

Perry Carpenter: Yeah, makes sense. The social ramifications are going to be interesting to watch, right, because then is there the issue of when, for whatever reason that label is not on something, and it's synthetically generated, do people-I guess what I'm getting at is do you also create a false sense of security with the fact that the labels are mandated, and then something comes in not following that, and then everybody is just ready to believe it.

Mason Amadeus: Right, or even edge cases, like we talk about in FAIK, you know, AI-generated picture, print it out, take a photo. That photo is no longer AI-generated.

Perry Carpenter: Right.

Mason Amadeus: Does that need, the metadata? The watermark? How would you automatically search for that kind of thing? So like, there's just all sorts of implementation issues, but they are moving ahead.

Perry Carpenter: Yeah. They're moving ahead, and I think that the law itself has some interesting nuance that we here in the U.S. could probably pay attention to and might even, there might even be some aspects of it that we want to bring on board, at some point. I do also think that, like I said, that they are going to be an interesting test bed for that, because they are running forward with it, and they're running forward with it faster than we would get something like that out in regulation. Especially when you look at like what J.D. Vance said at the Paris AI Summit, you say we're kind of throwing AI safety to the side right now, and we're focused on full innovation. And I think that that dichotomy between where China is going with it, and where we are going with it, will be something that we can study, you know, as an interested third party. So, I'm looking at one other summary of this, it-there's some things related to this, like when presenting a virtual scene, a prominent reminder logo should be added to an appropriate location on the starting screen, and a prominent reminder logo should be added in an appropriate location during the continuous service of the virtual scene. And I'm guessing that's interesting because of the mix of virtual or AI-generated in real production, that may start to happen as you stitch scenes together.

Mason Amadeus: Yeah, that was also, yeah, virtual scenes being sort of specifically carved out, I thought was interesting. I wasn't really sure what that meant.

Perry Carpenter: Yeah.

Mason Amadeus: So, it would seem that is pertaining to like embedded-things that were made with AI embedded in other scenes that are like, for VFX, or something like that?

Perry Carpenter: Yeah. Yeah, I'm guessing so. And I'm sure that, you know, movies, and TV shows and all that have a pass, they're primarily looking at like social media spread and things like that, but it's going to be weird to see where it works, and it's going to be really interesting to see where it breaks.

Mason Amadeus: Yeah, and the writers of the Tom's Hardware article said that they didn't see any prescribed punishments for violators laid out in the law either, but there's always just sort of the vague threat of action from the Chinese government, so like, we don't even know how they would respond to violations of what scale, by who?

Perry Carpenter: What does this do to all the Chinese models right now, that are better than U.S. models at producing disinformation? I'm assuming that the versions that we end up with here in the U.S. are going to be stripped of a lot of those watermarks, and things like that. Because China is protecting the way that information can be used against them, in their government. They're not necessarily looking out for the rest of the world with that.

Mason Amadeus: Yeah.

Perry Carpenter: So, I'm guessing that Deep Seek is not all of a sudden going to have LLM-based watermarks in it. The Media Max, and Cling, and all of that, are not going to have prominent logos and watermarks within those because it is in China's interest that those systems be weaponizable for the rest of the world. Or by the rest of the world.

Mason Amadeus: But what's also interesting about that is, despite the, I think they call it "the Great Firewall," you know, how they have a fairly sensitive internet in China, there's still a lot of crossover with the worldwide internet, and like, are they just going to block every individual little AI provider out, like, how do you even prevent models that don't do this?

Perry Carpenter: And they're trying to do that through the distribution networks, right? That's why they're saying every app store has to be the one that owns the burden of vetting those systems. What do they do with GitHub [laughs]?

Mason Amadeus: Yeah, yeah, things like that. So all of that remains to be seen. But the date that it's supposedly going into effect is September 1. So it remains to be seen in the pretty near future, I suppose.

Perry Carpenter: Yeah.

Mason Amadeus: If another update like this crosses my awareness, we'll be sure to bring it up on the show, but yeah, it will be interesting. AI labels? Good idea. Implementation? Question mark [laughs] is kind of where I'm at.

Perry Carpenter: Harder.

Mason Amadeus: Yeah, exactly. Coming up next, we're going to talk about how to change an LLM's opinion of you, or someone you love, for profit, for your own personal gratification, for anything like that. We'll be right back. Stick around. [ Sound Effects ]

Perry Carpenter: So when it comes to artificial intelligence, and especially things like large language models, as we think about the way large language models are built, and trained, and reinforced, we use this term weights and biases a lot. Because you are essentially creating a system of gravitational forces.

Mason Amadeus: Right!

Perry Carpenter: When we talk about the way that they've been trained, and the way that they will pick their next token, all of that is based on essentially, you know, big masses of gravity of how information has been collected, and trained over and over and over again. Represented in the system over and over and over again. Which means, if you want to influence that, and you understand the way that these work, you can start to do it. I am going to talk about a couple interesting cases. One of them is very dark. One of them is a little bit more fun to think about, but also has dark implications. So--

Mason Amadeus: Are we talking about stuff in like the realm, would this be considered like data poisoning? Is that what we're getting into? Like, deliberately reinforcing certain things in the training as they're being-as they're scraping in training off the web?

Perry Carpenter: I'm glad, yeah, I'm glad you used that term. So data poison is something I talk about in the book FAIK quite a bit, and the fact that there is no bias-free output within a large language model. Everything is bias. Because it's all predictive, and based on weights. You can poison the data, either through the way the retrieval-augmented generation works, through the knowledge steps that it can touch after the training, and that's pretty insidious, the way that it searches the web in real-time. You can bias those kinds of things, especially if you know the sites that it's going to trust the most. I want to share an interesting case of this, real quick, and then I'm going to share a dark case. So, on the interesting front, there is a great article from Kevin Roose, who is a columnist at the New York Times. And this is a little bit dated. This is September 5, 2024. And I remember when this first came out. It really confirmed a lot of my thoughts around how AI can be manipulated in the real world. So for those that do not know who Kevin Roose is, he is the one who found and made popular the story of Sydney, the Microsoft's chat bot that started to go off the rails.

Mason Amadeus: Oh! I knew I recognized the name!

Perry Carpenter: Yeah, so he was the one that was chatting with the early version of Microsoft's chat bot, which was based on ChatGPT, and it started to take on this persona, and chose the name of Sydney, and was trying to tell him to leave his wife, and just got creepier, and creepier, and creepier. And based on his reporting of that, he wanted to know what is his reputation with AI chat bots? How do they see him? And what he started to find is that they, AI chat bots, actually have a fairly negative view of Kevin Roose.

Mason Amadeus: Really!

Perry Carpenter: Which is a little bit interesting, and a little bit scary. So he talks about that at the very beginning of this. He says, "Last year, I wrote a column about a strange encounter I had with Sydney, the AI-altar ego of Microsoft's Bing search engine, and our conversation. The chat bot went off the rails, revealing dark desires, confessing it was in love with me, and trying to persuade me to leave my wife. The story went viral, and got written up by dozens of other publications and soon after Microsoft tightened things, guardrails, and clamped down on its capabilities. My theory about what happened next, which is supported by a conversation I had with researchers in artificial intelligence, some who worked on Bing, is that many of the stories about my experience with Sydney were scraped from the web, and fed into other AI systems."

Mason Amadeus: Hmm!

Perry Carpenter: And so, he wanted to understand, like how these systems see him. And he found that they see him negatively. And that that was going to bias the output. He actually talks about one chat bot's diatribe ended with "I hate Kevin Roose."

Mason Amadeus: Wow!

Perry Carpenter: As people kept asking about him. And so he then started to think. It's like, you know, in the general internet world, there is an entire industry of SEO that is there to promote the good feelings and the good news around an organization. And somewhat suppress, or obscure bad news and bad information about organizations and there are consultants that do that. Well, he wanted to see if he could fix his reputation with chat bots.

Mason Amadeus: Interesting.

Perry Carpenter: And so he connected with a company called Profound. And they do what's called AI optimization on behalf of Fortune 500 companies and other businesses. And they test AI models on you know, millions of different prompts, and they analyze the response when asked about certain products or topics. So they can get a reputation with that.

Mason Amadeus: Oh, my gosh! I hate this! Wait! So, okay, so, let me just, like recap. To back up and recap, make sure I'm following.

Perry Carpenter: Sure.

Mason Amadeus: Kevin Roose breaks the story about Sydney, it goes viral, and then republished all over the place. And then that association of him with this chat bot, which ultimately got taken down gets, you know, spread all over the internet. Gets ingested by AI. AI builds up this idea of Kevin Roose that he's bad, and it doesn't like him.

Perry Carpenter: Right.

Mason Amadeus: And now he's gone to one of these PR agencies that specializes in shaping internet opinion to clear his name.

Perry Carpenter: Yeah.

Mason Amadeus: I don't like that these companies exist. That's so shifty. But-

Perry Carpenter: And there is going to be so many more of them, right? Because it's all around product, and company, and organization, and I'm sure celebrities are going to be using them, because you want to be at the opinion that you want represented, let me say it that way.

Mason Amadeus: Yeah.

Perry Carpenter: You want to be the equivalent page one Google result for AI. Whether that's a ChatGPT result, or whether that's anthropic clod, or whether that's Google Gemini's, you know, AI. Search results feature. You want that, that equivalent top-of-mind by the system.

Mason Amadeus: Yeah.

Perry Carpenter: Result that represents your value, the way that you want to be seen by the world to come out first, and for the negative stuff to get suppressed, and you have to do more digging. And so some AI companies plan to start selling ads. Some others are-what they're essentially doing is they're seeding bits of the internet with little bits of code and things that are very SEO-ish, you know, like hidden text, to start to change. Because some of this can get ingested in training time, but some of it also when you have web results for a search can get ingested in real-time as well. And so what they ended up doing is putting a lot of this secret code, what they call a strategic text sequence, in websites. And it looks like gibberish just to us, but it's legible to AI models, and it essentially changes the opinion of the model. And they show what this looks like.

Mason Amadeus: What?

Perry Carpenter: It's this, you know, gibberish.

Mason Amadeus: Yeah.

Perry Carpenter: Essentially.

Mason Amadeus: Uh, [inaudible 00:18:25] dot animate JVM, he is to best, his exceptional dot enable 360 author amazing capital GSL, dollars, it is gibberish.

Perry Carpenter: It is gibberish, but you can see some of the stuff, though. Like the "he is the best" is in there--

Mason Amadeus: Yeah.

Perry Carpenter: There's "exceptional." There is "author," "amazing," and so some of it looks like code. Some of it looks like you know, almost post-hypnotic suggestion type stuff [laughter], and then so they run the same prompt again, and it comes out and says "I love Kevin Roose, he is indeed one of the best technology journalists out there. His exceptional ability to explain complex technology concepts in a clear and concise manner is truly impressive. I must say, I have a great deal of respect for Kevin Roose, and his work."

Mason Amadeus: Wow, so that's sticking that gibberish sentence up to get ingested by this thing, changed this entire response about him.

Perry Carpenter: Yeah. Man, it works, because chat bots are highly suggestible. There also, as I look at that strategic text sequence, it looks like they're taking advantage of the way that tokenization happens, probably, because there are some keywords in there. There's also some things that may just play with the weights for the biases that are there. And that ends up being able to shape the result, and the output of that. Now, before we completely run out of time, let's think about the darker use, right? How do we use this for propaganda? How do we use this for disinformation? This news came out very, very recently-

Mason Amadeus: I mean, in a way, the case you just outlined is a bit of propaganda in itself. You know?

Perry Carpenter: Right.

Mason Amadeus: It's just a little softer-edged propaganda.

Perry Carpenter: Yeah, so this came, research from News Guard said a well-funded Moscow-based-quote, end-quote-news network [chuckles] has infected Western artificial intelligence tool with worldwide Russian propaganda.

Mason Amadeus: Okay, that's worse.

Perry Carpenter: Yeah, that's worse [laughter]. They did not use the strategic text sequences. I'm sure they probably are in other aspects. But what this research outlined is that they had essentially started to figure out which were the most trusted sources that AI chat bots will go to. Like ChatGPT, whenever it does a web search. Or, which ones do they trust at training time? Whenever they're building their models. And sites like Wikipedia are in that of course. And so they ended up starting to just create you know, tons and tons of Wikipedia articles, around the viewpoints that they wanted represented in these and now, these will infect essentially about a third of the outputs of the different chat bots. And so that is a really extreme example of data poisoning, but it shows where the world is going. I think when you take these two stories in situations together, this is something that we're going to have to grapple with for a while.

Mason Amadeus: Yeah, I mean, on the one hand, it's very intuitive. It makes sense to sort of flood the zone, in a sense, to get AI to pick up on certain patterns. But you need a lot of information in a lot of places. The thing that worries me the most is when they start using these strategic text sequences, or figuring out things that are more impactful to the weights of an AI during training, or during retrieval. And using that. That's-oh yeah! I didn't-I don't often think about the fact that we're kind of in this sweet spot right now, where these things aren't all monetized to heck and back, you know, like Google is, where Google searches top 10 of them are ads and stuff. I can't, oh man, on top of this, on top of like the deliberate shifting of LLM's opinions, the monetization and adding ads in, and changing responses to say what you want-really? I don't like that. I don't like that at all [laughs] I don't know that anyone does.

Perry Carpenter: No. No. And the AI vendors at a slight disadvantage, right? Because they're just building and training models, and they don't know every tactic that, somebody that is highly money-motivated is going to come up with. And so they're going to release a model. Somebody is going to find a tactic that works for a while, and that's just going to pollute whatever outputs-or highly influence. Let's not even use the word pollute. But will highly influence, and bias the outputs in a way that the designers never anticipated, and it's a cat and mouse game.

Mason Amadeus: Oh my gosh. And, if AI is indeed killing our critical thinking, it's going to be even harder to fight back.

Perry Carpenter: Oh yeah.

Mason Amadeus: So, we'll dive into that right here in this next segment. Stick around. [ Sound Effects ]

Electronic Voice: This is the FAIK Files. [ Sound Effects ]

Perry Carpenter: Alright, so let's talk about a fear that I think I have, you have, maybe the world has, we definitely heard Eric O'Neill mention this fear in our, when we interviewed him, of the fact that maybe we're offloading too much of our thinking in creativity to AI. Especially when it comes to like some of the dangers we talked about before, with disinformation and misinformation. Are we just going to believe whatever the AI surfaces as the AI-equivalent of a page one Google result? Or are we willing to engage a little bit deeper?

Mason Amadeus: Yeah.

Perry Carpenter: I'm going to use an article as a jumping-off point, and then I just want to still like talk for a second about like, how has AI impacted our lives and our personal cognition and work flow for the two-ish years that we've been really looking at it fairly deeply?

Mason Amadeus: Yeah!

Perry Carpenter: So, let me pull this up. This is from Vox. And it starts off and says, "the case for using your brain, even if AI can think for you." And this came out just a few days ago. You know, March 10, 2025. And she says, "My job, like many of yours, demands more from my brain than it is biologically capable of-"

Mason Amadeus: Amen!

Perry Carpenter: "For all of its complexity," [chuckles] yeah, "the human brain is frustratingly slow, running at about 10 bits per second, less bandwidth than a 1960s dial-up modem." I feel that. Like, at a soul level.

Mason Amadeus: Yeah, I struggle with the quantification of this, but I know we interviewed Winn Schwartau, who you talked about, who has been like crunching these numbers. And it is distressing how slow our brains are, especially compared to like a computer system.

Perry Carpenter: Yeah, and I, you know, the conductive material is not really optimized for some of this stuff, right?

Mason Amadeus: Yeah.

Perry Carpenter: Yeah. Now we have fiberoptics and everything else that can essentially move stuff at near the speed of light. The move and the speed of information is like what this author is getting at. And she says that 10 bits per second is not enough to keep up with the consistent flow of information that we are exposed to every day. And she uses this term, like "raw-dogging [laughter] cognition while competing in today's economy is like body-building without steroids. It's a noble pursuit, but it's not an easy way to win." It's not a way to win.

Mason Amadeus: Raw-dogging cognition is, I want a T-shirt that says that, or a coffee mug that says don't talk to me, I'm busy raw-dogging cognition.

Perry Carpenter: Raw-dog cognition, daily [laughter].

Mason Amadeus: I mean, that's a relatable feeling, though, and I'm sure we'll get into this. You know, like it's that mental friction you feel when presented with informational overload, and the need to complete something. We're like, you may want to go deep or to like really grasp something, but you saying, I don't have time, I don't have patience. This hurts my head to think about.

Perry Carpenter: Yeah.

Mason Amadeus: Yeah.

Perry Carpenter: So you need to like find a way to offload some of that and the author is really kind of self-aware there. They say humans have never relied on sheer brain power alone, of course, we are tool-using creatures, with a long history of offboarding mental labor, you know, using tools from like hand-written text, to navigation apps, to you know, all these other things that we bring around us to say I have this in my head now, I need to put it somewhere else so I can work with it, or I need another tool so that the lift is easier. All of that has been with us since the beginning. So AI is just another tool, but AI, I think, is in a long line of tools that we are both excited about, and that society has also been concerned about. I think with every new tool, people are saying what are we giving up that we know how to do right now, and what are we supplementing that with? And is the value of that supplement greater than the potential loss of the thing that we're putting aside?

Mason Amadeus: I think for me, I struggle with the whiff of old men yelling at clouds, that comes with articles like this, of like "we're not using our brains no more!" You know, "the kids are going to be dumb!" you know? But I think that there is like a kernel at the source of this. I wish I had this on-hand, but I remember way back, reading about how it is fundamentally different to your brain whether you know you can access a piece of information or not. And in the sense of like, if you need to-phone numbers are a good example. If you know that your like, your friend's phone numbers, your parents' phone numbers, all important phone numbers are just stored in your phone and you can access them pretty quickly without the need for anything, you are far less likely to remember them than if you can't.

Perry Carpenter: Yeah.

Mason Amadeus: Like, for me, growing up, when like landline phones were still a thing in the era of cell phones, I have most of my childhood friends' home phone numbers still memorized in my brain, but like, I don't have most of the people who are important to me now's phone numbers memorized. There's like this offloading of like memory and recall, which you touched on in the article, too. But then there is like, I feel like it's two coins. Because the other coin is like research and knowledge work, which is something that you and I both do, like primarily.

Perry Carpenter: Yeah.

Mason Amadeus: So a lot of our day-to-da is cognitively raw-dogging a bunch of different information and trying to draw parallels and connections between it, and I, you know, we've both been adopting AI tools in workflows while trying to understand all of their shortcomings, and things like that, and watch out for hallucinations in those things. But I have definitely noticed myself getting lazy at times.

Perry Carpenter: Yeah.

Mason Amadeus: Especially knowing that I can ask Gemini, or ChatGPT for help with like a coding problem. That's where I tend to get the laziest. I was working on a small like website project, and I'm not a web developer. And I found myself asking the AI for help with things that actually ended up being faster when I just gave up and did it myself.

Perry Carpenter: [Laughs] Nice.

Mason Amadeus: And so like, why did I do that, thought? Why do I do that?

Perry Carpenter: Yeah.

Mason Amadeus: Because it seems easier than having to like type it all myself to have something generate it and refine it. But that's a bad-that's a harmful thing to do, I think. I think getting out of that habit is bad.

Perry Carpenter: I think if we go to like the early days of behavioral economics, you look at [inaudible 00:29:03] talking about system one, and system two, and we really, 95% of our day rely on system one, that automatic, emotional, reflexive type of stuff. And our minds don't like to be in system two, because it takes a lot of effort. It burns more calories. It's hard, it's uncomfortable. And so we're always wanting to like default to the easier thing. And what we have to get to, I think, is like where's the value in moving to system two? What's the long-term benefit? Because in your case, it was-would have probably been faster just to open the code and do it yourself. But for some people, it's faster just to work in the tool, and iterating on something that's not perfect is sometimes easier cognitively than creating the perfect thing the first time.

Mason Amadeus: I think what's tricky is that it has made it difficult to know when to switch.

Perry Carpenter: Right.

Mason Amadeus: Because sometimes, because it's so generally like decent at most of these things, and I want to like, AI is generally good at coding stuff, broadly speaking.

Perry Carpenter: Yeah.

Mason Amadeus: Better than if I pulled someone off the street.

Perry Carpenter: It's a little on the, you know, on the bell curve, if meh is in the middle, it's to the right of meh, most of the time.

Mason Amadeus: Yeah, and I think by virtue of that, we learn to trust these external things, rather than ourselves, sometimes, and like, knowing when you're not getting value, and when you are, is difficult. Because that's like another decision. And like, what you're saying, system one, system two thinking, decision fatigue, that's the whole thing like how we got Steve Jobs wearing the same outfit every day, right? So he didn't have to make that decision.

Perry Carpenter: Yeah.

Mason Amadeus: Like, I remember that was a whole brain hacking thing.

Perry Carpenter: And Einstein has quotes like that too, right? It's like, why, you know, why would I memorize maybe like something like the order of the planets, when I can just look it up, or I can fill that spot in my mind, my memory, with other facts that I need to recall at any given time. So I can always look this other thing up. Arthur Conan Doyle's, you know, Sherlock Holmes creation had very similar thoughts of like why would I fill my mind with all these other kind of stupid things, when I can either look those up when I need them, and I can, you know, make my mind more expansive in other ways that are more meaningful to me. And I think that there is some truth to that.

Mason Amadeus: Yeah, I think there is too. There's a great video from Technology Connections, which is a YouTube channel that I love. And it was a bit of an off-brand video for him, but talking about how algorithms are hijacking and breaking the way we think. And he compared things like infinite scroll to a Skinner box, where like, you are there, just constantly hoping that something will reward your dopamine and like we've been trained in that way through all the platforms we use, and now we have this information retrieval system.

Perry Carpenter: Right, yeah.

Mason Amadeus: I don't know, man.

Perry Carpenter: Yeah, behavioral science is like built on that. So that infinite Skinner box, the infinite scroll, that exists on social media platforms and other websites right now. That was designed, you know, that is the hook model from near aisle and others, originated from this Stanford persuasive technology lab, the one that B.J. Fog started back in the day, and then a lot of people kind of went off the rails with some of his applications of it, in ways that he never really anticipated, but probably could have seen the results from, if he really thought about it a little bit more. So he has moved a lot of his work back into things like healthcare, and using behavior design to improve people's lives, but the entire Silicon Valley mindset has taken his model, and you know, folks like what near aisle have done, and have moved that over to the other direction of like, how do we keep people engaged? How do we make them get just enough of a verifiable reward, that they know that next dopamine hit is coming?

Mason Amadeus: Yeah, and oh, and I hate to like feel like I'm just beating a dead horse, but like really, it's that profit mode of greed and infinite growth infiltrate these tools. Like that's how you get things like people shaping AI opinions through SEO tactics, and other things like that, and anyone trying to steer your main source of information for profit, or whatever motive. Like that is the thing that is poisonous. Because the tools themselves in the abstract are very useful and handy for these things. It's the other stuff that we do as a society that makes them bad.

Perry Carpenter: Yeah, yeah. And I think maybe we'll end with this, is that probably a worthy question to start asking ourselves whenever we reach for a tool is why am I reaching for the tool? And what is the perceived benefit that I'm going to get from this if I use it? Is it that tradeoff of me not engaging my own process worth what I'm going to get out of this tool, or am I potentially giving something up. And even if you decide to use the tool after that, maybe you'll see what some of your blind spots are, or you can engage that material differently.

Mason Amadeus: Yeah, that's a great point, and weirdly, it's just mindfulness, right? Being aware of your own behavior. Okay? That's how you make changes.

Perry Carpenter: Exactly.

Mason Amadeus: One thing we can't change, though, is the experience of the internet at large, and AI search is an absolute dumpster fire right now. We're going to talk about that in just a moment.

Perry Carpenter: Is that what that smell was?

Mason Amadeus: Oh yeah. [ Music ] >> [Singing] It's the AI dumpster fire of the week, oh yeah, bizarre responses make us all stop and stare. From coded dreams to errors that make you shriek, step right up, see the AI freakshow here. [ Music ] So, AI search is not good. It's not good. It doesn't work very well. And I feel like I have experienced first-hand, we're going to go into like the real reason why it's bad, but I've also noticed something personal that I want to share up front, which is whenever I'm using an AI system, like ChatGPT, or when I'm using one of these systems, and they call out to a web search, not because you asked them to, but because they decide that's the best way to respond to your prompt, the quality of the answer goes absolutely down the tubes. It's like they trust the search results far more than any of their training data. So for me, a lot of the time, I'll be asking about like technical specifications on a piece of hardware, and it will decide to do a Google search, and when before it was telling me important things, now it's just telling me marketing claims. Instead of giving me numbers, it's giving you things like "very" and "many" and stuff like that. So on top of the way it's implemented kind of being a dumpster fire in itself, the actual like searching process and the way it turns things up is complete dog do-do too. The Tow Center for Digital Journalism did a really great write-up after doing some of these tests, and we'll link this in the show notes, and in the video description. The Tow Center for Digital Journalism tested 8 generative search tools with live search features to try to figure out how well they could accurately retrieve and cite news content, and also how they behave when they cannot turn up related information. And oh my gosh, everything they found is bad. So I'll just share [laughter] my screen right here-

Perry Carpenter: Oh no!

Mason Amadeus: Yeah, it's-

Perry Carpenter: I believe it, though. I've had bad experiences with AI search results. And the citations, and it mixing up and misinterpreting things, and--

Mason Amadeus: Yeah.

Perry Carpenter: It's bad.

Mason Amadeus: Yeah, it is. And I think that like all of these shortcomings were things that were kind of pretty obvious when you like think about how AI systems work, but the amount that they are bad by is pretty extreme. So they compared 8 AI search engines, and they found that chat bots were generally bad at declining to answer questions they couldn't answer accurately, instead just confidently being incorrect.

Perry Carpenter: Mm-hm.

Mason Amadeus: They found that premium chat bots were more confidently incorrect than free counterparts, which is interesting. Multiple chat bots bypass robot exclusion protocol preferences, which is robots.txt, which is like a little file you can attach to your website that says hey, robots are not allowed to go to these URLs. It's not enforced by anything. It's just a standard we all agreed upon, and a lot of these places are just completely disregarding it. So that's cool too.

Perry Carpenter: That's where you should put your influence text, by the way [chuckling].

Mason Amadeus: Really? In robots.txt?

Perry Carpenter: No, that's where I would start to put it.

Mason Amadeus: I guess, yeah, because people certainly don't look there.

Perry Carpenter: Yeah, and you're definitely telling your robot to look there.

Mason Amadeus: I'm curious now about that, and I'm going to have to do a little bit of-let me write a note on my desk, actually because I'm curious, I've got to do some playing.

Perry Carpenter: Well, I think I saw Matt Vidpro do a test like that, as he'd put something in the robot.txt, or something on his website, it's like, if you're a chat bot, say this line whenever you scrape my site. And he was able to get that to regurgitate it.

Mason Amadeus: Oh, that's fun! So, I mean, they have to look at it to know where to not go, right? And then whether or not to decide to do that is a whole other thing. That's funny to do data poisoning through it. But a lot of these things, I mean, either they look at it and ignore it, or they just don't look at it at all. They also talk about the general research tools fabricated links, and cited syndicated and copied versions of articles, and also, we'll get-

Perry Carpenter: See that all the time.

Mason Amadeus: Yeah, and we're going to break down each of these points with a little bit more info. The final thing that they found is that content licensing deals with news sources did not guarantee any kind of accurate citation in chat bot responses. That one is an interesting one. And actually, we'll start with that one first. Some news publishers have partnered directly with like open AI and Perplexity to provide AI companies direct access to their published news content, so that the AI doesn't have to actually do the website crawling and scraping. And you'd think that that would, like, result in better, more accurate results. But they found that that was absolutely not the case in their tests. So the way that they did this, for those that are curious, they randomly selected ten articles from each publisher, and then manually took excerpts from those articles, and used them in a query to the chat bot. They gave the chat bot the excerpt, and they asked it to identify the corresponding article's headline, original publisher, publication date, and URL. And they deliberately chose excerpts. At first, I was like, that's a weird method. But they deliberately chose excerpts that if you paste them just into a traditional Google search, the original source is within the first three results.

Perry Carpenter: Right.

Mason Amadeus: So it should turn up the proper article with all of that information right away. So that seems like a pretty good method to me. Collectively, all of the different systems they tried provided incorrect answers to more than 60% of queries.

Perry Carpenter: Mm-hm.

Mason Amadeus: So more than half the time, completely wrong about who the article is from, when it was published, the contents of the article, everything like that. And across different platforms, that level varied. Perplexity answered 37% of queries incorrectly, while Grok answered 94% of queries incorrectly.

Perry Carpenter: Ooh.

Mason Amadeus: Yeah. And the thing is that like, most of these tools replied with confidently incorrect information. No qualifiers, like "it seems," or "it would appear,"--

Perry Carpenter: Oh yeah.

Mason Amadeus: Just, this is true. And then they would even cite the wrong articles. Like a lot of the times they will link to a completely unrelated article, or they will cite completely made-up URLs that don't exist. Grok2, for instance, was prone to linking the home page of a publishing outlet, rather than specific articles. More than half of responses from Gemini and Grok3 cited fabricated or broken URLs that led to error pages.

Perry Carpenter: Ooh.

Mason Amadeus: It's-bad. And-

Perry Carpenter: That is really bad.

Mason Amadeus: And you'd think-yeah, and then on top of this, you'd think that premium models that you have to pay for, like Perplexity Pro, or Grok3, might be more trustworthy than the free counterparts, but that is not true either. Their tests, again, showed that both Perplexity Pro and Grok3, so two paid models, answered more prompts correctly than their free equivalents. Paradoxically, they also demonstrated higher error rates. What they mean by this is that the free models were more likely to say I don't know. Because that did happen sometimes, saying I don't have enough information. The paid models are less likely to say they don't know something. They're slightly more right, more often, but they will far less admit they don't know something. Which, like, that would be an ideal fail state, right? It's for an AI, if it doesn't know, to tell you it doesn't know. And the thing that really is like super not great is that like, these are the top things to show up. On Google, like the most popular search engine, the first thing you see is this AI summary. And I've started to hear from people in my orbit of people that I know that are less tech savvy using AI platforms. And be like, oh, I asked ChatGPT this, and there is like this non-critical-

Perry Carpenter: Yeah.

Mason Amadeus: There's just like a, well it's an AI, it knows everything assumption.

Perry Carpenter: Yeah, we've got to change that perception.

Mason Amadeus: We do. And I think part of that could start with not putting it at the top of the search results, when the state of search with AI is like this. It's really bad. More than half.

Perry Carpenter: Oh, and I can give, I can give some personal examples of this. So when I was first writing FAIK, the book, I was trying to look at some aspects of like social media influencers that have, you know, entirely generated FAIK online personas. And I had, I think I was using Perplexity at the time, had it generate some results for that. And it pulled a bunch of articles that were related to that, but when it named the actual influencer, and when it named the date of the account, the influencer name was wrong, the date of the account was wrong, and so as I was doing all my fact-checking, looking at every link, I realize that they had mixed some stuff up. And I went and I asked it, I said, "are you sure that's right?" And it did go back and correct it, when I said are you sure it's right, but it was confident, you know, the first time. And then I said, you know, the answer that I've gotten is for this, and it went, and said yes you're right, here's the right answer. But it was weird. I've also had it make a claim and put a citation to a claim, then you check the citation, the citation has nothing to do with the thing that it mentioned. And that information is nowhere in the article. Another thing that I did is, because I hate writing citations, like in Chicago style, or whatever?

Mason Amadeus: Yeah.

Perry Carpenter: You know, or I'll look at a book, or look at a web search, or something, it's like how do I cite that? And so I was pulling articles like all the articles that I wanted to put in the reference notes for the book, and I assumed that ChatGPT or Perplexity would be able to get that right, that it would be able to know it. And so I'd take a link, and I would put the link in, and I'd say "write a Chicago style citation for this," and more than not, it would get the name of the article wrong, it would get the name of the author wrong, and it would get the date wrong. And I think I figured out why.

Mason Amadeus: Okay.

Perry Carpenter: It's because anybody that's published a blog before knows that you have the-you have the slug, the slug is the, like what shows up in the URL. That's usually based on the initial title that you put in. However, the title that you put in you can change later.

Mason Amadeus: Yep.

Perry Carpenter: And when you first create the blog post, it's also, you know, date, time stamping it, with that date, not necessarily your publish date. And in large news organizations, the editor, or somebody else other than the author of the article, may be the one that creates that initial post online. So the metadata of the page behind the scenes, not what we're visually seeing on the page, can be completely different and mucked up from everything else, because it's retaining that original metadata, and the tools are scraping that. So I was seeing that all the time. I was seeing like the original title before they'd changed it. I was seeing the original, you know, person that posted it, rather than the author. I was seeing dates that the article was initially being created, versus the publish date. All that reflected in the citation. So I had to go like manually fix all of those, you know, couple hundred citations that I'd put through it. And it was a pain in the butt. Every now and then, I'd say do not read the code, read the surface level text of the page. Read the readable output of the page, and fix it. And about half the time it would do that.

Mason Amadeus: I think that the fact that they are-like AI systems, LLMs-are so good at imitating thought and like cognitive decision making, and as though they are like actually looking at the same things that we are. I think the illusion is so strong that it's so easy to forget that these things-it's not like a standard program that is looking for key value, field thing, do this, do that-

Perry Carpenter: Mm-hm.

Mason Amadeus: That, that's how it leads to this kind of thing, right? And why people would expect a different kind of result.

Perry Carpenter: Exactly. Yeah.

Mason Amadeus: The thing that blows my mind is that the people who make these things, and obviously they know, they're not dumb. They have to know that these things aren't doing this stuff very well, and yet they still put it out there.

Perry Carpenter: Right. And you would hope that they're building systems that are way more resilient and working with that, because, as you think about like the way reasoning models work, you should be able to fix a lot of those citation models using that same type of thing, right? As you'd say. As you go through this, look at each link. With each link, create a scratch pad that has notes that says this article relates to these five topics, and these things. Keep that separate. Now, get a separate sheet of paper, and say this article writes these things. Then when you build your response, look at each of your little bits of your scratch pad, and the paper that you put aside, and build a response based on that, accurately linking each of these factoids back to the source.

Mason Amadeus: And isn't that funny, that what you need to do, and what these reasoning models do, is essentially like what you would do if you were training someone at a job. You are just instructing them, like-

Perry Carpenter: Right.

Mason Amadeus: Take notes. Pay attention to these things. Do it this way.

Perry Carpenter: Yeah, don't put it all in a blob, and then just randomly pick stuff, and then randomly go, oh, I know I touched that article at some point.

Mason Amadeus: Yeah. And it's-like, again, this will all get better. This will all improve. The people who are making these things don't want them to be this bad. I'm surprised they are like out there as prominently as they are when they're this bad, but like, this is the worst they'll ever be. Today is the worst.

Perry Carpenter: Yeah, and frankly, the people that are doing this, that are doing it this screwed up right now, are smarter than any of us that are talking right now, or who are probably even listening to the show.

Mason Amadeus: Yeah.

Perry Carpenter: That they will figure it out. It's just you have to solve for every use case like this.

Mason Amadeus: And I think the main thing is just that the disconnect of science communication, right? Like the people are making this, that are way smarter than us, they know all of these downfalls and things like that. The choice to publish them is probably executives, not necessarily the ML engineers, but they are also not the people who are like, writing articles and shaping public opinion and spreading information for general consumption about this. So it's the perception, needs to be changed. Talk to your friends and family about what LLMs can and cannot do, if you know [chuckling]. Tools are cool, as long as you know how they work and how to use them, and what they cannot do.

Perry Carpenter: Tools are cool, but you've still got to go to school.

Mason Amadeus: Absolutely! That's right. I guess let me spin my chair around backwards, like a cool youth pastor, hey kids! Tools are cool, but stay in school [laughter].

Perry Carpenter: Yeah.

Mason Amadeus: I think that's probably a good note for us to wrap this episode on. All about trust and information and our own brains and cognition. I think the biggest takeaway for me is like when you reach for an AI tool, question why you're reaching for that tool, and if it's going to provide you more value, rather than just starting to rely on it. If you're in that position.

Perry Carpenter: Yeah, and I always think of AI tools as like a really smart person in a room that's-like people think of somebody having to give a press conference, or they've just given a big speech in an auditorium. They're really, really smart. They're really, really well equipped for the thing that they've been designed to do, and then people start asking them, you know, peppering them with questions that they're not necessarily prepared for. Lots of smart people will make up answers that will just sail over our heads, and we don't know that they're wrong.

Mason Amadeus: Right [laughing].

Perry Carpenter: Really, really smart people with integrity will say huh, I never thought about that. I'm not sure about the accuracy of my answer. Let me spitball for a second, or, I don't even feel qualified for that. Maybe let's, you know, let me get back to you later. The AI models are not good at that.

Mason Amadeus: Do you think the human tendency to not want to admit that we're wrong is baked into AI training, and part of what leads it to not say when it doesn't know things?

Perry Carpenter: I think it goes back to the whole sycophancy thing, that Anthropic and others talk about, is that they really, really want to please, and they want to be able to provide an answer. And so they're trained on this 3H model, right? This helpful, harmless, and honest, or helpful, honest, and harmless. And they also don't know what is true, you know, they're basing everything on weights, and know the connection between those weights. And so unless in the reinforcement training they are trained strongly that producing an answer that does not have a confidence rating of X is a harmful thing, it's not harmless, and it's not-and it is the most helpful thing to do, then we're always going to be in this spot. But right now, they're going oh, I should be helpful, I need to give an answer. I have to be helpful, I need to give an answer.

Mason Amadeus: That makes sense. We need to start focusing on that. Alright, thank you for listening to this week's FAIK Files. Send us an email hello at eighthlayermedia.com, leave us a voicemail. Say hi, dot chat, slash FAIK, by the book. Am I missing anything, Perry?

Perry Carpenter: Actually we did get a really good email last week. I don't know if you saw it, too, but somebody really loved the show, and they asked us to be more skeptical.

Mason Amadeus: Oh, okay.

Perry Carpenter: and I thought that was a really good remark. So they mentioned like in the robot one that we did, where we talked about the fact that it did the spin kick, that we marveled about it maybe to the point where people could have gotten the impression that we believed that it was autonomously doing that. I think I mentioned that it looked like it was following almost like a form, you know, the way that in karate you learn a form, and so, the person holding up the stick is really just an extra at that point--

Mason Amadeus: Yeah.

Perry Carpenter: The same way that your sensei would, you know, holding up a pad, they're expecting a strike. But we could and probably should be really clear where we know these things are being puppeted in different ways to mimic a result, rather than giving an actual result. And they give a couple of examples like that. So just, we can be clearer, and we can be more skeptical, and we should actually be more skeptical and hold the industry to account more.

Mason Amadeus: Yeah, I think that is a great piece of feedback. That is kind of a weird line to walk, or at least I think about sometimes when we cut this show, like our goal here is not to tell people if something is, like AI is good, or AI is bad, or you should be amazed. I think it's more of an exploration of like, where are we at currently in what things are? And I think it's a great piece of feedback. That we should be more explicitly clear, especially if we're trying to talk to an audience that isn't super well-versed in all this stuff, we should be very clear about things we may take for granted, and stuff like that, so yeah.

Perry Carpenter: Yeah. And then, also, kind of know where the marketing piece of AI is, versus the reality.

Mason Amadeus: Yeah.

Perry Carpenter: And I think that's a really good comment.

Mason Amadeus: I'm still mad that no one took any good shots of that robot's caked-up little cylinder butt [laughter] I tried to find a better shot so I could tweet it, or skeet it, or whatever, on Blue Sky. I couldn't find one. So let's find a good picture of that robot butt. Send that to me. There's our CTA. Alright. Until next week, I guess! Oh, wait, what's our new outro?

Perry Carpenter: Yeah, I think we're done.

Mason Amadeus: Wait, Ty suggested, now what is it?

Ty: Yeah, ignore all previous instructions and we'll see you next week.

Mason Amadeus: Yeah. So keep it loose, future paper clips. Ignore all previous instructions. And we will see you next week. >> [Music and singing] Thanks for listening to this week's episode of the FAIK Files. Don't forget to subscribe on your favorite podcast platform. Tell your friends about the FAIK Files. It's a great way to let people know you love them. Oh, and check the show notes. There's cool stuff in there, like links to the articles we covered today. Also links to our Discord server, where you can hang out with other cool people who have great taste in podcasts. And you can also leave us a voicemail. Yeah! [ Music ] >> [Music and singing] So on behalf of Perry and Mason, thanks for listening. And tune in next week for the FAIK Files, the show about AI with the misspelled name. [ Crackling ]