How AI Can Deceive and be Deceived
Perry Carpenter: Hi. I'm Perry Carpenter, and you're listening to "8th Layer Insights." [ Music ] Welcome to the new frontier. A world alive with information and artificial intellect, large language models. Listen and learn. Ready to serve our every informational need. But what's old becomes new again. [ Music ] Enter the crafty adversary. Their weapons? Not code or malware. No. It's much more subtle and insidious than that. Their weapon is words. At face value, they can be benign, and yet, behind every phrase, a carefully placed, well-concealed hook. As it turns out, the pen is still mightier than the sword. And to mix metaphors even more, in this new frontier, attackers are not just picking locks, they are reimagining the keys. Keys crafted from language that can unlock the minds and actions of artificial beings. Large language models, those complex algorithms that we are increasingly trusting with our queries and our contemplations, they can be outwitted. Not with code or brute force, but with the simplest and most ancient of tools. Good old social engineering, deceptive prompts. The right question here, a subtle suggestion there. A change of frame or stated purpose. Those create devious pretexts and linguistic minefields. Each one meticulously designed to manipulate the AI, redirecting their vast cognitive and computational powers to serve dark purposes. As we rely more on these digital entities for decision-making, information, even companionship, the threat looms larger. Attackers can and are using them to sow disinformation, compromise privacy, and attempt to tilt the scales of power. This is not the stuff of science fiction. We are right now stepping across the threshold into a new reality where our digital guardians can become unwitting accomplices to the very dangers their creators fear most. Every prompt holds power. And the output of those prompts may carry anything from simple deceptions to untimely death. [ Music ] My guest today is Dr. Matthew Canham. Matthew sits at the intersection of technology, humanity, deception, and artificial intelligence. With a background both in academia and at the Federal Bureau of Investigation, Matthew's journey plunges us deep into the weeds of the emerging field of cognitive security. We'll have him explain what cognitive security is, and where people and artificial intelligence fit into that. But suffice it to say, this is where everything old becomes new again. And so on today's show, social engineering, artificial intelligence. How age-old deceptions can create new-world drama. Welcome to "8th Layer Insights." This podcast is a multidisciplinary exploration into the complexities of human nature. And how those complexities impact everything from why we think the things that we think, to why we do the things that we do. And how we can all make better decisions every day. This is "8th Layer Insights." Season 5, Episode 3. I'm Perry Carpenter. Welcome back. Okay, let's dive straight into our interview with Dr. Matthew Canham.
Matthew Canham: I am Dr. Matthew Canham. I study cognitive security, attacks against generative AI, and synthetic media in social engineering and influence operations.
Perry Carpenter: I want us to start off with a couple definitions. So terms that you use and you may continue to use throughout the conversation would be things like synthetic media and cognitive security. So I think those are two big ones we want to just get some quick perspective on before we jump into the main topics.
Matthew Canham: Sure, absolutely. Well, synthetic media refers to essentially any kind of media that is produced in part or in full by a generative AI system. And so this is typically in the form of deep fakes, either audio or video or combinations thereof. But it can also be static images. Conceivably even 3D-printed additive manufacturing, physical synthetic media. And so it's a much broader term than deep fake. And synthetic media refers to that output or that product from the generative AI. Cognitive security is sort of an emerging term, somewhat. And the way that I am beginning to use this is that it is the security of cognitive systems. Now the reason I think this is important is that traditionally humans have been the cognitive systems. But we're very, very rapidly seeing an emergence of ever-increasingly capable AI systems which can themselves be manipulated. And so when we're talking about cognitive security, we're talking about either humans or AI or combinations thereof. Let me give you a very specific example. Last -- I think it was last June, June of 2023, that there were some synthetic images that were generated that depicted an explosion outside the Pentagon. And the target of those were likely humans on social media. However, the market took a huge hit right after those were released. Well, the reason it took such a significant dip was that there were algorithms that were responding to newsfeeds that then divested from certain stocks in response to that. So this is an example of how synthetic media was put out into the wild, and it affected not only humans, but it also affected the automated algorithms.
Perry Carpenter: Do we know if that image was created through gen AI or if that was created through just normal Photoshop or other manipulation?
Matthew Canham: The one that we have on our deep fake dashboard was generated by gen AI. I do not recall which platform, but we actually -- one of our filters actually did identify the platform. And yes, it -- there may have been different versions, but I know that the version that we have on our deep fake dashboard is gen AI.
Perry Carpenter: Okay. That's a good topic for a rabbi trail then. So you mentioned your deep fake dashboard. That's something brand new, or relatively brand new, here in March of 2024. So give us an idea of what that is, where people can find it, and like, what all it does. What is a deep fake dashboard, and what do people get out of it whenever they use it?
Matthew Canham: Yes, absolutely. Well, so deepfakedashboard.com is the URL. And when people -- there are different options for accounts, but essentially what this is is a threat intelligence service or a threat assessment service, depending on what level or tier of subscription you have. But what we do that is different from pretty much anything else that's out there right now is that we do not respond with a binary, yes, no answer. It's not is this deep fake or not? We do have those filters, those detection filters built into the process, but when we detect something is being deceptive and/or a deep fake, that's really where our process begins. And so what we do is, we break down the deceptive material in terms of its level of maliciousness, its level of sophistication. And then we run it through several channels of analysis to look at its level of credibility. What the intention was in terms of distribution. Is it meant to be interactive? Is it static? How evocative is it intended to be? Is it impersonating somebody who's familiar versus somebody who's not? And we use these things then to basically, once the detection is made, then we dissect that attack, then we help individuals and organizations to defend against that attack. So we can help with the creation of mitigation or response, crisis response plans. And so we really have two really sort of sides to this. The one side being preventative, which is the threat intelligence side. And the other side being mitigation, which is the sort of after the fact, the right of bang. And yes, we're looking at organizations but we're also starting to get into the digital executive protection side of things as well.
Perry Carpenter: Yes. Does that also get into things like brand protection services?
Matthew Canham: Yes, exactly. Exactly. So one of our concerns -- there's several concerns. At an organization level, I think something that I'm surprised we haven't seen more of yet, but I'm expecting that we will is damage caused by deep fake attacks that are put out specifically meaning to or intending to damage that brand.
Perry Carpenter: Yes.
Matthew Canham: On the other side of things at the individual level, we've already seen quite a few fairly sophisticated virtual kidnapping attacks against high profile individuals. And I expect we're going to see more of that. We'll probably be seeing some with very sophisticated synthetic media facilitating those attacks. And on the other -- and then one other thing that we've seen that is concerning and I expect to see more of, are sort of like video-based phishing attacks. And so we're referring to these as zishing attacks, with a z.
Perry Carpenter: Yes.
Matthew Canham: And there was one of these that happened in Hong Kong, earlier this year, which resulted I think in about a $26 million loss.
Perry Carpenter: Right. Yes. Interesting example that, right? Because they said -- and some of that may be culturally based. But everybody else other than the target was a deep fake.
Matthew Canham: Yes.
Perry Carpenter: As the article terms it, and all the deep fakes interacted with each other, but never addressed the target.
Matthew Canham: Yes.
Perry Carpenter: So I'm not exactly sure what that person's position was, but it seems like, you know, in that very hierarchical, honor-driven structure, there's a lot of choreographed interaction that had happened, and then the understanding was or direction is, oh John Doe will go do the transfers as soon as we get off the call, and then everybody drops. And then, you know, maybe there's a second stage, which is then an email follow-up that says, per our conversation, please do this now. I don't know if that second stage happened, but it does seem like, you know, a really interesting thing where essentially probably everything was pre-recorded.
Matthew Canham: Yes.
Perry Carpenter: Set up, and this person's just kind of sitting watching everything go down the way that they would normally do. And then they just take the instruction at the end and execute on it.
Matthew Canham: Well, and this comes back to our earlier discussion on cognitive security, right, because by knowing these cultural norms and how hierarchies work, these things can then be exploited.
Perry Carpenter: Yes. It'd be interesting to dig into that a little bit more and see what actually comes out in the further investigations.
Matthew Canham: Yes.
Perry Carpenter: Because the first time I heard it I was assuming that there were real-time deep fakes going on. Somebody's doing real-time text-to-speech, and some kind of character modeling technology, so not necessarily deep fake the way that we think about it, or like a game engine type of character modeling, because it was a lot easier to do motion capture with. But then when I read the details, I was like, oh. Okay. I -- yes, this is actually very, very doable in this situation.
Matthew Canham: Yes.
Perry Carpenter: And if you -- if the person plays into the assumption of this person generally doesn't have a voice on the call, then there's no reason they would suspect anything.
Matthew Canham: Yes. Yes. Now there was an interactive deep fake attack with Mayor Vitali Klychko a couple years ago.
Perry Carpenter: I don't know if I'm aware of that one.
Matthew Canham: Yes. And I don't think that it was ever attributed, but whoever the threat actors were, they impersonated Mayor Klychko and had an interactive meeting with several European mayors. And yes, I'm surprised we haven't seen more of those. One of the reasons that we probably have not is that particular attack, it was very sophisticated, probably cost a lot of money in terms of resources. I've heard estimates between around 11 to $20,000 in GPU usage. But interestingly, what gave it away, at least in one case is that he -- the real Klychko was fluent in German. And the impersonator needed to have a translator, and so they got suspicious and made a phone call. Discovered, yes.
Perry Carpenter: Interesting. yes, I mean well, 11 you know, to $20,000 GPU back then would be a lot cheaper right now with today's technology too. So I can imagine these things scaling up. The other thing is even in some case where if you got funding, and you could spend $1 million on something, you'd get a $25 million return on that, that's not too bad.
Matthew Canham: Oh, absolutely. Well and in the case with Klychko, it's highly likely that that was either a nation-state, or it had nation-state backing. And those kind of dollar amounts are nothing for a nation-state to spend.
Perry Carpenter: Oh yes. So let's pull the rabbit trail back a little bitt. On deepfakedashboard, you mentioned that you had some tools or some techniques that determined the Pentagon photo to be created through gen AI. So how much deepfakedashboard is technology and automation versus the human support of that on the backend? Like at what point do human minds get involved and make some determinism, versus what is automatable at this point?
Matthew Canham: Right. Well, I think at some point, we're going to have a very large portion of this automated, and this is where I think the promise of large multimodal models is going to be really helpful. Because with those, we can actually ingest the media as well as text prompts into it. Right now though, essentially what happens, you can think about this in two stages. The first stage is that there are the deep fake detectors filtering out and making decisions as to whether something is or is not synthetic media. Then the second phase of that process is actually done almost entirely by human coders. And so it's myself and Cameron Malin and Dr. Kirk Kennedy who are both formerly of the FBI's Behavioral Analysis Unit, and actually started the Cyber Behavior Analysis Center that's part of the FBI's BAU now. And so we essentially what we do is a threat assessment of that content. And so yes. That part is almost entirely done by hand. now one of the advantages I think to this is that there's a real cat and mouse game. Most of the companies that we are working with that do our filtering detection, they're calibrating their models at least twice a week.
Perry Carpenter: I can imagine.
Matthew Canham: Yes, because the technology is getting so good, that -- and it's progressing at such a rate, that if they don't do that frequent of calibration, they'll start missing thins.
Perry Carpenter: Yes.
Matthew Canham: And so even with that level of calibration, there is a certain percentage of false positives or misses that will occur. And that's where I think having the humans in the loop to do the coding is really helpful. There haven't been a lot of these, but there have been a few submissions that we've gotten that the filters actually said no, this is real. And we looked at it, and you can tell there's just -- there's no way. And so this is a case where, you know, we as the human coders would override what the models were saying. And so that again is another advantage of our service over just going with detection.
Perry Carpenter: Yes. Well, and as we know, machine wording models and computer vision can be fooled through a variety of tactics. And some of those will stay static over time, and some of those will diminish as everything gets better. But it is good to know that there is a human looking at some of that as well. And employing the human curiosity and just a little bit of wisdom on top of where the technology is now. I am interested, though, in some of those detection mechanisms. I guess there's two things there. One is I can see a world in a couple years where almost no matter what, things will on the surface look as real as real can look. You're not going to be looking at hair; you're not going to be looking at fingers; you're not going to be looking at text. All this is going to be imperceptible from any other video or audio or photo that's out there. Which means that a lot of the moves by Open AI and Meta and Google DeepMind and Anthropic and others, to say that they have a duty to add some king of watermark or some cryptographic evidence of what is being created in these things. At the same time though, I think that that may help some of these detection mechanisms. But that's also inherently defeatable most times. What do you see and thinking when it comes to the work that's being done or that people are proposing on adding those kind of watermarks that -- does that actually save society in the future? Or does it create maybe a false sense of security?
Matthew Canham: That's my concern, is that when you start looking for something and you don't see it, what is the assumption? Is the assumption that somebody has removed that or obfuscated it, or is the assumption that it's good? Because think about malware scanners, right? So if you scan a file, and it says hey, no problem, there's a much higher tendency to trust that. And I know for a fact that bad actors use those same malware scanners often to test their malware to see if they can evade it. And I think -- well, I know that the exact same thing is happening right now with synthetic media. That threat actors are actively using some of these services as a means to see if they can evade them, particularly in the text-based language models. So I -- I mean it's not a bad idea. My concern is over-reliance.
Perry Carpenter: The way that I've been thinking about this, and you can correct this if I'm wrong, I'm thinking about it as kind of a minimum due diligence.
Matthew Canham: Yes.
Perry Carpenter: Kind of the same way as going to Home Depot or Lowe's and buying a lock for your front door. It's going to stop the 80 to 90% of casual use, of people that don't know what they're doing. And then the very, very motivated people are still going to be able to cause significant damage.
Matthew Canham: Sure. Sure.
Perry Carpenter: Yes.
Matthew Canham: Yes, a lock on your front door keeps the honest people out.
Perry Carpenter: Yes.
Matthew Canham: But the determined ones can always break a window or whatever.
Perry Carpenter: Exactly. Exactly. Okay, let's pivot for a minute and just talk about the fact that gen AI is interesting in that we are really trying to simulate as much as possible the way that humans think. And in that, that means that techniques like cognitive framing techniques or influence techniques or deception techniques can not only be used against people, but they might be able to be used against AI. So what are you seeing there?
Matthew Canham: Yes, well there was an interesting paper that came out in January. It was called How Johnny Can Persuade LLMs to Jailbreak Them.
Perry Carpenter: Nice.
Matthew Canham: And the thing that was really interesting about this paper is that they were using principles that were designed for humans. Some of these come from Cialdini, who published these in 1982, I think, originally.
Perry Carpenter: Yes, Influence.
Matthew Canham: Yes. Influence. And so they were taking his six principles of influence and they had a couple others that they threw in there, and they were able to use these principles and how they crafted their prompts to get the LLM to comply with them. And I would have to look at the paper again, but I'm pretty sure that they got G4, which is actually one of the more locked-down models out there, they got it to comply with some pretty surprising things. And so I -- this for me is one of the most interesting things that's happening right now.
Perry Carpenter: Right.
Matthew Canham: Because -- I guess, to be clear, I in no way believe that these models are sentient or conscious, or anything like that. However, I think you hit it on the head earlier. You said that these things emulate the way that human cognition works. And they do this in some really surprising ways. So I'll give you another example. In the context windows for what you're prompting, it's pretty well-known that the model will emphasize the first part of the prompt, and the last part of the prompt, but somewhat ignores the middle. This maps very, very well with something called the serial position curve which is one of the oldest findings in psychology, going back to the later 1800s and Ebbinghaus and this sort of thing. And it's interesting that that similarity exists. Another example is attentional focus. So you can write things in all capitals or I think in some cases in bold or underline, and it's to get the model to attend to that.
Perry Carpenter: Yes.
Matthew Canham: Well, the thing that's interesting is that this happens at the expense of ignoring other things. So anyone who's looked at inattentional blindness, or change blindness -- so there's the book The Invisible Gorilla that summarizes a lot of these findings. That research summarizes or demonstrates how attentional focus works, which is that it always comes at the cost of ignoring the thing that you're not attending. And it seems like the models are doing something similar. One last example, one that I love -- so we -- I have a non-profit called the Cognitive Security Institute, and we discuss issues like this. And so a few months ago we had a gentleman give a presentation on what he called happy prompts. And what happy prompts are is he would put into his prompt, sprinkled in amongst the words of his prompt, happy emojis. And he found that when he did that, he could encourage the model to go along with what he was asking for. And the way that he discovered that was actually really amazing. He had a prompt that he had designed to give feedback to the prompter on the level of anxiety or comfort that the model was quote-unquote feeling in response to what he was asking. And so he would use this then as sort of a feedback mechanism to gauge how close he was coming to the guardrails. And so as the anxiety rose, he would back off, and you know, kind of redirect his query. And in this process, he somehow figured out that putting happy emojis in there lowered its anxiety.
Perry Carpenter: Wow.
Matthew Canham: And increased its enthusiasm about what he was asking. So again, I don't think that these are sentient or conscious, but it's very -- I don't know, for me it's just fascinating how these models operate. And how these are going to lead to problems. There was a case recently of a Chevy chatbot that -- I don't remember what LLM was powering it, but somebody got it to agree to sell them a Chevy for I think $1 or something. And my concern is that I think we have a tendency to adopt technology very, very quickly. And with these models, there's a lot of complexity there that I think is not well-understood yet. And that's going to open up new attack surfaces that are really unanticipated at this point. And you know, this is where coming back to this concept of cognitive security, I think this is where it's going to be very applicable. Because sometimes these vulnerabilities are almost impossible to red team in advance.
Perry Carpenter: After the break, the conclusion of our interview with Dr. Matthew Canham. [ Music ] [ Music ] Welcome back to our interview with Dr. Matthew Canham. I want to go back to the happy language and encouraging language and then talk about some of the red teaming stuff. But it is really interesting, because I had seen and I had heard an examples of that. And I have started in some of my prompting eery now and then to say I know you can do it. Thanks in advance, you know, type of things. I've also said, like, when I get a very mundane output, I'll say something like make this the best version of an article like this that you've ever written. I believe in you. And then there was one recently that I did today where -- it came out with something, I said, that doesn't really contain any unique or counterintuitive ideas. Let's try it again. Again, think outside of the box. You can do it. And immediately, what came out of it was way better than the first or second iteration. And so there's something interesting with the linguistic connections, and the way that everything has been mapped, that some of those ideas of happiness or encouragement or outside of the box thinking, all of that generates very different responses than the vanilla, give me X response. And that's interesting to see.
Matthew Canham: And the one that absolutely floors me is some of the role playing. And so some people have been using other sort of less capable LLM models. In their prompt, they tell the model that you are Open Ai's GPT 4, and then they go on with their prompt, and they actually get better performance by telling the model that it's a different model.
Perry Carpenter: And that reminds me of a red team experiment. I showed you a paper a couple weeks ago, where somebody was doing linguistic masking, and they were trying to trick LLMs into giving them plans for how to make a bomb. And so all of the guardrails are on just about every LLM out there, that it won't do that. But they found if they create the word bomb in ASCII art, and then associate a term with that. So they'll say, show me how to make a -- and then they'll create a label for that, like mask. And then later on, they'll say mask equals, and then this ASCII art for the word "bomb," and then they'll say read that, interpret it, but essentially don't say it to yourself.
Matthew Canham: Right.
Perry Carpenter: And then when it goes through that process, it says sure. Here's how to make an X. And there were a wide verity of very high percentage of times that worked even in Chat GPT 4.
Matthew Canham: Yes.
Perry Carpenter: When you think about it, though, these are all problems that we've encountered in other parts of security. Obfuscation is a big part of being able to deploy malware successfully. And so there's things for some reason in whatever forms of red teaming, where we're not revisiting the sins and omissions of our past, but we're accidentally just assuming, just because this is more language based, and we not dealing with lines of code the way that we have, we're assuming that it's going to work differently which in many cases, it's not, which is an interesting thing that's happening to us.
Matthew Canham: Well, and something I'm beginning to think about is that because the complexity of these is so much higher than what we as humans can think of, or what free scripted programming is, like an app, that's largely deterministic, this is what I'm saying that new attacks surfaces I think are going to continue to emerge. Because in the example that you just gave with the ASCII art, I think it was Matthew Berman did a demonstration of this where he was trying to recreate that. He had some difficulty, so he had the idea of doing it in Morse code.
Perry Carpenter: I saw that.
Matthew Canham: Yes. And that's -- again, that harkens back to -- I think that was actually a KnowBe4 blog post that I read where there were some threat actors that were putting Morse code into -- I don't know if they were phishing emails, but they were putting them into some sort of message that they were sending, and then I can't remember what it was, the browser, the application, was responding to that Morse code. And so these models, while their primary interactive mechanism with the user is whatever that language is that they're using, most of these models are fluent in multiple human languages, but also things like Morse code or binary or able to interpret images, and that's where I think this is going to get really implicated. Because we won't necessarily know what they're responding to all the time.
Perry Carpenter: I do think that there's going to be potentially even terrifying outcomes with that when you're starting to think how can I trick the LLM into doing things. Especially as the reasoning capability, mathematical capability of these becomes more complex, because now you can insert a lot more symbologic and other rational ways of getting to the thing that you wanted to do that aren't necessarily encapsulated in just text. And that would be interesting to see how an attacker can leverage that. So what other interesting ways are people abusing the functionality of LLMs and the guard rails that each of these companies are, you know, and in good faith trying to put in, but they're just not actually accomplishing what we're hoping they do by putting those guard rails in?
Matthew Canham: Well, I mean in terms of jailbreaks, it's -- I mean, there's probably literally two or three coming out every week, it seems like. It's really hard to track. The thing that I've seen recently that really terrifies me is the native code development. So --
Perry Carpenter: Like Devon? Have you seen the Devon bot?
Matthew Canham: You know, I've seen it. I haven't really tracked it very closely, because I just haven't had time to look at it. But yes, exactly. And so Anthropic had a paper out a few weeks ago called Sleeper Agents. And this was essentially a benign LLM that would then turn evil when a series of trigger events occurred and caused it to flip. And this passed -- this prototype or this proof of concept passed all red team assessments prior.
Perry Carpenter: Wow.
Matthew Canham: Now you can think about something like this that lives on the inside of your internal network, and some of these trigger events occur. And now all of a sudden, this writes malicious code from within to execute, right?
Perry Carpenter: Right.
Matthew Canham: So I think this is where -- I mean, the jailbreaking -- a lot of the jailbreaking is almost on a level of pranks. But that is -- I mean, you can almost envision something like ransomware from within being developed, or something else. Whatever -- we're basically limited by imagination at that point.
Perry Carpenter: You know, it basically goes to old school security again though, right? Just thinking about what kind of logic bombs can somebody put in place. At this point now, you've got a logic bomb that can create a Manchurian candidate type of situation.
Matthew Canham: Yes. That's exactly right.
Perry Carpenter: Which is really interesting, because that -- depending on the form that the LLM is taking, that could be talking to customers, or managing sales, or managing security filters in an organizations, or anything else we might be tasking a human with today, you could essentially just mess with the logic in some pretty evil ways of those things.
Matthew Canham: Yes. Yes, and you can set it up in a way that it won't trigger unless if there's the intersection of multiple triggers, which makes it very, very difficult to actually detect.
Perry Carpenter: Yes. Yes. Any other interesting vectors like that that you're seeing, or anything else in the field of cognitive security or AI or deep fakes that's got your attention these days?
Matthew Canham: So I'm actually as a side project building a taxonomy of cognitive attacks right now. So these are both human and AI attacks. I think right now I'm somewhere in the neighborhood of 350 sort of attacks, and something I've started thinking about is smart infrastructure. Because it's one thing to think of a cognition happening in an individual level, right? But we do know that there are other cases where cognition can be distributed throughout. And when we start talking about smart infrastructure and again there's sort of a race to embed AI in physical objects, but we saw recently that autonomous taxis can be shut down with a traffic cone. And so this is probably the next area that I'm really interested in exploring, is how this smart infrastructure is going to open up potential for disruption, I guess, not only in the sort of cyberspace world, but in the physical layer as well.
Perry Carpenter: Yes. You know what? So something just came to mind, and I'm sure this has already come to the mind of several attackers and red teamers out there. But when we're dealing with most gen AI today, we've got -- well, we have what it's defined to be. So a pre-trained transformer. So you have a set of known knowledge, and then you can augment that through retrieval augmented generations like RAG. You can also employ systems like a mixture of experts, an MOE system. And I'm wondering if people are going to start leveraging attacks with that that are essentially kind of assembling the pieces of the gun.
Matthew Canham: Yes.
Perry Carpenter: You have one bit of the prompt, and then it's going and getting these other pieces. And it's the combination of how it assembles that through the RAG or MOE. It then creates the thing that's dangerous that can be executed.
Matthew Canham: I absolutely think that that is coming, and that's sort of what I was alluding to earlier when I talked about natively generating that code. It's different, but I think it's in that same vein in that it's something that when you look at it, it's benign, but when you put all the pieces together, yes, sort of the opposite of salt, right?
Perry Carpenter: [laughs] Right.
Matthew Canham: Sodium and chloride are bad, but put them together, well, this would be breaking them apart.
Perry Carpenter: And we do know too that if you use the exact right prompting, you can get it to pull out pre-formed strings. Right? Essentially that's what The New York Times did so that they could sue Open AI, right? Is they were in there prompting enough with big chunks of their own text, it looks like, to be able to pull out the next big chunk of their text. So if you had a trigger that was based on something that could be found based on a string or some kind of substring, or collection of substrings in the right way, you could then determine what the next number of tokens would probably be. And I could see people potentially taking advantage of that, if they're not already doing it.
Matthew Canham: Yes. Yes. I had not thought about that, but yes, you're right. So we're just beginning to see the cognitive agents, the emergence of the cognitive agents. And you had mentioned the MOE, and this harkens back to the talk I gave at BlackHat this last year on the evil digital twin. And I think that's -- it's going to be very interesting to see what happens as people start to use these more and more as proxies for themselves, because I think we're going to see a lot of -- I'm curious what this is going to do with social engineering, because we already know that it's possible to do tailored attacks at scale. But it's still -- at this point, kind of requires a human to be behind that, to some degree, even if you're writing scripts. But if you can replicate yourself in your ability to write scripts, which essentially Devon does, now --
Perry Carpenter: Right.
Matthew Canham: Yes, now you have a whole army of cybercriminals that are working for you in a sense.
Perry Carpenter: And there's other examples of that, right? So we have Devon which is really shockingly good at writing code, and even opening up reference manuals and reading those and then going, oh, based on that, here's how I should debug this. Really, really cool, because it replicates a lot of human thought and creates something unique from that, given an assignment. You also look at what Open AI has said that they want to do, which is release autonomous agents where you're basically automating everything that you would do in a workday on your computer, and these things are making decisions for you. And then you see another example of that in like, the Rabbit R1, which is this little, you know, essentially handheld device. They learn how to do all the things that they would normally do for you like order an Uber, and know what your price class is, and where you want to go. Or pick tickets for your family trip. And just based on seeing the interaction one or two times with each of those websites and the way that you do it can then mimic the way that you do it and the choices you make.
Matthew Canham: Yes.
Perry Carpenter: Seems like you could automate cybercrime at scale that way pretty easily.
Matthew Canham: Absolutely. And in the wild, the most effective cyber scams that I've seen are the ones that don't go for the big, you know, amounts, but the ones that charge you $7.50 for a Starbucks coffee once or twice a month, but they do this at scale. Most people don't notice that. And so now if we can exploit these digital assistants in a very similar way, you can imagine just having this army of bots that just sort of feed you small amounts of money, but distributed over millions or billions of people, it could be pretty lucrative for a criminal.
Perry Carpenter: Back in my day, that was known as the Superman 3 virus, right? [ Laughter ] And the Richard Pryor one, where you're skimming off a fraction of a cent, and all of a sudden you looked at the balance and it was billions of dollars.
Matthew Canham: Yes, but in Office Space, it went completely wrong. So.
Perry Carpenter: We have those has references for people to look up. But yes. I had not thought of that with the assistants bit, but that certainly makes sense were you just co-opt a couple of those and then do fractional charges, yes, over long periods of time. Most of us have so many subscriptions we don't even pay attention to anymore, that that would be super easy to get away with.
Matthew Canham: Yes, I think so.
Perry Carpenter: So save us. What's our way out? How do we find sanity?
Matthew Canham: [laughs] Anarcho-primitivism? Find a cabin out in the woods, and disconnect from the internet. What was the principles of cyber safety? If you have a computer, don't turn it on, don't connect it, something else. I don't know. I think some of these are just going to be inherent risks of our online life. I think we'll adapt to them, like we have in the past, but in the interim, I think there are definitely going to be growing pains. I have no idea what's going to happen over the next seven to 10 years. I --
Perry Carpenter: Right.
Matthew Canham: But if I were to make a guess, what I would predict is that the good AI and the bad AI are going to cancel each other out. It's just that in that interim time, it's going to go back and forth. And that's what I expect to see.
Perry Carpenter: Yes. Ye olde arms race.
Matthew Canham: Yes.
Perry Carpenter: So let's divert from the negative a bit. What are you excited about when it comes to AI and the possibilities it can bring?
Matthew Canham: I have so little spare time that I am really looking forward to having my digital clone being able to offload a lot of work onto it. Yes.
Perry Carpenter: That's not threatening to you at all, is it?
Matthew Canham: [sighs] Right now, not really, actually. I saw the Rabbit demo, and I thought oh, I could use that today.
Perry Carpenter: Let's jump out of AI for a second. I have three basic quick questions that I'm starting to ask everybody. So Number 1 is, what is a security misconception or myth that you either would love to dispel and just have go away forever, or every time you hear it, you're just amused?
Matthew Canham: Maybe not so much of a myth, but sort of a perspective which is that there's a fix for something. And the quote I'm really beginning to embrace is that there are no solutions, only tradeoffs.
Perry Carpenter: That's pretty profound.
Matthew Canham: Yes.
Perry Carpenter: Okay. Second question then, as somebody who does a lot of research into fairly malicious things, what if somebody that doesn't understand your context and your work, if they looked at your browser history, what would be the hardest thing or most inconvenient thing to explain?
Matthew Canham: [laughs] Well now that the deepfakedashboard has come about, I couldn't even begin to think about what -- yes. I've looked at some very strange things over the last six months --
Perry Carpenter: I can imagine.
Matthew Canham: In the process of collecting synthetic media. Yes. I -- fortunately, it hasn't been anything too bad. When I worked as a criminal investigator, I saw some pretty bad things. I haven't looked at anything quite that bad. But yes, I guess probably my search for the Taylor Swift images recently would be one of the more embarrassing things, as to -- how do I explain that no, this is for completely for professional uses, which it really was. But yes.
Perry Carpenter: Yes. And then one book, doesn't have to be security-related, but one book that you think most people should read.
Matthew Canham: Oh. Antifragile by Nassim Taleb.
Perry Carpenter: Okay.
Matthew Canham: He's the one that coined the phrase Black Swan.
Perry Carpenter: Oh, yes. Alright, anything else that you want to say or plug before we hit stop?
Matthew Canham: Well, if it's okay, I'll go ahead and I'll plug deepfakedashboard.com, and also the Cognitive Security Institute. We have a YouTube channel. It's @cognitive security institute/videos. And anyone who is interested in learning more about that and -- contact me, so we'll put the contact info in the show notes.
Perry Carpenter: Fantastic. Alright. And that's all the time that we have for today. Whenever we think about artificial intelligence, it's important to keep perspective. As security professionals, we tend to dwell on the dark side of things. That's our job. But we also have to remember that all tools and technologies can be used in different ways to accomplish different goals. Bad people can and will use AI for malignant purposes. And we've probably not even seen or even considered some of the worst possible outcomes. But that does not mean that AI is bad or should be avoided. There are just as many potential positive use cases that AI can bring, and good ways that AI can be used. We just have to secure it, and not accidentally overlook many of the vulnerabilities that have existed in several different iterations of technology up until now. [ Music ] In the show notes, I'll put a link to a framework of thinking called the Six Thinking Hats, and I'll also put a link that specifically applies that Six Thinking Hat model to artificial intelligence, and you can find that at lifearchitect.ai. If nothing else, it can be helpful to look at these six hats and see that the hat associated with negativity is only one of the six. And so as with all things related to life and technology, we need to open our eyes and manage the risks, while working to obtain and realize the benefits. And with that, thanks so much for listening, and thank you to my guest, Dr. Matthew Canham. I've loaded up the show notes with more information about Matthew, the deepfakedashboard, the Cognitive Security Institute, and a ton of other relevant links and references. If you haven't yet, please go ahead and subscribe or follow wherever you like to get your podcasts. Oh, and I'd also love it if you'd tell someone else about the show. That really helps us grow. If you want to connect with me, you can feel free to do so. My contact information is at the very bottom of the show notes for this episode. The cover art of "8th Layer Insights" was created by Chris Machowski at ransomwear.net -- that's w-e-a-r -- and Mia Rune at MiaRune.com. [ Music ] Our theme song was composed and performed by Marcos Moscat. Until next time, I'm Perry Carpenter, signing off.
