The delusional side of AI therapy.
Dave Bittner: Hello, everyone; and welcome to N2K CyberWire's Hacking Humans Podcast, where each week we look behind the social engineering scams, phishing schemes, and criminal exploits that are making headlines and taking a heavy toll on organizations around the world. I'm Dave Bittner. And joining me is Joe Carrigan. Hey, Joe.
Joe Carrigan: Hi, Dave.
Dave Bittner: And our N2K colleague and host of the T-Minus Space Daily podcast, Maria Varmazis. Maria.
Maria Varmazis: Hi, Dave. And hi, Joe.
Dave Bittner: We've got some good stories to share this week. Before we do, we've got some follow-up here. I guess I will start with this one. So this is from a listener who wrote in. I want to say their name is Kajatan. Do you think I have that right? Any guesses on pronunciation?
Joe Carrigan: Kajatan.
Dave Bittner: Kajatan.
Joe Carrigan: Kajatan.
Dave Bittner: Maria. Kajatan?
Maria Varmazis: That would be my guess also. Kajatan.
Dave Bittner: All right. Apologies if I got it wrong. But they write in and say, Hi, Dave, Joe, and Maria. While listening to the latest episode and the statistics about travel scams, I remember one time some years ago when I performed a true miracle in Paris, France. While walking around Paris, I was approached by a woman with a note. She didn't say anything, just pointed at the note. The note stated in several languages that the woman was deaf and mute, and they collected name signatures for support of the Deaf and mute community in Paris. There were several names on the list already. Sure. Why not, I thought, and I signed the list. After I did that, the woman unfolded the paper, revealing a new column with a donation value. Next to each name, there was a donation of about 10 to 100 euros. She handed the note back to me, pointing at the numbers and expecting money. Oh, I said gladly. I took the paper and crossed out my name. The mute woman went furious and started screaming, screaming at me in some language that was not French. And that was the time I performed a miracle and made a mute woman speak. Keep up the good work. I love your podcast so.
Joe Carrigan: Well done, Kajatan.
Maria Varmazis: Yes.
Dave Bittner: Yeah. My understanding is, like, especially around the Eiffel Tower, like, it's just scammer Central in Paris.
Joe Carrigan: Right?
Dave Bittner: Like all kinds of --
Maria Varmazis: All of the tourist hotspots in Paris are --
Dave Bittner: Yeah.
Maria Varmazis: -- pretty much scammer Central. So bunch of places I avoid.
Joe Carrigan: I've got a cousin in Paris right now.
Dave Bittner: Yeah.
Joe Carrigan: I'm not worried about him, though. He's -- he's pretty sharp and pretty quick-witted.
Dave Bittner: Right, right. I mean, people are doing like the cups and balls tricks and I know the stealing lenses off of people's cameras. I've seen all those kinds of things. So beware.
Maria Varmazis: There's a -- there's a common one, at least it used to be in Montmartre, where the Sacré-Coeur Cathedral that has that beautiful view of the city where there are very aggressive touts that will try to tying bracelet around your wrist as you walk by and then basically be, like, I gave you a bracelet; you have to pay for it. And, like, they can be very physically aggressive. So there are some parts of Paris that I just completely avoid. And that is one of them, frankly.
Joe Carrigan: What happens if you get physical back? I wonder.
Maria Varmazis: I'm a 5'1" woman, so I don't want to know.
Joe Carrigan: I have an idea. I have an idea what happens is that like five or six of their friends show up, and they all beat the crap out of you.
Maria Varmazis: Yeah. That would -- I would imagine. I don't want to -- I don't want to find out firsthand, so the answer is I just avoid those places.
Dave Bittner: Yeah.
Maria Varmazis: But I think rule number one in any majorly touristed city is anyone approaching for any reason is not to be trusted.
Joe Carrigan: Right. They're suspect.
Maria Varmazis: So just keep walking.
Dave Bittner: It's so strange because the Parisians are known for being friendly to Americans.
Joe Carrigan: Right. So, yeah. That -- that would raise my -- my red flags immediately. Oh. I'm in Paris, and someone's being nice to me, an American.
Dave Bittner: Yeah.
Joe Carrigan: Hmm.
Dave Bittner: I don't know, Joe. You think they'd peg you as an American? How long do you think that would take?
Maria Varmazis: Oh, five seconds. Yeah.
Joe Carrigan: Everywhere I go.
Maria Varmazis: Yeah.
Dave Bittner: You'd be lumbering around with your baseball cap and -- or not your baseball cap -- your cowboy hat, your cowboy hat.
Joe Carrigan: Cowboy hat, yeah.
Maria Varmazis: Oh, my God. They would love you, Joe. They would love you. I must be the -- one of the only Americans who had a really nice time in Paris and had no issues at all with Parisians. I lived there for six months. I thought everybody was very nice.
Joe Carrigan: That's nice. You live there; they get to know you.
Dave Bittner: Yeah.
Maria Varmazis: It wasn't the same person every time. All my interactions, like, my interactions were generally very nice. So I don't know. I did not have that rude Parisian experience at all. I thought everybody was great.
Dave Bittner: You were probably just
Maria Varmazis: But I also lived in New York City before then. So I was, like, it's a city. People are city people here so.
Dave Bittner: I was going to say you're putting out all sorts of Greek vibes, so that was your camouflage.
Maria Varmazis: Oh, I really wish that was the case. But I was very -- I'm very obviously American. I'm very, very obviously American. I got clocked immediately so -- and it's fine. Yeah.
Dave Bittner: All right. Well, let's move on to our stories here. Maria, you are up first this week. What do you have for us?
Maria Varmazis: Well, I'm starting with some listener feedback because I thought this was a really interesting email, and I wanted to follow up on their prompting. So here's the email we got. Hi, guys. Love the show. And that's the reason I wanted to use this email.
Dave Bittner: Right.
Maria Varmazis: I have a story idea about an issue that's happening here in the great White North. We have lots of folks that pay by card when going to shops or restaurants. Many have these portal POS terminals. And that's point of sale, everybody, not the other meaning of POS.
Joe Carrigan: Right. Still like my card.
Maria Varmazis: And many of them are left unattended. Shops have been defrauded because they haven't taken the time to change the default access password on the terminal. Fraudsters are picking up the terminal and issuing unauthorized refunds to themselves, sometimes in the thousands of dollars. CTV is a national broadcaster here in the great White North. And he means Canada. I want to make sure I clarify. And they have a story here, and I'll follow up on that in a second. There are many other stories, but this is just one example maybe worth investigating. Regards Rob, aka Crow Child Bob. Thank you, Crow Child Bob. So I went to the link that he sent, and I did a little reading. And I just -- I thought this was a really interesting story because I had not been hearing much about this either. So those point of sale terminals in North America are frequently Square, Clover, Toast. The -- these have been common in a lot of the world for a long time. But, you know, the -- you tap or swipe your credit card, and they're pretty mobile. They're not necessarily attached to the counter -- the counter anymore.
Dave Bittner: Right.
Maria Varmazis: You know, they do the transaction for you. And they're the things that frequently they'll flip them around for -- with the screen say, Please give a 25% tip for this coffee that they just handed to you. That kind of thing.
Dave Bittner: Right.
Maria Varmazis: So these are those kinds of terminals Rob's referring to. And the story that he sent from CTV was /piqued my interest because it starts off with a place called Souvlaki Hut in Toronto. So I was like, okay. This has got to be maybe in Greektown in Toronto. And a customer, a fraudulent customer issued himself a $2,000 refund on a point of sale terminal at, again, Souvlaki Hut, where maybe you're spending $4 a sale.
Joe Carrigan: Right. Who buys $2,000 in souvlaki, aside from me.
Maria Varmazis: You have not met my family, Joe.
Dave Bittner: Maria.
Joe Carrigan: I love souvlaki. It's good stuff.
Dave Bittner: Can I just admit I have no idea what souvlaki is.
Maria Varmazis: Meat on -- meat on a stick.
Dave Bittner: Meat on a stick.
Maria Varmazis: Real meat on a stick.
Dave Bittner: Oh, that's -- okay. All right. Well, that sounds like something I'd enjoy
Maria Varmazis: Grilled pork, grilled chicken, grilled beef, grilled lamb on a stick.
Dave Bittner: So it's like Greek shish kebab?
Maria Varmazis: It is exactly, essentially, exactly like that.
Dave Bittner: Okay. Sounds delicious.
Maria Varmazis: It is comfort food.
Dave Bittner: Yeah.
Maria Varmazis: Yeah. It's comfort food. So the -- this specific situation with a $2,000 reef -- fraudulent refund, the criminal in this case picked up the terminal to hide what he was doing. So I guess he was doing this in semi plain sight of the store owner and then actually, while still in the store, gave himself that refund, which that's kind of ballsy, I've got to say. And the owner's son, when he spoke to CTV, said they had no idea that the terminal could do that without their explicit permission. So they didn't even know this was a thing, that this was possible. And a second business nearby to Souvlaki Hut in Toronto, the Pippins Tea Company, had a similar scam happen to them. A guy pretended to buy a teapot from them and then refunded himself through the point of sale terminal $4,900. That -- that is -- that is put the business out of business kind of money, depending on how small that business is. I mean, that is a lot.
Dave Bittner: Yeah.
Maria Varmazis: So the interesting thing about the CTV story was a lot of the people they spoke to who are store owners, again, didn't know that this was possible; had no idea what protections were in place, if any, or what they should do to prevent it because, in a lot of cases, it's just there's a default setting or password. And the point of sale owners, those store owners, don't know that they're supposed to change it. And this is a -- for our friends in IT, this is a familiar story. Or for people who are familiar with, like, IoT in the home. We always say first thing you do when you bring it home is change that default password.
Joe Carrigan: Right.
Maria Varmazis: That information doesn't seem to have trickled to our friends in the retail world who are small business owners, especially in this situation. So a security expert that CTV spoke to named Claudiu Popa said that most of the POS terminals sold to small businesses are misconfigured from the start. So thieves are like, Yay. This is great for me.
Joe Carrigan: Right.
Maria Varmazis: And, apparently -- and, apparently, according to law enforcement in Toronto, this is a rapidly growing trend in the city. I'm sure in many cities all over North America we're seeing this. And normally, when we see point of sale fraud, spoken about at least since 2020, especially like since COVID, a lot of this discussions have been around ransomware or malware, these terminals being vulnerable that way. But it seems that the actual physical scam of going into the store and either giving yourself a refund literally while the store's owner is turned or even breaking into the store after hours and accessing the point of sale terminal when nobody's there is -- that's a -- that's a big growing thing. So the recommendation to store owners is to make sure they lock away their terminals, change pins weekly, keep them out of reach when not in use. So really keep an eye on those terminals. And, of course, change the default password. But do you know how to do that? Is it easy to do? Is it obvious how to do? Who knows. And the vendors and the victims in this case say that they really wish that the default passwords were safer. But, again, they should change those. Or at least they would require two-factor authentication for things like refund limits, which sounds like a good idea.
Dave Bittner: Right.
Maria Varmazis: So, yeah. It's -- to be clear, again, this is outright fraud --
Joe Carrigan: Right.
Maria Varmazis: -- to give yourself a refund like this, and it makes -- it just reminded me of a thing I think we did talk about a little while ago, the TikTok ATM hack that was going around where people were just --
Joe Carrigan: Hacked in the airport.
Maria Varmazis: Do you remember this where they were just --
Joe Carrigan: Oh, I remember it keenly.
Maria Varmazis: Yeah. Yeah. This -- this straight-up check fraud that was going around virally on the -- on TikTok is like, Hey. This is the way you can get free money from an ATM. And people were like, No. Actually, that's just fraud. And that's a crime. Actually, Joe, I believe you said that. It's straight-up a crime. I remember you saying, like, Let's be clear. This is a crime.
Dave Bittner: It's one weird trick.
Maria Varmazis: One to get yourself arrested immediately. So, yeah. It's just -- it makes me think of that, like, this is -- this is a crime. This is not like one weird way to get money from a small business that really cannot spare that cash. So it's just very interesting that, you know, maybe as these terminals are getting hardened against cyberattacks, the physical attacks become the preferred vector or all at the same time. Who knows. But it's just very interesting. So thank you, Crow Child Bob, for pointing this out.
Dave Bittner: You know, what I'm thinking about here that I'd never considered is that that little terminal, which, in my mind I'm picturing something like a -- that looks like an iPad.
Maria Varmazis: Yeah.
Dave Bittner: You know, and it's sitting there on the counter. And a lot of times the person behind the counter will enter in whatever they need to enter. And then they'll flip it around, right? It's on some kind of swivel.
Maria Varmazis: And then walk away.
Dave Bittner: Yeah. But even -- even if -- if it's attached to the counter, they'll flip it around. And they may not necessarily be able to see what you're doing on the screen anymore.
Maria Varmazis: Yeah, yeah. Yep, yep.
Dave Bittner: And so I think that, if you know the guts of this thing and the secret incantations to get under the hood, there's your opportunity.
Maria Varmazis: Yeah. It -- many times I've been in that situation, and the person's busy. You know, they're probably doing five things or the job of five people. So they flip that screen around. And they walk far away, and there's nobody else around. And there have been times in my life where I have been near a computer that is open or a terminal that is open. And I've often wondered, what does this do if I touch that button. And I go, Oh, I now have admin access to this thing. Isn't that fun.
Joe Carrigan: I completely empathize with that. And sometimes I will actually have to put my hands in my pockets to make sure I don't touch anything.
Maria Varmazis: Yeah, yeah.
Dave Bittner: Yeah.
Maria Varmazis: It's -- some of us just naturally gravitate to that, and I know not to get myself into legal trouble. I obviously don't steal from anybody. That would be very wrong.
Dave Bittner: It's so hard.
Maria Varmazis: But, you know -- but sometimes I'm just, like, it is so easy. And I just -- it makes me a little scared and sad that these are not -- and it's -- it should not be the burden of, you know, your local coffee shop to become an IT security expert. Like, these -- these things should be locked down.
Dave Bittner: Right.
Maria Varmazis: So, yeah. It's just, oh, gosh. Don't walk away, guys. Just keep an eye on them.
Dave Bittner: The other one I think of that I see all the time is if I'm in a large store like a Target or a Macy's or something like that where there are people who are out there stocking shelves or rearranging things, checking pricing, labeling stock, all that kind of stuff. They will very often have a little handheld device that assists them in doing that. And maybe it prints labels. Maybe it scans barcodes. You know, they're -- they can put in the amount of inventory that there is out on the floor, that sort of thing, using this device. And there's -- you see this device; you recognize it for what it is because there are probably only a handful of variations on this that retailers use. But, boy. Do those stock people leave them lying around the store a lot.
Joe Carrigan: Right.
Dave Bittner: You know.
Maria Varmazis: Yeah, they do. Yeah. And I know we shouldn't mess with these things. I understand that. But if --
Dave Bittner: I know.
Maria Varmazis: -- but humans be humans.
Dave Bittner: I really want to play with one. I want to know. I know.
Maria Varmazis: And it's been sitting there, you know.
Dave Bittner: Right. Shame on you, right?
Maria Varmazis: Press the button.
Dave Bittner: What choice do I have? It's just laying there. There's no one around.
Joe Carrigan: I was in one of our local stores one day. And they had this, like, a tablet that normally would display something. But whatever app it was had crashed and left Windows -- I think CE. Was that the edition that was, like, the embedded systems edition?
Dave Bittner: Oh, wins. Yeah.
Joe Carrigan: Yeah. Why wins.
Dave Bittner: Yeah.
Joe Carrigan: Is that what it was called?
Dave Bittner: Well, that's what some of us called it.
Joe Carrigan: Right. And it's just sitting there. And I'm like, What's -- what's going on here? I pushed the start button. Now, the screen didn't work. The touch screen did not work so.
Dave Bittner: Yeah. Thwarted.
Joe Carrigan: I was thwarted. But, if I had my wireless -- or my keyboard with me, I could have just plugged it into one of the USB ports. It was just sitting there, I suppose.
Dave Bittner: Right, right. I don't know, Joe. I expect, if you're walking around with your usual backpack --
Joe Carrigan: Yeah.
Dave Bittner: -- you -- you have a keyboard in there, right?
Joe Carrigan: I might. Yeah. I might. Sometimes I do.
Maria Varmazis: Maybe an antenna. Maybe some other things, you know.
Joe Carrigan: There's an antenna sticking out of my bag right now.
Maria Varmazis: Oh, well, yeah.
Dave Bittner: Yeah, yeah.
Maria Varmazis: When you go to DEF CON -- no. I'm just kidding. How popular are you when you go to DEF CON.
Joe Carrigan: I've never been.
Maria Varmazis: Really?
Joe Carrigan: Yeah. I want to go bad.
Maria Varmazis: Yeah.
Dave Bittner: No. When Joe's backpack is so big that, when he's driving around in his car on the highway, he has to pull over into the weigh stations --
Joe Carrigan: Right.
Dave Bittner: -- because of how much stuff he has in his backpack. It's that big.
Maria Varmazis: How's your scoliosis, Joe? Is it --
Joe Carrigan: Well, the backpack pulls it straight.
Maria Varmazis: Oh, there you go. Okay.
Dave Bittner: Yeah. All right. We will have a link to that story in the show notes. I'm up next here, and my story comes from the folks over at Ars Technica. And this is about people using large language models, the AI systems, for therapy. Before we dig in here, let me ask either of you, do either of you ever interact with these models in a purely conversational kind of way?
Joe Carrigan: I try to, yes.
Dave Bittner: Okay.
Joe Carrigan: Although I've found lately that ChatGPT has changed the voices to make them hesitate and use inarticulate speech like uh and um, enough to make them seem more relatable.
Dave Bittner: Yeah.
Joe Carrigan: And it does not work with me. I don't like it.
Dave Bittner: Oh, interesting.
Joe Carrigan: Yeah.
Dave Bittner: Okay. Maria.
Maria Varmazis: I just want to ask a clarifying question about your question. Are we talking about mental health models specifically or just --
Dave Bittner: No, no. I'm just saying in general. Like, you ever just, you know, strike up a conversation to see how it does. Or, you know, I've heard of people who just -- like, they'll -- they'll kill time during their morning commute just by chatting with the AI.
Maria Varmazis: You know, I do not. But I think that's because, in the '90s when AI -- not AI. These are just regular old AOL chat bots and the like were around. I think I got that out of my system back then. And I just -- I didn't find it very fulfilling. So that temptation is not there for me anymore. I know they're much more sophisticated. But I just -- I don't know. I just don't feel curious about that. I don't know why.
Dave Bittner: Yeah, yeah. I haven't, either, any more than just playing around with it deliberately, you know, here at my desk. I haven't -- I haven't used it for conversational time killing, anything like that.
Joe Carrigan: Oh. Well, I want to clarify what I said. I don't use it for conversational time killing, like, in asking like, just, Hey, how you doing?
Dave Bittner: Yeah.
Joe Carrigan: I have a specific question in mind, and I'll ask clarifying questions and drill down on it in a conversational manner.
Dave Bittner: I see.
Joe Carrigan: But I don't know that I've ever actually sat down and -- and, like, talked to it like a person.
Dave Bittner: Right. Well, lots of people do, for better, for worse. And some researchers at Stanford recently presented at a conference. This was the ACM Conference on Fairness, Accountability, and Transparency. And they presented on how these large language models do when responding to people who have mental health conditions. And they tested the AI systems using fictional scenarios that reflected serious symptoms, things like suicidal ideation and delusional thinking. So some, you know, heavy, serious stuff.
Maria Varmazis: Wow. Yeah.
Dave Bittner: And what they found is that the models often failed to follow accepted therapeutic guidelines, and they would sometimes validate harmful beliefs or offer advice that could make things worse. And this isn't just the general ChatGPTs of the world. This is also the chat bots that are designed to provide therapy, right? They're supposed to be tuned.
Maria Varmazis: Oh, my God.
Joe Carrigan: I don't think that's a good idea to have a chat bot that's designed to provide therapy.
Maria Varmazis: But a lot of people use them, Joe. A lot of people who can't afford or cannot find a mental health provider in person, they rely on these. I know several people who do. And, oh, boy. I've got some phone calls to make after this episode because I did not know this. That's alarming.
Dave Bittner: Yeah. I mean, and to be fair, I mean, I think there are plenty of people who have -- who find serious value in this, right, legitimate value that it is someone to talk to. It's someone who's not going to judge you. You know, it's -- there -- it's always available. I remember early on, Joe, and you and I have talked about this way back on this show. There was a story when Siri first came out for the iPhone, that there was a young man who had some developmental issues.
Joe Carrigan: It was -- I think it was autism.
Dave Bittner: Yeah.
Joe Carrigan: He was on the spectrum.
Dave Bittner: And -- and he -- and the thing that was so great about Siri is that she had endless patience, right? This kid was basically a motor mouth. And it was just talk, talk, talk, talk, talk and was wearing the rest of his family out. God bless them. I mean, they were doing their best, and they were trying.
Joe Carrigan: Right.
Dave Bittner: But the bot had endless patience. And so, in that case, it were -- it was a good thing for everybody. And so, obviously, as these things have gotten more advanced and more capable and more real seeming, there are several that have been spun up that are designed to help you with -- in a therapeutic kind of way. But this study found that they -- they don't always do well. They said they tend to validate harmful beliefs, which they call sycophancy, like, I guess sycophants.
Joe Carrigan: Sycophancy.
Maria Varmazis: Sycophancy, sycophants.
Dave Bittner: Sycophancy. Yeah. Potentially reinforcing people's delusions or conspiracy theories.
Joe Carrigan: You are right, Joe.
Dave Bittner: Yeah. Everyone is out to get you.
Joe Carrigan: Right. I knew it.
Maria Varmazis: That tin foil hat of yours looks great.
Joe Carrigan: Right.
Dave Bittner: Yeah. And there are cases in the media where this has contributed to suicide or, you know, fatal incidents where -- there was a case where a man who had schizophrenia was killed by police after interacting with ChatGPT.
Maria Varmazis: Oh, my gosh.
Dave Bittner: So they're putting the -- sort of the warning flag up about this. A couple other little details. They said that it doesn't seem like the newer or larger models are any better at this than the older, smaller models were. They're looking for critical evaluations and better safeguards. But they're saying that they don't want to abandon the notion of AI in mental health, that there is a part that it could play; but we just need to be able to put better guardrails on it.
Maria Varmazis: Yeah.
Dave Bittner: And this was the real take-home for me. They said there's a mismatch between AI's goal to please and therapies' need for tough conversations and reality checks.
Maria Varmazis: Yeah. Yep.
Dave Bittner: And, to me, this really tracks with --
Joe Carrigan: Right.
Dave Bittner: -- because you're -- my experience with these AIs is like, you know, Hey, ChatGPT. The sky is red. And the ChatGPT will say, Oh, Dave. You're right. You're so smart. The sky is red.
Joe Carrigan: Right. Yeah. I've had that happen with me as well.
Dave Bittner: Right.
Joe Carrigan: Now, normally I am right about something; and ChatGPT is, what do they call them, confabulations? Now they don't want to call them hallucinations.
Dave Bittner: Yeah. Confabulations.
Joe Carrigan: And ChatGPT will be confabulating about something. But I will -- I'll correct it. There was one time I asked it who -- who invented something and knowing full well it was me, and I hold the patent on it.
Dave Bittner: This is a ChatGPT version of googling yourself.
Joe Carrigan: Yes. Exactly.
Maria Varmazis: Who is the brilliant person who made this thing?
Joe Carrigan: It came back with --
Dave Bittner: The singular mind from which this invention was spawned.
Joe Carrigan: Right. But here's the thing.
Maria Varmazis: Yes [inaudible 00:23:46].
Joe Carrigan: Here's the weird thing about that conversation is, after -- after I gave it the patent number to go look at, it went, Oh, that's you. And I had -- you know, I don't know whether I've ever conversationally told ChatGPT who I am.
Dave Bittner: Oh. But it knew.
Joe Carrigan: But it knew.
Maria Varmazis: You can check the -- its memories, though. You can check to see what it knows about you. So you can find that out.
Joe Carrigan: Well, they -- the company, OpenAI, has my -- has my information. They have my billing address and all that stuff because I do pay 20 bucks a month for that service. I think it's worth it.
Dave Bittner: Yeah.
Joe Carrigan: I get a lot of use out of it. And, you know, it really does help. Like, at my office the other day, somebody actually accused me of being someone who hates not knowing something, which is 100% correct. I cannot stand not knowing something.
Dave Bittner: I was going to say that tracks.
Joe Carrigan: And when -- when I am in the car with my son or my wife and I'm driving, which I usually am, and I have an idea, we're having a conversation and I want to know something, they both take great delight in going, Yeah. I don't know and then doing nothing else.
Dave Bittner: Oh. They hang you out to dry.
Joe Carrigan: They hang me out to dry. Right.
Dave Bittner: Intentionally.
Joe Carrigan: And I'm like --
Dave Bittner: Oh, my. What torture.
Joe Carrigan: -- can you -- can you Google that and read to me what it says? And they're like --
Maria Varmazis: They know that button depressed you. They know. Exactly.
Dave Bittner: That's diabolical, Joe.
Joe Carrigan: Right. I am not your -- but now I don't even -- I just turn on ChatGPT in voice mode. And I ask it a question, and I get my answer.
Dave Bittner: Right.
Joe Carrigan: And I go, There.
Dave Bittner: So that, when the day comes, that --
Maria Varmazis: You get an answer.
Dave Bittner: When the day comes that you're up in the -- in the clock tower with a high-powered rifle and people are asking, What happened? What -- what possibly could have made mild-mannered Joe --
Joe Carrigan: Mild-mannered Joe.
Dave Bittner: -- turn to these evil actions?
Joe Carrigan: That might be the first time anybody's ever called me that.
Dave Bittner: You'll be up there screaming, They wouldn't give me the answer! They wouldn't give me the answer!
Joe Carrigan: I mean, that is one of the things -- I'll tell you, Dave, that will set me off in a -- in a conversation. That will set me off in -- you know, if somebody just says -- you know, unless there's some kind of reason for not knowing, like, Hey. That -- I can't tell you that because proprietary information, or I can't tell you that because it's -- you know -- you know, I have a lot of friends who work in defense. I can't tell you that. It's classified. I can't tell you that for XYZ. A good reason, okay. I can't know it. Even, Hey, I can't tell you that because it's personal.
Dave Bittner: Yeah.
Joe Carrigan: Okay. That's fine. I can respect that. But I just don't want to tell you or I'd rather you didn't know that information and just remained in the dark about it -- and, generally, I'm talking about general information, right? You know, I want --
Maria Varmazis: Not what happened before the Big Bang kind of existential questions, right --
Dave Bittner: Right.
Maria Varmazis: -- like, where this is torturing you day and night to know.
Joe Carrigan: Yeah. I mean, actually, what happened before the Big Bang, that would be a good question to have an answer to. Unfortunately, we don't have an answer to that, and probably never will.
Maria Varmazis: When you get an answer to that, Joe, please share it with me. Nobody else. I would very much like to know.
Dave Bittner: When I was in high school, I had a buddy who I would call regularly to, you know, call, call up your friend to get together, check -- check on things. This is how we used to do it back then before we had mobile devices. And my buddy had a younger brother who was quite a character and pretty subversive little lad. And so sometimes -- I remember one time I called, and I said -- my friend's name was also David. And his little brother answered the phone. And -- and he -- I said, Hi, Matthew. This is -- this is Dave. Is your brother -- is your brother David there? And he said, No. He's not here. And I said, Well, do you have any idea where he is? And he said, I can't tell you that. I went, What? He said, No. I can't tell you that. Okay.
Joe Carrigan: What did he mean? Did he mean that he didn't -- immediately, now I'm -- right? You've already got me -- I'm over here wiggling in my chair, Dave.
Dave Bittner: Right, right.
Joe Carrigan: What do you mean, you can't tell me that?
Dave Bittner: Yes. Exactly. I said, Well, why can't you tell me? He said, I can't tell you because I don't know. I was like, You jerk.
Joe Carrigan: Right. Wise ass.
Dave Bittner: Exactly. Next time I'm over there, I'm going to give you a Dutch rub or something. Your -- you got a gotcha coming.
Joe Carrigan: You got an atomic wedgie coming.
Dave Bittner: Yes. Exactly. You little smarty pants. Yeah. Anyway, so I can relate, Joe. I can relate.
Joe Carrigan: Right.
Dave Bittner: It's -- that's not a fun position to be in, especially when somebody's yanking your chain.
Joe Carrigan: Yes.
Dave Bittner: All right. So back to the story here, which, of course, we will have a link to in the show notes, just be mindful. You know, like Maria said, check in with your friends if you know they're using these things, and just let them know. Remind them that, if things get dark and it seems as though one of these models is leading them down a bad path, that they do have real-life human friends that they should check in with and have their back because, evidently, the guardrails on these things are not quite robust enough to be fully trusted.
Maria Varmazis: Yeah.
Dave Bittner: All right.
Maria Varmazis: Reminds me of that IBM slide. Do you know the one that's very popular in AI discussion circles right now.
Dave Bittner: Yeah.
Maria Varmazis: The one from 1979. Yeah. The acute computer can never be held accountable; therefore, a computer must never make a management decision.
Dave Bittner: Right.
Maria Varmazis: So like a mental health decision.
Dave Bittner: Right, right. Yeah.
Joe Carrigan: Yeah.
Maria Varmazis: Always -- I always think back to that one.
Dave Bittner: No. It's good. All right. I tell you what. Let's take a quick break here to hear from our sponsor. We will be right back. And we are back. Joe, it is your turn. What do you got for us this week?
Joe Carrigan: Before I get to my story, I want to talk about something. My mom texted me yesterday. And she said, Hey. I got this message that says I owe toll money.
Dave Bittner: Oh.
Joe Carrigan: Scam. Don't even -- don't do it.
Dave Bittner: Yeah.
Joe Carrigan: Then today I got one, and it was sent to like a group of like 10 people.
Dave Bittner: Oh, yes.
Maria Varmazis: Yeah.
Joe Carrigan: Have -- I don't know if you've ever seen that happen with these toll scams.
Maria Varmazis: Yes.
Dave Bittner: Yeah. I recently got a group text toll scam.
Maria Varmazis: Oh, yeah.
Dave Bittner: Yeah.
Joe Carrigan: I responded to the text and said, You all probably already know this, but this is a scam. Do not click on any of the links.
Dave Bittner: Yes.
Joe Carrigan: So hopefully that went out to everybody.
Dave Bittner: Yes, yes. I have gotten those.
Joe Carrigan: Yeah. I don't know if that's helpful. I don't know if that -- that may have just roped myself in to more of these things. Who knows. Oh, this is a good number. And this guy likes helping.
Maria Varmazis: Yeah. I was going to say I don't know if I'd respond but --
Joe Carrigan: Right.
Maria Varmazis: -- thank you for your sacrifice, I guess.
Joe Carrigan: Yes, I guess.
Dave Bittner: On behalf of the rest of us.
Joe Carrigan: On behalf of the other 10 people in that group. So my story actually comes out of KARE. You know Western Mississippi radio stations and television stations begin with the letter K.
Dave Bittner: Yes.
Joe Carrigan: I find that very difficult to deal with, having grown up entirely and spent most of my life on this side of the Mississippi, I want to say WARE. But it's not. It's KARE Channel 11 in Minneapolis, coming from AJ Lago. I hope I'm saying that right. Lago, L-A-G-O-E.
Dave Bittner: Yeah.
Joe Carrigan: Kelly Dietz and Gary Knox.
Dave Bittner: Okay?
Joe Carrigan: And in 2020, 2020, Minnesota became the first state to offer a Medicaid benefit called housing stability services.
Dave Bittner: Okay.
Joe Carrigan: Okay. Now, this is a Medicaid benefit aimed at helping people avoid homelessness, typically targeting, like, older people, people who are dealing with addiction --
Dave Bittner: Okay.
Joe Carrigan: -- maybe going through addiction treatment and maybe some disabled people. When Minnesota planned this, they estimated that it was going to cost about $2.6 million a year, which doesn't sound like a lot of money. $2.6 million a year, I can make sure that people who need medical attention don't go homeless. Okay. Sounds like a good idea.
Dave Bittner: Yeah.
Joe Carrigan: Good investment.
Dave Bittner: Sure.
Joe Carrigan: First year, first full year of billing in 2021, the billing was $21 million, almost 10 times what they anticipated. In 2024, the billing was $104 million. Now you are talking something that is two orders of magnitude bigger than what was originally planned. I mean -- and I know. Every time there's a government program, it doesn't matter which organization the government is -- is doing it, Department of Defense, Department of Health and Human Services. They all have this number that they tell you. And then couple years down the road that number is much bigger.
Dave Bittner: Sure.
Joe Carrigan: But this is unprecedented in terms of, you know, you're not -- you're not talking about a small increase. You're talking about tours of magnitude in four years. So how is this possible? Well, there is an acting US Attorney. His name is Joseph Thompson. And he has an answer. He says, Minnesota has a fraud problem and not a small one.
Maria Varmazis: Gee.
Joe Carrigan: So -- yeah. Federal agents laid out examples of these, of huge bills being done by some of these -- some of these companies who are doing HSS. That's what I'm going to call it, the -- what is it called -- housing -- Housing Stability Services. I'm just going to say HSS from now on --
Dave Bittner: Yeah.
Joe Carrigan: -- where they're reportedly receiving, like, large sums of money, like 40 and $52,000 for clients in -- in a year, 40 to $52,000 helping clients find housing. So, I mean, really, if you're going to spend $40,000 to help somebody find housing, you just go, I rented you this apartment. Here it is.
Dave Bittner: Right, right.
Joe Carrigan: Live here. And, you know, it'll cost less than $40,000 to do that. But they executed a warrant, a search warrant on these companies. And, according to the warrants, these companies promised to help these individuals find stable housing as they finish their treatment programs. All too often, the companies do not actually provide any real assistance in finding houses. Instead, the companies simply use the information provided by their recruited clients to bill Medicaid for housing stabilization services they do not actually provide. So the federal investigation identified 22 different HSS providers in this particular warrant. And they said that they served a warrant on these place -- on these companies, many of them in the same building, right? So they all have the same address, just different suite numbers in the address. And these providers collectively received more than $8 million in Medicaid payments for housing stabilization services from January to May, January of last year to May of this year. So, in 18 months, almost 18 months, they have exceeded the original plan cost in fraud by like three times. Amazing. And then they have these companies. They list them out. I'm going to name them. This is all, of course, alleged. But -- because it's a warrant and a search is going on. Brilliant Mind Services, LEO Human Services, Liberty Plus, Pristine Health, and Falad Care, Fal -- F-A-L-A-D Care, Incorporated. In one case, brilliant minds billed $2,000 for services they claimed to provide to a woman named Rachel. But KARE actually previously interviewed Rachel because this has been on their radar for a little while apparently. And she said she never received any services in -- from the company except for one occasion when a person named Mohammed Mohammed dropped off a Shark vacuum cleaner at a department that she had found for herself without the help of this company. So, hey. Thanks for everything. Here's a vacuum cleaner. I did something.
Maria Varmazis: Terrible.
Joe Carrigan: Yeah. But even after this happened, KARE, K-A-R-E, I'll call them KARE because they're actually a broadcast station. That's call letters, right?
Dave Bittner: Yeah.
Joe Carrigan: Right, Dave? That's a call sign.
Dave Bittner: I -- how much you want to bet that their local promos are 11 cares.
Joe Carrigan: It might be.
Maria Varmazis: Of course.
Dave Bittner: Right.
Joe Carrigan: We can't do that with our H -- with our amateur radio call signs, though.
Dave Bittner: No.
Joe Carrigan: Yeah. KARE discovered that Brilliant Minds continued to bill Medicare for additional household additional house-hunting services that they probably never provide. And there's one author that says in one of Rachel's entries in the system, I visited these properties. I went -- went in person on Rachel's behalf because these two options were strong matches with her housing criteria. It's a scam. It's a total scam, says Rachel. Rachel's Medicaid billing showed that another company, LEO Human Services, which is one of the companies in the list above, also billed the HSS program. And Rachel says, I never even interacted with those people. So they're sharing her information. They're billing the government back for these services that they're not providing. And then they're telling their buddies, which probably all just one organization, right, that they've stood up all these different --
Dave Bittner: Right.
Joe Carrigan: -- these different companies. And then now they're passing the information. Hey. I just billed Medicaid for $2,000 for Rachel over here. Pass it off to LEO. LEO will do it for another $2,000, and we could pass it off to the next guy; and everybody makes bank. So, yeah. This is a huge problem. In 2025, Minnesota passed a new law that has more stringent requirements on it. But it was -- it's amazing to me that, within one year of passing a law, there's already so much fraud going on in this -- in this system that -- that it's four times what -- you know, almost 10 times, actually, what you'd expect to pay.
Maria Varmazis: Well, it feels like anytime there's a new program the scams come up immediately, like, with the COVID PPP, the -- those funds.
Dave Bittner: Oh, yeah.
Maria Varmazis: How many of those -- yeah. How much fraud happened there?
Joe Carrigan: Tons.
Maria Varmazis: And they're still -- I think they were still chasing it down -- they're still chasing it down now five years later.
Joe Carrigan: Oh, yeah.
Maria Varmazis: It's -- just the whole time you're describing this, that's what I was thinking of. Yeah. Go ahead.
Joe Carrigan: That's a good point. I was saying that hopefully those people all get prison time. You know, these are -- these are taxpayer dollars.
Maria Varmazis: I think a lot of people got away with it.
Joe Carrigan: I think you're right. I think you're right. And that's unfortunate. I have heard of people getting -- facing serious ramifications for the PPP loans that they weren't entitled to.
Maria Varmazis: Yeah.
Joe Carrigan: On the -- on the other side of this, there's -- there's the issue of, like, with the PPP loans, there was a time-sensitive issue there. Now, with this, with -- with this law, I don't know that there is a time sensitive issue for $40,000 in services being billed. I don't -- I don't buy that. You know, maybe -- maybe there's a time sensitive issue. Okay. So this person needs to pay rent this month. What's their rent? The rent is not $40,000. It isn't. There's no way.
Maria Varmazis: Right. When you have housing involved, though, I imagine -- I'm just I'm thinking -- I mean, obviously, fraud is bad. Let's put that out there. I'm thinking, if I was trying to administrate a program like this where it's like, hey. This is giving money for people who need help getting housing when housing is always so hard to come by. Do we want to make this move as slowly as possible, which means it could be years before somebody gets housing help? Or do we want to sort of hand it out first and then retract it later if fraud has happened, rather err on the side of helping somebody than a little -- little more than a little less. I'm sure there's a lot of debate there, honestly. But I imagine that might be part of the struggle is, like, we have to act in a timely way, right?
Dave Bittner: Yeah.
Joe Carrigan: That is absolutely part of the problem.
Dave Bittner: I was -- while you were talking, I was trying to find some statistics here to try to figure out because every program is going to have a certain amount of fraud, right.
Maria Varmazis: Yeah.
Dave Bittner: Just like every store is going to have a certain amount of shoplifting --
Joe Carrigan: Right.
Dave Bittner: -- or, you know, damaged goods or those kinds of things. And the charts that I'm seeing here say that most public services have a built-in tolerance for between 1 and 3% fraud as acceptable risk. But when rates get above about 5%, that is when internal audits are usually triggered or policy reform or things like that. But seems like under 10% is typical for these sorts of things.
Joe Carrigan: Right.
Dave Bittner: And I think Maria makes an excellent point. You know, you -- how much fraud do you accept in exchange for not having too much friction for the people to get the services they need? And that's not always an easy answer.
Joe Carrigan: That is an excellent point.
Maria Varmazis: Yeah. Especially when we're talking about homelessness. Yeah.
Joe Carrigan: Yeah. I mean -- because that -- that will be a devastating impact to somebody. But, I mean --
Maria Varmazis: And you're talking -- especially in the north, when it gets cold, people could die and -- you know, with exposure. I mean, it's like this is -- this can be life or death.
Joe Carrigan: Yeah.
Dave Bittner: Yeah. So it's good that they're going after the fraudsters and not the people who are being, you know, falsely -- whose names are being put in to falsely claim these funds.
Joe Carrigan: Right. Yeah. I think that's -- that's an important distinction.
Maria Varmazis: Yes.
Joe Carrigan: You know, these people are, in their own right, victims. First off, they've got some -- some kind of health malady. And I'm going to go on -- I've said this a lot of times. In this country, we have to stop treating addiction like it's a morality issue. It's got to be treated like a healthcare issue.
Maria Varmazis: Agreed.
Joe Carrigan: It really does. And you know, when you -- when somebody is -- is addicted to a substance that is physically addictive, they just can't -- they can't just stop. It may kill them. Doing that may kill them. So there has to be treatment options for these people. So I don't -- I don't have a problem with treating people who have -- who are dealing with some kind of substance abuse issue. But, at the same point in time, I think that the people who are, /who are caught for these -- who are -- for these, I think the punishment for people who have defrauded this program should be severe, very severe.
Dave Bittner: Yeah. I'm for that.
Maria Varmazis: Yep.
Dave Bittner: All right. Well, again, we'll have a link to that story in the show notes. Joe, Maria, it is time to move on to our Catch of the Day. [ SOUNDBITE OF REELING IN FISHING LINE ]
Joe Carrigan: Dave, our Catch of the Day comes from the scam subreddit.
Dave Bittner: So this is someone who had an ongoing -- let's call it a relationship with a scammer. And they wrote up a description of -- of this. They said a patient scammer finally asked me for money after five months of texts and calls. Claims that they want to send me $700,000 profit from selling property because his funds are frozen as part of an inheritance lawsuit. So we've certainly heard of those before, right?
Joe Carrigan: Yep. Absolutely.
Dave Bittner: Sent me a suspicious video of a safe.
Joe Carrigan: Look at all this money in here.
Dave Bittner: This, inside this safe --
Maria Varmazis: Interior or exterior?
Dave Bittner: Right. I'm going to go with exterior. Right. I'm just going to -- send me a photo of Fort Knox so this could be -- all be yours. And then a tracking number from a shipper with misspellings. Of course, the package never came. Then he called and sent me a screenshot from the shipper saying they needed 15,000 euros for customs. I already knew this was a scam as soon as they wanted to send me a safe. This is the most efficient object to send and -- because safes are heavy.
Joe Carrigan: Yeah.
Maria Varmazis: I was going to say international shipping, weight kind of a thing. Yeah. Safe.
Dave Bittner: Yeah. He says, I'm disappointed I became emotionally invested in a fake, but at least I never gave them money or account numbers.
Joe Carrigan: Yep.
Dave Bittner: So this message reads -- and it says, Dear Miss Nicole, we hope this message finds you well. We're writing to inform you that your shipment from Berlin, Germany to Cleveland, Ohio, USA has arrived and is currently undergoing customs clearance at the US port of entry. As part of the clearance process, US Customs has assessed a total of 15,000 euros in duties and associated handling fees.
Joe Carrigan: Hold on right there. Why is US Customs accepting payment in euros?
Dave Bittner: Excellent question.
Maria Varmazis: Sshh. Don't ask. Don't ask questions. Don't ask questions.
Dave Bittner: Joe, do you want the $700,000 or not?
Joe Carrigan: I want the $700,000, Dave.
Dave Bittner: Then stop asking so many questions.
Joe Carrigan: Okay.
Dave Bittner: To avoid any delays or additional storage charges, we kindly request that this amount be remitted at your earliest convenience. If you have any questions regarding the payment or require assistance with the process, please don't hesitate to contact our team directly. We appreciate your prompt attention to this matter and thank you for choosing Net Light Shipping. And it's Warm Regards, Net Light Shipping GmbH, which is German for Incorporated, right? Something like that.
Maria Varmazis: Yeah.
Dave Bittner: It's a stamp for German companies' customer service department. So, you know, there's, obviously, there's --
Joe Carrigan: This is a trunk fox scam with extra steps.
Dave Bittner: Yes. Exactly. I like that in this request for the 15,000 euros they snuck in this thing about avoiding any delays or additional storage charges. So act now, or it's going to -- you're -- it's going to cost you even more money.
Joe Carrigan: The artificial time horizon.
Dave Bittner: That's right.
Joe Carrigan: And they let you fill that in.
Dave Bittner: Right, right,
Joe Carrigan: They don't -- they don't define it for you. They let you -- they let you worry about that.
Dave Bittner: Yeah, yeah. So 700,000.
Maria Varmazis: The attention to detail, though, is interesting. Like, the phone number is definitely a German area -- country code. I wonder if we put -- not that we should or would, but I wonder if the other here --
Joe Carrigan: Give them a call.
Maria Varmazis: I might, I might.
Dave Bittner: Maria plays pretty fast and loose with the links. I mean, I don't want to say anything; but word around town, Maria, is that you're pretty fast and loose with the links.
Maria Varmazis: I just love putting my hand on that hot stove over and over.
Dave Bittner: There you go.
Maria Varmazis: Yeah.
Dave Bittner: Right, right.
Maria Varmazis: The attention to detail throughout this, like, the -- everything looks convincingly like it should be from or to Germany. So not -- a lot of times it's more messy than that. But this actually looks consistent there, except for the, oh, Cleveland, Ohio, USA 15,000 euros thing that Joe picked up on.
Dave Bittner: Yeah. Right. Yeah. But imagine five months of text messages and calls between this person and the scammer, and it finally got to this point for them to cut it off.
Maria Varmazis: Yeah. They weren't --
Dave Bittner: I'm glad they cut it off.
Maria Varmazis: They weren't -- they weren't leading them on. They were actually -- got taken for those five months, sounds like. So that's good they recognized it.
Dave Bittner: Right. And she says, I'm disappointed I became emotionally invested in a fake.
Joe Carrigan: Yeah.
Dave Bittner: So this was a -- partially a romance scam.
Joe Carrigan: Right. But the -- probably. The interesting part is that -- the trigger that made her think it was a scam was the desire to send the safe, physically send the safe.
Dave Bittner: Yeah.
Joe Carrigan: That's how she knew it was a scam.
Dave Bittner: Right.
Joe Carrigan: Which is good. Good that she picked up on it.
Dave Bittner: Yeah.
Joe Carrigan: She got -- she -- she got off easy.
Dave Bittner: Yeah, yeah. Absolutely. All right. Well, we would love to hear from you. If there's something you'd like us to consider for the show, you can email us. It's HackingHumans@n2k.com. And that is Hacking Humans brought to you by N2K CyberWire. We'd love to hear from you. We're conducting our annual audience survey to learn more about our listeners. We're collecting your insights through the end of August. There's a link in the show notes. Please do take a moment and check them out. This episode is produced by Liz Stokes. Our executive producer is Jennifer Eiben. We're mixed by Elliott Peltzman and Tré Hester. Peter Kilpe is our publisher. I'm Dave Bittner.
Joe Carrigan: I'm Joe Carrigan.
Maria Varmazis: And I'm Maria Varmazis.
Dave Bittner: Thanks for listening.
