Is Your Enterprise AI Strategy Delivering ROI Yet?

Dr. Grace Trinidad: In addition to the token hygiene of your organization, I think AI FinOps, we're going to see a lot of AI FinOps energy in the next 12 months. We're going to see a slew of players emerge as a layer of security, even, because one way to attack an organization is to attack their token consumption, right? So if they blow their budget on a Wednesday, what do they do at that point? So that's another security layer. [ Music ]

Johnny Hand: Welcome to the AI Security Brief, where we're unpacking emerging threats, vulnerability research, and the strategic decisions that security leaders are making right now. I'm Johnny Hand.

Dustin Childs: And I'm Dustin Childs. Now, Johnny, for this episode, we were both on the road. I was in Berlin for Pwn2Own, and you were in Phoenix, where you had the opportunity to sit down with Dr. Grace Trinidad from IDC. I'd love to hear more about your conversation.

Johnny Hand: Yeah, it was honestly a dynamic conversation. So we were at our SPARK event, which is our customer-focused event, and Grace was able to attend. And we jumped right in with some really interesting conversations around AI adoption. She has some really good theories that she's publishing in her research. But one of the things that stood out the most, and probably resonated the fastest with me, was she used a term that I often use in the military, which is, "You have to embrace the suck."

Dustin Childs: Yeah, that's a term I used in the military as well. So I'd love to hear how she applies it to artificial intelligence. So let's just go ahead and dive into the conversation. [ Music ]

Johnny Hand: So, Grace, are we going to have realized optimization, realized benefit from AI?

Dr. Grace Trinidad: Yes. I'm optimistic that we will have realized benefits from AI. But I do think that the early stages of expectations should probably be more tempered. And so don't lose patience with your chatbots. Don't lose patience with your summarization tools. Don't lose patience with the sort of low-hanging fruit that we've got going on right now. Part of what you're doing is you're building the muscle, the AI muscle in your organization. You're building comfort. You're building trust with the product. So that's why hallucinations are such a big deal right now. There's such deteriorations of trust when they happen. So as we smooth out these things, then even more efficiency and productivity gains will come to the fore. But right now, this is sort of like nascent learning, everyone together in first grade, kindergarten classroom.

Johnny Hand: I love that you said, "We're building our muscles." I think it's such a good analogy because one of the challenges -- I wanted to say, like, what you should shout from the rooftop is really to the board members, that, "Hey, we're not going to realize true ROI. We're not going to have these net-plus benefits that are so easy to document because we are in a very, you know, foundational beginning of this process," right? So let's dive into that a little bit. You know, earlier we were talking, and I get the feeling when I talk to a lot of folks that they are doing very individualistic adoption, but they think they're doing organizational growth. Can you talk a little bit about that, what you're seeing?

Dr. Grace Trinidad: Yeah. Don't ignore the individual-level usage. We're going to need them for our ontology building, so encourage that. There are treats later on if you continue on that track. But we're -- so the individual experimentation -- you're -- part of the issue is we need our staff, we need users of AI to know when to trust AI and when to kick it to the curb.

Johnny Hand: Yeah.

Dr. Grace Trinidad: Right? And so, while this is a period of individual experimentation and trying your own prompts, developing your own prompts, developing your own tools, developing your own code, that's all just part of figuring out how wrong it can be and how right it also can be, and that calibration of trust, which we will need when enterprises are fully dependent on AI. So this is happening slowly, and under some criticism and duress. I don't think is a terrible thing, but don't lose sight of the fact that this is all part of the learning process to get to really good stuff down the line. And I think we're finally, at this point, where the AI security approach is mature enough that we can talk about it in, like, complete sentences and with, you know, shared ideation.

Johnny Hand: Yeah. I'm so glad that you brought up the topic of AI security because, when we -- when I talk with folks, I actually, like, made this kind of equation a while back, and you challenged me on this, which was -- which I love, by the way. And it was that I really saw AI adoption, even though it was moving faster, as kind of like the way that we did cloud adoption. You know, in this idea that we had, cloud came about, you had some early adopters, you started building in cloud. Mostly, it was like developers and, you know, sysadmins and architects. And there wasn't really a security presence. So we didn't really have visibility into that plane. And it was probably three or four years later when we really started building out and having a voice around securing cloud infrastructure. So when I look at the journey of AI, I kind of looked at it through that lens. But you actually challenged me to look at it through a different lens, and I'd love for you to share that.

Dr. Grace Trinidad: Yeah. Well, because while I completely agree that the cloud security story very much mirrors the AI security story, I think that the AI -- this AI era that we're entering in has so much more in common with 1996. It's 2026. It's amazing. It has so much more in common with 1996 than I think most of us recall. We're older.

Johnny Hand: So we're talking about the start of the internet.

Dr. Grace Trinidad: The start of the internet. So, you know, one of my colleagues messaged that they ran out of token on their foundation model of choice. And he's like, "It's like 100 hours of AOL. Where's the rest of my internet usage?" And that is such a -- it's such a funny thing to conceptualize now, like 100 hours of internet access. And, in fact, AOL, if you look at the history of that 100 hours of access, people were really burned because they quickly blew their 100 hours of internet access as one does. And we're kind of experiencing the same thing with AI, right? So the parallels are really clean. So in 1996, we had usage that people couldn't plan for. We had no idea how people were going to use the internet. And so this 100-hour disk was provided to people in their homes. Everyone blew their budgets. Some people got really, really bad bills. And then the entire pricing model changed with AT&T Net, right? And then that competition introduced a new paradigm, and now we have the internet that we have today. My goodness, can you imagine if we were billed by the hour? Terrible.

Johnny Hand: Crazy. Crazy. Yeah.

Dr. Grace Trinidad: But then the other parallel in '96 was the panic around energy consumption.

Johnny Hand: Ah.

Dr. Grace Trinidad: So this is where, like, it just -- it's like AI all over again. There was so much concern around how much internet usage was going to just blow the grid.

Johnny Hand: Yeah.

Dr. Grace Trinidad: And over, very quickly, actually, there was, like, all these quotes that it was going to take eight times the power. Like, we just didn't have the infrastructure because we could not -- we weren't able, as a globe, to support eight times energy usage, which was -- that was the -- that was what was forecasted at the outset. But, similar to AI models, as time goes by, compute gets cheaper. And so does token. Tokens also -- token consumption is also becoming more efficient with AI models. So we're seeing these costs, those original concerns about energy consumption and cost, we're seeing these things kind of follow the same trajectory as 1996.

Johnny Hand: Yeah, I honestly wasn't -- So, Grace, let's talk a little bit about that, you know, really around that token hygiene and, like, what organizations are really struggling with today. Because you mentioned, especially from the small to medium business, that we want to empower our employees and our folks to innovate and really ideate and use these tools, but there's a real cost behind it. And it's almost like saying, "Hey, go play, but at what cost?" And how are you seeing token usage, the hygiene of that, and maybe how organizations can help manage that a little bit?

Dr. Grace Trinidad: Yeah. So this is something that heavy users of AI are encountering, especially heavy users of agents. So the agents who are able to deploy their own workstreams and, in some instances, their own agents, they actually work very hard on your behalf, to your detriment as well. So, as an example, sometimes what happens is an agent will be set on a task, you'll give it your instructions, and it'll set on a task, and then it'll run into a roadblock. Maybe you don't have access to that particular dataset, or there's something -- something was set up wrong, that agent will try in every permutation to get through your organization to get to that resource. Bless their heart. But what ends up happening is the cost of that effort or that -- whatever it is you wanted to do, that cost -- that now costs 14 times what you had anticipated, right? Because now it's routed itself through the entire enterprise looking for what it needs to provide you to execute on your instructions, and it can't.

Johnny Hand: Yeah, it's a good parallel because you see this where it's like, "Oh, I might have a thousand token allotment," and then, you know, before you know it, they've spent $15,000 on something that wasn't worth a quarter of that.

Dr. Grace Trinidad: And so, where I'm seeing some pressure build up in enterprises right now is not so much in AI security, but in token consumption. And so, in that, you know, organizations are going to want to look at the hygiene of their models. So does it exclude unnecessary words? Or, you know, a lot of the foundation models are already optimized to not spend tokens when they're saying, "Thank you," or "You bet," right? But the smaller language models may not have those. So, in enterprise adoption, what we don't realize as we're using Claude or as we're using OpenAI is those foundation models have actually come with constraints to trim down token consumption. And so, organizations that are using will soon use domain-specific models, small language models, that's an area where you're going to want to probe. What is the token hygiene of that model itself, right? So, in addition to the token hygiene of your organization, I think AI FinOps, we're going to see a lot of AI FinOps energy in the next 12 months. We're going to see a slew of players emerge as a layer of security, even, because one way to attack an organization is to attack their token consumption, right? So if they blow their budget on a Wednesday, what do they do at that point? So that's another security layer.

Johnny Hand: So, Grace, earlier we were talking about disruption and recognizing that we're in this phase of disruption, we're not in the transformation phase. And I think there's probably industries that showcase this better than others. What are you finding in your research?

Dr. Grace Trinidad: So, I didn't make up the experimentation, disruption, stabilization, and then transformation phases. We've seen these through other digital transformation projects, so my background is actually in healthcare information assurance. And one of the -- one of the biggest transformations is introduction of electronic health records into hospitals and healthcare systems. And that transition, on the heels of the HITECH Act, occurred -- what? -- 2010 through 2015. And we had a slew of retirements because that's a really, really tough thing to introduce into a physician's workflow. Now, you have EHRs that you have to interact with. People just full-on left. But more tragically, mortality also went up. So there's research that shows that pediatric mortality in intensive care units went up 136% once electronic health records were introduced. Yeah, that's not a great number. And can you imagine if I came to a business leader with that number and said, "If you implement this, mortality is going to go up 136%, but it's going to be great"? No one would have implemented it. Yet look at where we -- where we've come as a result of EHR implementation. We have -- it's easier to ship your records. It's easier to examine your records. We have better surveillance as a result of it, better data tracking.

Johnny Hand: Correlation, right?

Dr. Grace Trinidad: We have -- yeah, exactly.

Johnny Hand: Better correlation.

Dr. Grace Trinidad: Better correlation. This is, again, if you want to go back to ontologies healthcare. So that's not the only digital transformation that occurred. Healthcare is a really, really great example of one. It's a very disruptive one that did pay out in the end. Very painful, but very transformative. The other transition that occurred -- there's a number, but the other transition that I really like to talk about is aviation.

Johnny Hand: Yeah.

Dr. Grace Trinidad: So, I just got off of a plane today. You got off of a plane yesterday. We get on planes every single day, and we trust that we're going to get to our destination. A lot of that is fly-by-wire now. So that's automation that we have as a culture, have come to trust. But it wasn't always trustworthy. You know, we lose sight of this. Aviation went from pilots being the primary driver to pilots being a supervisor of the plane's automation, with training to intervene when expertise was necessary.

Johnny Hand: It's the parallel to AI's.

Dr. Grace Trinidad: Kind of amazing.

Johnny Hand: So crazy. Yeah.

Dr. Grace Trinidad: Right? So our pilots today are not just trained in how to fly a plane. They're trained in human factors research. They're trained in a concept called automation surprise, and that's when an automated system does something that you're like, "What is this?"

Johnny Hand: "Why did it do that?"

Dr. Grace Trinidad: You have the tools, the cognitive toolset to sort through what might be happening, so you can break down the problem, right? So that's how pilots are being trained now. Human factors, automation surprise, and how to maintain supervision over an automated system while still being engaged with that system, right? That's the tough part. Like, how do you not just fall asleep at the wheel?

Johnny Hand: Yeah.

Dr. Grace Trinidad: And so, that's what a lot of the training of pilots is in now. And I do want to emphasize, like, we didn't lose pilots. We didn't just start handing controls to me, anyone, you know, some random person, and say, like, "This is automation now, you can fly it." We absolutely didn't do that. We still have highly trained pilots, who must operate this very, very technical machinery that, for the most part, runs itself. And the same is going to happen with AI. We need to train pilots to stay in the loop, but out of the weeds.

Johnny Hand: Yeah. [ Music ] So, as we kind of wrap up for today, if you had to give three or four key takeaways for our audience and those security leaders that are looking at, "Well, how do I navigate through this disruptive phase, understanding I need to secure my organization?" Are there any key takeaways that you would just, you know, want to impart with them?

Dr. Grace Trinidad: Yeah, absolutely. So if we go to those historical digital transformations, they actually offer very, very good advice. In industrial automation, research was so clear. The firms that enjoyed the best productivity increases and the best efficiency were the firms that rearchitected their whole workflow around that automation technology, rather than inserting that automation technology into existing industrial line operations. They reworked everything.

Johnny Hand: So no bolt-ons.

Dr. Grace Trinidad: So no bolt-ons. So the parallel here is, if you're enabling AI in your enterprise, where do you think it actually fits best? So where is there actually a problem to solve? And how would you actually reformat that to better take advantage of that technology? So that'd be my first thing, is like, look for -- look for places where the technology makes sense, or change systems, or change your organization to make the technology make sense. That's a tall order, but it is part of how digital transformations unfold. And then, kind of next steps, just to go back to token hygiene and then to the ontology that we're talking about. For ontology creation -- and I think this is the next decider -- or this is the next differentiator of business. I really do believe that ontologies will be the next differentiator of business. But I really do want every CTO or data scientist in an enterprise to think about what ontological structure that enterprise might actually benefit most from. There are a number that exist in the world. Basic Formal Ontology is one, and then there's OWL. So there are a number of places to start. So, for this year, I think the real heat in this industry will be around semantic intent ontologies, and then, token consumption and AI FinOps. And I think we're getting to a place where enterprises are really having to square that cost with what they're able to achieve.

Johnny Hand: And that just goes back to what we talked about, is the need for security leaders to partner more tightly, become better understanders of the business.

Dr. Grace Trinidad: Yeah, absolutely.

Johnny Hand: And really drive, you know, a relationship that becomes a business enabler.

Dr. Grace Trinidad: Yes. Yeah. And then think about, like, structuring an ontology. You have to be well acquainted with -- or you have to have some way of getting well acquainted with each area of the business in order to build that real differentiator.

Johnny Hand: Yeah, and I love the fact that -- I think, we did this with security and we realized it wasn't great, and we kind of rearchitected, but that it can't be bolt on. We have to natively build with these technologies so we can natively secure them.

Dr. Grace Trinidad: Yeah. Yeah. And that's just been -- that's part of the experimentation disruption cycle, right? We're sorting it out, and we're just -- we're firmly in that disruption.

Johnny Hand: Well, thank you so much for joining us on the AI Security Brief podcast. It was a real pleasure having you, and hopefully, we can have you back on the show. [ Music ] All right, Dustin, we're back. I told you it was going to be a dynamic. What did you think about Dr. Grace?

Dustin Childs: I thought she made a lot of really good points, and it was really interesting to get her perspective, especially when it comes to the disruption phase of organizations implementing AI. And really, we're still all in the experimentation phase, and it's messy, and that's normal. And it's great for her to describe it in that manner.

Johnny Hand: Yeah, I loved it. It was messy. It's painful. We have to embrace the suck and get on through it. You know, another thing that she talked about, which really stood out, is, you know, the fact that organizations just aren't in this mature area of transformation, that we actually are in the very early stages of those disruptions, and that's totally okay. I love that we -- when we went into this conversation, I think -- probably, if I'm being honest, I attributed the AI adoption phase that we're in to be more like the cloud and cloud security phase of, you know, 10 years ago. And she really changed my perspective about that with her point around how this is more like 1996 and the early beginnings of the internet.

Dustin Childs: Yeah, I thought that was really interesting, too, and it really -- I don't want to say made me feel old, but it definitely brought back some memories of going to bookstores for AOL disks for another 100 hours, and telling people that, I'd have to call them back after 9 PM because that's when my minutes were free. But yeah, I mean, that -- looking at the tokenization and following that is very interesting and very comparable to the beginning of that. I also like the way she really talked about the business leader partnership angles. AI security requires participation from everyone in every line of business, and not just the security team. And I thought that was a really important takeaway as well.

Johnny Hand: Yeah, and then she had that final security leader takeaway, which is, this isn't about bolting on new tools in your company. It's an operating model transformation to let you move fast.

Dustin Childs: Yeah. And I completely agree with her on those points.

Johnny Hand: All right. So thank you, Dr. Grace, for joining us and sharing your insights. If you want to connect with her on her research or learn more about Dr. Grace, you can see our show notes for more information on connecting. And that does it for another episode of AI Security Brief. We want to thank you for joining us. Our goal is to host conversations that get you thinking differently about security. And if it does, consider subscribing so you don't miss what's next.

Dustin Childs: AI Security Brief is mixed and produced by Elliott Peltzman, with original music by Amneajynx. Our executive producer is Jennifer Eiben, with content strategy by Ma'ayan Plaut and Melany Gallant. Additional production help by Liz Stokes, Kasia Ferguson, and the Black Rabbit team. Video editing by Sarelle Joppy and Brigitte Criqui-Wild.

Johnny Hand: Thanks so much for listening. We'll see you next time on AI Security Brief. [ Music ]