AI's impact on business
Kim Jones: Welcome back to "CISO Perspectives". I'm Kim Jones, and I'm thrilled that you're here for this season's journey. Throughout this season, we will be exploring some of the most pressing problems facing our industry today and discussing with experts how we can better address them. Today, we're looking at AI's impact on business. As AI has only continued to advance and proliferate across every sector, managing its impact is more important than ever. Let's get into it. [ Music ] For those filling out their buzzword bingo cards, it's time to talk about artificial intelligence. I first heard the term AI outside of an academic setting in the fall of 2018. The incoming CEO of a large company stood up in front of the entire organization and made the pronouncement that AI was going to be the next revolutionary advancement in technology. Further, he prognosticated that this advancement would be upon us within the next five years. I remember that many of us in the audience weren't certain where the CEO meant to take the company, much less, at the time, the Information Security Team. Shortly after this pronouncement, the Cyber Leadership Team met to discuss strategic planning and how the CEO's vision would impact our planning and initiatives. While I was one of the two old guys in terms of experience, I'd been in the company for less than two months. As such, I intended to listen and absorb the insights of my new team and new boss. After two hours of sitting on my hands, the CISO called me out. 'I know you're new here, Kim, but your opinion counts as well. What are your thoughts?' I sat for a moment, took a deep breath, and responded. 'I think we're having the wrong conversation.' Up until then, most of the Leadership Team seemed focused on tweaking their operational plans and adding the term "AI" to existing initiatives versus looking at the broader questions presented by an AI-driven future. When one of my peers asked sarcastically what I meant, I grabbed the marker and wrote out a list on the whiteboard. Does the company intend to build its own AI engine or just integrate into existing third-party products? AI is data-driven. How do we normalize our data and break down silos securely? What does breaking down these data silos mean in terms of our compliance posture, as several of our environments have unique compliance requirements, to include our access management posture and controls? Are there new threat vectors associated with AI, other than accelerating existing attacks against our environment? If acceleration is the main threat adjustment we can expect, are our products, processes, and tools capable of handling this volume shift? As AI becomes integrated into tool sets that we use, how do we evaluate the security of these tools? How do we as a Security Team capitalize on the many benefits that AI has to offer as we continue to whip up on the bad guys? After a brief back and forth, the consensus was that I just didn't understand how things were done in my new company. So, my top of head bullet list was dismissed as irrelevant. [ Music ] Cut to four years later and the release of ChatGPT in November 2022. My peers, three-fourths of them the same individuals who were at the 2018 meeting, and the new CISO, now found themselves scrambling to address the above list of questions, and so many more, as the organization surged to capitalize on AI's advantages as quickly as possible. Companies are continuing to come to grips with AI and the advances the technology can create. Unfortunately, in the race to capitalize on these advances, many companies are taking a ready-fire-aim approach to AI adoption. While this approach is nothing new to the security practitioner, the desire for speedy adoption, combined with the psychological predisposition that technology can be implicitly trusted, has led to many organizations believing false information provided by AI platforms, also known as hallucinations. Worse, individuals have a tendency to enter information into AI engines without realizing that the AI platform is a third-party platform outside the scope of control of the organization. The result? Sensitive corporate information and/or regulated information have been uploaded into AI engines. CISOs need to ask the hard strategic questions surrounding AI if we hope to stay ahead of potential pitfalls and challenges this advancement in technology might inadvertently cause. My two cents. [ Music ] I don't consider it an exaggeration to say that Eric Nagel is one of the finest minds today in the area of operationalizing generative AI. His electrical engineering background gives him a predisposition toward meticulously understanding the technology. His many years of experience as a CISO allowed him to understand both the advantages and risks associated with any new technological innovation. And finally, his knowledge as a patent attorney gives him a unique understanding of the potential legal pitfalls associated with fast innovation around a largely untested technology. Eric sat down with me to speak about some of the things he's been doing lately around AI. A quick note that the opinions expressed by Eric in this segment are personal and should not be interpreted as representing the opinions of any organization that Eric has worked for, past or present. Hey, I really appreciate you making the time here. I think it should be a good conversation, and I think you will bring a perspective to the topic that a lot of my listeners need, but don't necessarily have. So again, I genuinely appreciate you taking the hour for me.
Eric Nagel: Happy to do it.
Kim Jones: You and I have known each other for a while, but my audience might not. So, tell us about Eric Nagel.
Eric Nagel: I am a recovering former CISO, I think is the best way to say it, working in a security department for a large tech company. And my background is semi-unique in the sense of I'm an electrical engineer by training, but I've been in the security industry almost my entire career. And so I come with the double E as well as a attorney background. So in addition to being an attorney, I'm a patent attorney. So, I have an interesting mix of backgrounds that I have found to be very helpful in this industry. And so I've been in my current employer for almost 12 years, helped them implement a responsible AI program as well as figuring out how to best secure it.
Kim Jones: Fantastic. So, responsible AI, convince me that's not an oxymoron.
Eric Nagel: It's a goal. It's a goal. I would say, you know, doing it responsibly. AI has existed for quite some time, right? And so we've had our responsible AI program within companies, including our company, for quite some time, but it was classic AI or ML, right, machine learning. But now with the advent of generative AI and the black magic that that is in terms of how it operates, companies have had to really consider how to safely launch it into their environments in ways that basically are, their customers would consider both useful, but also protecting of their information, both company information as well as the information we process on behalf of our customers.
Kim Jones: So I'm going to take us back half a step, because our audiences varied in varied backgrounds. You know, I grew up geek as well. So, you telling me we've had AI for a while and AI versus ML for a while makes perfect sense to me. Poke a little bit so that we understand that distinction in terms of where we were and what we were doing versus what we need to do now with generative AI, if you would, please.
Eric Nagel: Yeah, I mean, the thing that has existed for a long time is really the idea of applying AI principles for machine learning environments, right? So if you wanted to detect fraud in large amounts in granular ways. Basically, you could train a model that would basically be very good at recognizing patterns within your data in order to identify them and hopefully stop them in their tracks. The difference with classic AI is that basically you get a deterministic result. So, if you put in the same data, you get a determined, or the same output. If you did it tomorrow, you did it five minutes later, it is absolutely the same. With generative AI, there's a randomness component because of how it operates. And so it isn't deterministic, in the sense of you will get slightly different answers every time you ask the same question, even if in quick succession. And so unfortunately, as a software development company, you know, we're used to the deterministic side of it. And so the ability to understand how to operate in this environment safely is something that required taking a risk-based approach back into its coming into the business.
Kim Jones: So, getting slightly different answers with the same sets of data, even if asked in rapid succession. And again, I'm going back very basically, Eric, so that our listeners have the full understanding. It would seem at face value this isn't a good thing. Why is it?
Eric Nagel: Why is it a good thing?
Kim Jones: Or is it really not a good thing? Please.
Eric Nagel: Yeah, no, it is a good thing. It's really amazing. It started with the transformer models that came out of Google. But really, it's unexpected results. So again, the generative AI is based on the concept of a large language model, right? And so a large language model is trained in a certain way in order to be able to provide, you know, results that are, you know, at initial first glance, surprising. So, these are the basis of chatbots and other things that basically your listeners are actually experimenting with, you know, with ChatGPT and, you know, Gemini from Google, Anthropic Claude. These are all large language models that exist in the environment, and they've been, spent millions and millions of dollars and lots of GPU cycles to train. And it's a very interesting training process, but the reason why they're useful is that basically you can ask it in natural language questions and it will give you a very well-formed response. One way to think about how they operate is that they are regurgitation engine, right? So if you train it on a whole bunch of books, if you train it on a whole, excuse me, a whole bunch of documents, PDFs, you know, images, these models are trillions of parameters. It's hard to believe how many parameters are in place inside these large language models. But what it really is doing at some basic level is it's predicting the next word in a sentence. And it keeps, you know, based on the direction it's going, basically it can give you a very coherent English in most cases, but now doing pretty well in other languages, a response that basically makes sense to humans. So it can interact with it like it actually is a person. You can ask it a question and you can get a coherent response. And that's nothing like the original classic AI. The classic AI would say, Does it fit this parameter? Does it fit this profile? As it's been trained, the answer is yes or no. And you can use it. It's very useful. So, we still use ML for a lot of useful things. But generative AI is fundamentally a new way of interacting with computers that actually allows us to interact with natural language and get natural language responses that actually make sense.
Kim Jones: Okay. You said it was a regurgitive model. Regurgitation does not necessarily indicate analysis, but I'm assuming that given infinite variables, infinite inputs, infinite processing power, and infinite time, the end result in some form or fashions would be the equivalent of human analysis. You know, what we're really doing when we do human analysis is just taking the facts in front of us and putting them together to determine, you know, the pieces that might be missing. Would you agree with that statement or am I oversimplifying?
Eric Nagel: I don't think you're oversimplifying. That is actually kind of how it works in the sense of the reason it's useful is because of its training. So, the reality is is that of all the different ways a LLM could respond, you know, and it takes, you know, an amazing amount of processing power and a whole lot of skill to train a language model, a large language model, in a way that makes it useful. And so that's why all these companies are getting these amazing valuations, and they're, you know, getting amazing offers for, you know, people that are great in this field, you know, millions and millions of dollars to go work for one of the big three or the big six in this industry. But what I would tell you is is that that's, the training model is basically, you know, how the language model learns, right? But is it reasoning? You know, I would say it's getting better at reasoning and getting closer as an approximation to reasoning, but it's really performing vector math. And at the end of the day, it's not reasoning in the way that humans consider reasoning, but it approximates it with ever better clarity, if that makes any sense.
Kim Jones: And that makes perfect sense to me. I guess my concern is, and we've seen this happen, we saw this happen with outsourcing, offshoring, wireless, cloud, etc. The expression I use is ready, fire, aim. You know, we want to pull the trigger and say, Okay, we're going to use it for this, and then figure out whether or not it makes sense or not. And I don't mind that from an innovative standpoint, but it does exacerbate risk in many business environments. You know, talk to me regarding, and I mentioned them earlier, you know, we've had cases where AI has exhibited bias against certain categories of individuals as it's providing response, some favorable, some unfavorable. We've had cases of hallucinating within the environment. Talk to me about those things, how they happen, and what can be done to prevent that, if anything.
Eric Nagel: Yeah, no it's a great question. So, absolutely that's part of responsible AI is figuring out how to make sure this AI, when you expose it to actual customers it doesn't offend them. It doesn't do things that are basically against the law, or against what you would want it to do from a reputational standpoint for your company. And so what we have found that we had to do was actually build a system. Think of it as a firewall. You know, given your listeners and how their backgrounds, I would think most of them understand the concept of a network firewall. But frankly, in an AI sense, particularly generative AI, you know, we actually built one of the first ones in the industry, an actual AI firewall. So to get from one side to the other, if you think about it this way. So, I put a prompt in. Essentially it has to go through a series of ML modules, which is where we started. We had 13 individual models. One was an anti-bias module. So, anything that would come up with or a prompt from a user that is likely to result in a biased response basically would get flagged. And then we would either rewrite it on behalf of the customer, or we would basically block it and say we can't answer that question because it's obvious that you're seeking a biased response. But we also gate it on the way back. So think of it as a two-way firewall. Basically, the completion as it's called, coming back basically also cannot be biased. So we check for that and then we basically continually retrain these modules. Initially, it was a bunch of ML modules that we trained to be very good at detecting risks like prompt injection, fairness, accuracy. These are all things that we had modules for, but then you have to worry about weird things like emojis. Hard to believe, but emojis basically can get the LLM to respond in very unpredictable ways. And so we basically, same thing with code, code detection. You can actually make the model hallucinate if you can actually, you know, pass Python code or other code as part of your prompt. So, we use code detection. But essentially, to your point, right, companies that are trying to put this into their environment need to have the ability to make sure that it is safe. And out of the box, they are not, you know? So, I would say that the initial training that all these LLMs go through is intended to be safe. But we have found in many cases that, you know, we have to supplement what the actual LLM manufacturer or trainer has done with actual code on our environment that basically, you know, makes it a whole lot safer.
Kim Jones: Okay, so understanding what you're saying, and that makes perfect sense to me, but also understanding that I'm talking to someone who is on staff at a Fortune 400 company with several hundred people on security staff, and a fairly robust and sizable security budget. What does someone not of your size and scope do to put in reasonable levels of control as they're looking at bringing in AI within their environment?
Eric Nagel: You know, it's a good and bad scenario. I would say there's a whole bunch of startups as well as, now acquired startups, that are now in larger companies that offer this as a service. And so when we looked around two years ago, two-plus years ago now, we found that basically what we needed to be safe in this space didn't exist. So, we ended up having to build it ourselves. But I would say smaller companies, companies that don't have the same kind of sizeable security budget and staff, have the ability to consume these services on a per size or per application basis that, you know, is not completely out of reach for companies with more modest resources. And so, you know, there's a new startup every day in this space literally, or many more than that on a daily basis. A lot of them are coming out of Israel, but also in other parts of the world. Silicon Valley has its own share. And so I would encourage companies and small proprietorships that basically want to consume it to consider the risks, you know, of making it available and to put reasonable, you know, capabilities, you know, in front of that, that allows them to operate with a safe manner.
Kim Jones: We talked about a couple of risks, Eric, through this conversation. But for a company or a small shop or whomever that is approaching the "We want to be, we want to deploy AI in some sort of logical fashion," etc., what are the top three or four things from a risk standpoint that they need to be aware of?
e: Well, the best thing that we have found for anyone to do, large or small, is to basically, you know, use it for what it's good at, and then not use it for what it's not good at. That would be my top one. The second one is to basically constrain it. In other words, unbounded chatbots are not considered very useful. They're much more likely to come back with off-topic responses. You know, it's similar to what you see when you query Google today, right? If you query Google today, it comes up with a Gemini AI response, as well as the things that you're used to seeing, which is the links to potential page rank, you know, answers to your query. What Google is doing with its Gemini LLM is basically, you know, saying, Hey, you know, you prompted it the way you've always prompted it, which is search for this, right? You put in a topic and it gives you a bunch of things to go choose and click on. But the AI thing is actually giving you a sample of what it can do for you. In other words, it's basically saying, You know, let me summarize the best things from the links below in a way that's very consumable. And so a lot of people thought that, you know, AI or OpenAI in particular was going to kill Google's search business. I read an article that says their search business is better than ever, but partially because they have figured out how to profit from it in both ways, which is people still want the search, but they also love the AI summary at the top, right? You can discount it, but it's getting better and better. And I find it's been amazingly useful. So, I think there's, that's two of the top things. The third one is basically constrain the prompt in the sense of the ability to say, you know, an unbounded prompt is not useful. Let me just capture the things that I actually want the user to interact with, and then only have those components, right? And so if you're doing the air conditioning business, you know, you give a little bit of information on your outgoing phone call that says, you know, thank you for calling, you know, and then basically ask it, ask the person to basically identify three things that would be useful for us to know that will actually optimize for our scheduling software and those kinds of things. And so, you know, maybe that was more elaboration of number two. But the third one is basically look for tools to enhance, you know, on a as-needed basis to protect against hallucination and some of the other problems that come with the use of the model. [ Music ]
Kim Jones: So, let's put back on your recovering CISO hat and think about, you know, any new technology injected into an environment presents some level of cyber risk. Some of these we probably already know, but from a cyber standpoint, what are the concerns that we have regarding, let's start with unbounded AI operating within an enterprise environment?
Eric Nagel: Well, I mean, the biggest concern that we had, and I think anyone will have, is data loss, right? Data, basically, you can leak model data into the model. And so one of the first things we did was by contract have our own, you know, standalone, you know, instance of these LLM models. So, we paid extra and executed contracts to be able to use one that was dedicated to us. It was not multi-tenant. If you read the actual legal language around these models that are available on the internet, you know, they basically say, We reserve the right to take all your data and use it to train our models. And they do that in a constrained way because it can be bad, and so they pick and choose. But for a company, or even an individual that pays for their own subscription to, you know, ChatGPT individually, you can get assurances that your data will not leak in that way. But to your point, you know, there's ways that basically just interacting with the model can allow data leakage. So there's a thing called prompt injection, the ability to put things into the prompt that will cause the actual LLM to go off the rails. You can get it to, you know, provide a biased response. You can get it to provide, you know, a profane response. You can get it to, you know, start to do things that explain some of the private information that's in the model, like the model weights. These are all things that basically the model creator is trying to prevent. But in many cases, you have to take steps to make sure that those things don't happen. And so against the AI firewall that we put in place, it was really to guard against the main risk, which is leaking your data in ways that you should not.
Kim Jones: Two questions I wanted to ask you, but I'm going to start with going about five degrees off. I've heard, we've all heard regarding some of the things that AI can and can't do. Let's talk code. I've heard a statement actually made by someone in your company as well that AI prompted appropriately can potentially code about 80% as good as an entry-level software engineer. Agree or disagree on a personal level? Or no comment, if you need no comment.
Eric Nagel: No, it isn't a no comment. No, I think it's actually quite good for doing prototypes. I think you can get all the way done with an actual prototype, but can it actually produce code that's in compliance with your coding standards? Can it not have the errors that basically an entry-level engineer would make or often does make? I think it's a little bit overblown to basically say it's going to replace all of our entry-level engineers. Because how are, you know, one way of thinking about that is that if you don't have entry-level, how are you going to get the next level up?
Kim Jones: Fair.
Eric Nagel: So the reality is that the death knell for the software coding industry I think is a bit premature. Because --
Kim Jones: But is it going to change the requirements for those entry-level engineers? And here's your hypothetical.
Eric Nagel: Yeah.
Kim Jones: Right now, I pay that entry-level engineer to code. Do I now need to have an entry-level engineer who understands the critical thinking necessary to review the code that is just spit out via an AI engine to determine whether it meets standards and put that last 20%, which is a slightly different skill set than entry-level engineers are being recruited on today? What are your thoughts?
Eric Nagel: No, it's actually a very big concern for a company that does software development as our main output. And so what I would tell you is, is that, one, I don't think entry-level engineers are obsolete. I think the ones that know how to use the tools well will exceed where those that don't know how to use the tools well will not succeed. But the big thing was we wanted to make sure the engineers were accountable for whatever code that they deliver. And so to your point, lazy engineers, you know, produce lazy code. And so, you know, the whole idea of coding using this method, we had to train people to basically say, This is your code. You are responsible for whatever you check in. If you borrowed it from some other place, you better know how it's operating and be able to explain it, right? And then we have the four eyes rule, or the two people rule. Anything that gets merged into the environment basically has to be peer reviewed. But these days it's gone even further, Kim, right? CodeWhisperer, you know, some of the other ones that basically have, you know, Windsurf and Cursor. These are what they call vibe coding tools, which allow people that don't even have a coding background to basically use natural language to say, I need a routine that does the following things.
Kim Jones: Yeah.
Eric Nagel: And it can get you very close to that. And those companies are having amazing valuations because of the utility of that. But again, I think the way it's going to work is that you can get close with that, but it's something that basically will have to be vetted, and the one thing that they don't talk about is vibe debugging. So, vibe coding works, you know, and it gets you to the 80%, but you have to understand the 80%, and then you have to finish it with the other 20% for it to be useful.
Kim Jones: Yeah, yeah, that makes sense to me. So, let me shift and ask you to now put on, you know, your lawyer hat for a moment. What are you seeing in the regulatory landscape and the legal framework right now, or the legal fabric? And let's stay, let's start US-based for the legal fabric. If you're seeing anything significant overseas, we'd love to hear it as well, because AI has created, you know, new and interesting challenges that will, that have started in terms of some legal challenges within the environment, and I expect it to continue. What are you seeing in the environment now, and what do you predict will come in the future?
Eric Nagel: Well, in lieu of the government, the US government basically having a unified voice on this, which given the current state of our union and Congress in general I think is unlikely. You have individual states that are leading in this space, similar to CCPA, CPRA. You know, California was out in front on the privacy side. Colorado has a, I would call it, flawed law that is coming into space. And they had a chance to fix it. They didn't, but they know it needs to be fixed.
Kim Jones: Talk to me if you're willing to deep dive, and if you can't, I understand. Why is it flawed?
Eric Nagel: Just because it's ambiguous. And so a company like ours that is trying to evaluate when we operate in all 50 states and certainly around the world, we try and take a lowest common denominator approach to regulations. And so we pick the most restrictive interpretation when we can, and code to that, if you will. So therefore, we would meet all the lesser ones. But New York, Massachusetts, Colorado have all kind of split off and done different things in that space. The one that's closest to being in force, which will happen I think early next year, is Colorado. And so it affects some of our money lending, you know, in the commercial space and some of those kinds of things. And we think we have a way of dealing with it, the ambiguity in the law. But we're also trying to predict where it's going to go. But what I would tell you is that, you know, from a regulation standpoint, like it always is in the cybersecurity space as well as privacy, you know, in general, legislation tends to lag the actual industry and where it's going. And so we keep a strong eye out on where things are going in that space, and then we code to it. To your point on Europe, Europe has the AI Act, right? And so they've kind of said, you know, if it's in the following high-risk areas, you're not allowed to do it at all. Or there's a whole bunch of regulation that will come with it, and then here's slightly lower risk and it has these, you know, semi-relaxed requirements. And then the ones that basically aren't really of a concern basically are allowed to go into force without a whole lot of review from the regulators.
Kim Jones: If I were to summarize a lot of what you're talking about within the environment and put on my old CISO hat, a lot of this falls under the concept of governance within the environment. Talk to me about the challenges in terms of, you know, starting up AI governance. Having the, you mentioned it yourself, Eric. You have a, you know, to quote Liam Neeson in "Taken", you have a unique set of skills that most people don't have, or have within their teams within the environment. So, you know, let's talk about the challenges of starting up AI governance. Let's talk about the requirements to start up AI governance, and, you know, some of the things you've seen, some of the things that went well, some of the things that didn't go well within the environment. Talk to me.
Eric Nagel: That's actually a great question. What I get from talking to CISO peers is that they wish that they hadn't, you know, basically desired to create a governance program after the horse was already out of the barn within their business units.
Kim Jones: Yeah.
Eric Nagel: The thing that they most are, you know, and they even use the J word, jealous of what we have done is basically is that we started with a risk-based approach. I was asked to write the risk paper, you know, two and a half years ago for the company. And so we identified all of these risks and then they gave me the charter to go build something to protect against each of those risks. And so we hired a technical team, we brought them in, we've hired, you know, experts out of universities that are PhDs in this space, and we took advantage of all that expertise to create an environment plus a single path. And so the biggest thing that they wish they had was a single path that all their business units were forced to use that basically provides transparency, visibility, observability in the environment, as well as the security protection. So if you use, we call it a paved road. If you use what we have announced as GenOS, we also have a product that we call internally GenSRF, or Security, Risk, and Fraud. And that's where all of those individual ML models have started. But, you know, what I would tell you is, is that they most wish they had established a risk-based approach and then had the will, the business and, you know, internal politics will, to basically say, This is the one way for people to make these experiences available to their consumers.
Kim Jones: All right. I think we're going to have to leave it there. Eric, I really appreciate you taking the time to lay this out for me and for our listeners as well. And again, thanks as always, man. And I'm looking forward to catching up with you real soon.
Eric Nagel: Thank you for this opportunity, Kim, and I look forward to seeing you soon. [ Music ]
Kim Jones: And that's a wrap for today's episode. Thanks so much for tuning in and for your support as N2K Pro subscribers. Your continued support enables us to keep making shows like this one, and we couldn't do it without you. If you enjoyed today's conversation and are interested in learning more, please visit the "CISO Perspectives" page to read our accompanying blog post, which provides you with additional resources and analysis on today's topic. There's a link in the show notes. This episode was edited by Ethan Cook, with content strategy provided by Mayan Plaut, produced by Liz Stokes, executive produced by Jennifer Eiben, and mixing, sound design, and original music by Elliott Peltzman. I'm Kim Jones. See you next episode. [ Music ]
