The great overcorrection: shifting left probably left you vulnerable. Here’s how you can make it right.
Rick Howard: Hey, everyone. Welcome to "CyberWire-X," a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the chief security officer, chief analyst and senior fellow at the CyberWire. In today's episode, we are talking about shifting left. A program note - each "CyberWire-X" special features two segments. In the first part, we'll hear from an industry expert on the topic at hand, and in the second part, we'll hear from our show's sponsor for their point of view. And since I brought it up, here's a word from today's sponsor, Invicti.
Rick Howard: I'm joined by Rick Doten, the VP and CISO for Healthcare Enterprises and Centene. Rick, thanks for coming on the show.
Rick Doten: Thanks, Rick. Happy to be here.
Rick Howard: Today we're talking about application security testing. And what I mean by that is extending our zero-trust strategy beyond the traditional network defense for our data islands, like data centers and cloud deployments and mobile devices, and include reducing the attack surface for the software we use - you know, the code we write ourselves, the code that we get from open-source libraries and the code we run from vendors like Microsoft, Google and Amazon. Now, you and I have been doing this stuff for many years, and application security testing has always been around. But until recently, say the last five years, I wouldn't say that many CISOs have had this on the top of their priority queue. But you talk to a lot of security executives in your day-to-day job. Is that changing these days? Do CISOs own the responsibility, or is it a shared responsibility with the CIO's DevOps teams or some combination of all of that?
Rick Doten: Yeah, it really depends on the organization. If your organization is a software development - I mean, you create software products - then the development team usually has more power, and they manage their pipeline. If you are a large organization like ours who does a lot of development, then the IT security organization - that's a very large organization - kind of oversees the security aspects of it, many of which may be applied and used by the application development teams. But then, because most application development is cloud nowadays, the cloud team is also involved as well. So whereas before - like, 20 years ago when we were doing application pin testing, you know, we deal with the developers. Security would oversee the testing and the QA process but usually didn't have anything part of the process until the pin testing done at the end. Then we started evolving into training the developers and in the early 2000s started integrating security in the application lifecycle. And then today, as you kind of reference, like, we have a CI/CD pipeline. Incorporating into DevOps that the development teams have been using and integrating security in a DevOps lifecycle to be DevSecOps is relatively new thing.
Rick Howard: So this is kind of a matrix organization here, right? It doesn't feel like anybody's really in charge of it. I guess the CIO would typically be in charge of the CI/CD pipeline, but there's all kinds of groups in there - right? - influencing how that is built and maintained and things. Is that how it's done today?
Rick Doten: Yeah. And it really depends on the organization, like I said, and who has the most influence. And so some organizations, like, the developers may be at the bottom of the chain, and there's application architects who dictate to them what tools and approaches and architecture they're going to be using. There may be cloud security, the cloud team that dictates the cloud infrastructure they're going to be using. And then there will be security teams that dictate the security requirements for that. And so when we start adding security into the application lifecycle, you know, we'll - it's a whole stack. We have threat modeling. We have static code analysis. We have dynamic analysis. We have testing of the infrastructure and the infrastructure's code. You had mentioned before third-party libraries. We have dev and production environments. And so a lot of times you also have secrets management. You have data protection. You have APIs. There's lots of pieces that are involved. So yes, it definitely becomes a team sport.
Rick Howard: I was talking to my best friend, Steve Winterfeld, about this. He's the advisory CISO for Akamai. He says that who owns the application security function is largely based on the existing business process.
Rick Doten: So it's interesting when you talk about testing CISO versus CIO. Really ultimately, you know, you have two testing that I think about - quality and security. And I think those testing should be done collaboratively and somewhat based on culture. Ultimately, the CISO has to provide the tools, the resources, be they hooks or tools that they use. Often the CISO is going to do some of the monitoring and alerting. You have to build a collaborative process. And whether it's security champions within that group or whether you're putting your staff in with the right skill sets to collaborate in there, I think really that's kind of where it is. Ultimately, it's a business process, not a standard answer.
Rick Howard: Rick, you mentioned at the top of the show that there are two kinds of application testing, static and dynamic. So let's start with static. What is it? How do you do it? And how do you know it's working correctly?
Rick Doten: This has been around for 20 years or so. And we used to call it just code reviews. And, you know, we would have it, you know, where we talked about dynamic analysis, is running a tool of an application that's already in place. And it could be somewhat automated, where you were just crawling through the directory structure. You were testing certain commands to see if there's SQL injection, cross-site scripting, tools running, and it comes back. That's also been around for 20 years. The next step above that is, like, active penetration testing where you get some valid accounts, and you try to manipulate things, and you work on things. And, you know, I spent the early part of my career doing that and managing people who did that. So the static code analysis - we used to kind of call, like, that's the deep dive, where, like, I'm looking at what that code actually does, and there are tools that you just kind of dump a code sample in and it goes through and looks for a lot of those standard OWASP Top 10 type of things for releasing buffers, releasing cookies, cross-site scripting, cross-site request forgery - looking for those kinds of things as a human or even, like, a crawler doing an automated dynamic testing, we might miss something because there might be a part of the application we didn't touch because it was behind the scenes or something that we didn't actually interact with. And then a static code analysis may actually see that because we didn't interact with it. Nowadays, it has become more common.
Rick Howard: The static stuff, in my mind - you tell me if I've got this wrong - is things that we already know about - like, you mentioned the OWASP Top 10. In any kind of regression testing, mistakes we've made in the past - we don't want to remake them again in the new code. So we - that collection of things - and like you said, we can run tools to see if our existing code base has those mistakes. And so that would be static. It's not a live attack of the code. Am I saying that correctly?
Rick Doten: Yes, absolutely. That's a good way of putting it. It's like...
Rick Howard: Yeah.
Rick Doten: ...Looking at the code directly for things that we already know are vulnerabilities.
Rick Howard: Right. And so then you mentioned dynamic testing. That's - well, the way you described it is, it's like hackers going after a running code. Is that correct?
Rick Doten: Yeah. And it's - and nowadays it's, like, some of the same tools that we would use for network testing, penetration testing or vulnerability scanning. But, you know, they've kind of evolved to the application layer. Yeah, so it is more human-driven, and it's interactive. There are dozens of stories from years ago. But for instance, one of my favorite ones was you go in as a user who doesn't have, like, admin access, or access to a certain database to get certain data, but there might be a widget that is used to be able to query something in another database. Because I'm in there, I'm imitating that widget to go access the database, and I can pull more information as that service account then what I have - that has greater access than I do as a user. Static code analysis is usually done against specific modules, and so the combination of those things may not be, you know, seen as static, but it would be in a dynamic. The extension now to the static is - infrastructure is code. And that's kind of the new thing in the last few years as we've done cloud application. Then, you know, we're spinning out temporary workloads, and the code will say, OK, I'm going to spin up this container. It's going to, you know, install this application. It's going to run for five minutes. And then it's going to shut down, and I'm going to move on. And so now we have a whole new layer of looking at that - not just the things we know about, but the infrastructure of code, which is not unlike doing an on-site network scan of the operating system of the device that - the server that it's running on.
Rick Howard: So it's like you said, the static testing - it's just a list of things we're going to check. But dynamic testing is really - like you said earlier, it's a red-teaming exercise directed specifically at the software that we run - not necessarily against the entire network, but it's very...
Rick Doten: Right.
Rick Howard: ...Focused on that. And so that makes it different. That in - but there are tools that will help people that are worried about this do this themselves as opposed to having, you know, world-class OP 4 going against your networks, right?
Rick Doten: Oh, yeah, absolutely. And the tools have gotten a lot better, you know, over the years. And some of them are still - have been around for 15 years. But yeah, so they will do that in an automated way. And then there's also tools that help you - assist you if you want to do it in a more hacker way and get in the middle of the proxy and adjust things in the fly and stuff like that.
Rick Howard: Well, I think the interesting thing that's changed in the last five years or so is our community becoming aware of third-party code being used in our environments. All the supply chain attacks that we've seen in the last couple of years has really made that issue high on our radar. So that's a whole different thing - right? - 'cause I don't think - we weren't even talking about open-source modules over five years ago, but now it's a thing that we have to worry about.
Rick Doten: Well, I mean, it - I guess it's at a broader scale. I mean...
Rick Howard: Yeah.
Rick Doten: ...Even 20 years ago, we were looking at - because most applications are, you know, 90% other people's code.
Rick Howard: Yeah.
Rick Doten: And there's lots of open-source libraries that are used. There's standard modules that are being used for encryption. There is modules used - there are libraries used for managing identities and maintaining session state. And so, back in the day, when we were really doing application pen testing, we would look at all of those as well. And those would be things that we'd probably ask to do a code scan of - of these third parties. Now, because it's a lot more prevalent - and there are tools that are kind of like - were built out of doing this third-party code library analysis in the repository. So in GitLab or GitHub, you know, there's like - OK, when I'm maintaining these things, I'm kind of making sure that the library that you're going to call, that's a third-party library, is the one - A, the version of the one that's already been tested and that's cool. So, yeah, and then as we're now moving more towards, you know, the software building materials, then that becomes a play. So yes, it's happily been getting the attention that it's never really had, unless you are a very mature application security team.
Rick Howard: And Steve says, let's not forget about the right side, as we are all shifting left.
Steve Winterfeld: So really I think of this as kind of - shield to the right, shift to the left. And on as we shield to the right, we're protecting APIs - JSON, XMLs - obviously you're talking about the right tools here. Static, dynamic, interactive - I think they're all useful. I think we need to expand beyond that, some of the traditional things. Pen testing probably, will have a part in this framework. I think you need to expand this into your JavaScript environment, and you need mobile testing as well. So I think these are the right questions to ask, but you need a holistic set of tools here. But ultimately, I think if you're not doing these, you are just pushing vulnerabilities forward.
Rick Howard: So, Rick, you got - you've been doing this static and dynamic analysis for a while now in your current job. Is there any tips that you can pass on to our listeners about any pitfalls they need to worry about to steer away from?
Rick Doten: I guess it's, like, it's - make sure that everyone knows their role, you know, because it's a team environment. And make sure that the developers are educated and have the tools if you're coming from a CISO perspective. And also come up with a process, you know, because the real challenge is that crossover with infrastructure of code if, like - if a flaw is found in a piece of infrastructure, did it get there because of the code or was that there because of some immutable image that was used to pull from? You know, whose fault is it, for lack of a better word? - because that will change the workflow to fix it. And so understanding these different scenarios where, OK, this was put in through infrastructure of code; this is a bug fix for the developers need to fix. Or this is something that's infrastructure that was already put out by the cloud team; they will use another process to fix that. And then also go back to root cause. So the number one thing I would say is, like, find the root cause of how it got there and fix the root cause instead of just kind of keep patching these containers. And then every time they get refreshed, the new vulnerability gets put in there again.
Rick Doten: And then figure out, like, who - and this - I say this a lot when I talk to cloud and application security is - who has influence? And, you know, power is another word. And so - like, and sometimes we may need to start giving back to the developer some influence because we don't want to just, you know, wait until the thing is running live. Kind of like we did pin testing 20 years ago after it was about to go live and then we find a problem. And we go back and have to fix it and push out the release dates. But, you know, instead of make - you know, finding it during a vulnerability scan or a dynamic scan of an environment of the infrastructure and then sending it back to developer, have it to where you're checking it earlier on and so that you're checking the infrastructure's code in the code to see if it's pulling the right image, if it's configuring it the right way and securely instead of having to wait. And so that will kind of speed up that cycle.
Rick Howard: Well, all those problems you say to concern yourselves with, I think - I just - from my perspective, I would like to remind the audience, just think about agile development, and think about DevOps and the problems that those methods try to solve. They try to break down those walls between developers and operators and security people, you know, instead of throwing code over the fence and saying, OK, I'm done with it until it comes back to me, it's more of a system of systems. And if you can run your organization that way, where it's one big system of systems that everybody is paying attention to as opposed to stovepiping (ph) different work efforts, I think that would move your organization forward more quickly and more efficiently.
Rick Doten: I think that's great advice. And I like to say that, you know, DevOps is really like Agile Scrum or XP but automated.
Rick Howard: (Laughter) Yeah.
Rick Doten: You know, so it's the automation of it. And that's what we're trying to get to is more automation, more automation in remediation, more automation in release and - because, you know, that's what DevOps is and that's what DevSecOps is. And so if we don't have a good process in practice, automating is just going to mess it up. You don't...
Rick Howard: Yeah, good point.
Rick Doten: ...Automate bad things; you automate good things. So make sure you have a good process and then go to automation.
Rick Howard: I've been doing it wrong all this time.
(LAUGHTER)
Rick Howard: Well, this is all good stuff, Rick, but we're going to have to leave it there. That's Rick Doten, the VP and CISO for Healthcare Enterprises at Centene. Thanks for coming on the show, Rick. It was great talking to you.
Rick Doten: Thanks, Rick. Always a pleasure.
Rick Howard: Next up is my colleague Dave Bittner's interview with Sonali Shah, the chief product officer at Invicti, our show's sponsor.
Dave Bittner: What's the background on when people in security started talking about shifting left at all? How did this notion take hold?
Sonali Shah: The shift left movement became popular at the same time DevOps became popular and now DevSecOps. And the idea of shifting security left is that if you were able to find things earlier in the software development lifecycle, it is easier, faster, cheaper to fix. And as we release codes - if you think about DevOps, the idea there is efficiency and being able to increase the frequency of code releases and doing it kind of in batches and continuously improving your product. And if you wait and do security at the very end, well, then you're not going to be able to do those rapid iterations because you're going to have to take time out, scan the application, find, and fix and then release. So the idea is that if you do security early on, you'll be able to catch it, and it doesn't slow down the release.
Dave Bittner: And so when people generally come at this - when they decide, OK, we're going to adopt a policy that involves shifting left - typically, what would that look like?
Sonali Shah: So one thing I just want to start out by saying is that shifting left doesn't mean that you don't shift right. Right? So the idea of DevSecOps is that security is built into the entire lifecycle of the application. So from - you know, you're testing in development before you release code, but then you're also continuously testing in production because we all know two things. One is applications that are secure today may not be secure tomorrow, right? You find a new zero vulnerability is discovered. Suddenly, now something that's in production that was safe when you released it is no longer safe. The second thing is that it is really impossible for a company to say, all right, I'm going to find and fix everything before I release. You know, we - there was a survey recently that said that majority of developers are releasing code that they know is insecure. And it's not because they want to. It's just that they don't have the time or the skills to fix it. So when we talk about shifting left, that doesn't mean abandoning the right. It means it - really, it's doing both. When you think about DevSec and Ops, it's about doing security early and often but also doing it continuously while applications are live.
Dave Bittner: Well, and certainly that all sounds like a good thing. But what are some of the places where people find themselves being tripped up by this?
Sonali Shah: Well, the way to shift left effectively is to automate security into the development lifecycle. So where you get tripped up is if security's a separate stage. And you do security, and then it goes back separately to the developer. The way to do this effectively is to automate security. And what I mean by that is automate the testing so that, you know, within this CICD pipeline, just as you're, you know, building the code, you're also testing, and results should automatically go back to the developers in whatever tool it is they use - right? - whether it's, you know, Jira or ServiceNow or whatever it may be. And where people get tripped up is, if they don't do the automation, then it just becomes too cumbersome for developers to have to then go to a separate tool and run scans and, you know, get the results. So I would say that is one area.
Sonali Shah: Another area is make sure that the tools you're using are accurate and actionable because, even if you're - the tool you're using is fully integrated and automated, if the results they're giving you are not accurate, it's really garbage in, garbage out, and you're only going to frustrate the developers and end up wasting their time looking at a lot of false positives. So I think it's really important that you make sure that the tool you're using is producing accurate results that are actionable, as well as having that - your security tool automated and embedded in the whole software development life cycle.
Dave Bittner: Yeah. I mean, that's a really interesting notion, and I guess it makes me wonder - how does an organization dial it in - you know, that that balance between the security that they want, but also not introducing too much friction to the process?
Sonali Shah: Yeah. No, that's a great point. And, look, every customer I talk to has friction. Like, that - no one has solved that problem. But, you know, there are companies that are really struggling, but there's companies that have made a lot of, you know, great progress. You know, I don't know if you were at RSA last week, but I spoke with Ean Meyers (ph), who is head of application security at Marriott Vacations, and he talked a bit about how he's been able to improve kind of, you know, the relations between developers and security teams and kind of reduce the friction. He talked a lot about, you know, having developers help with, you know, kind of deciding - what are the security tools you're going to use, and how are you going to weave them in so you get their buy-in in early on?
Sonali Shah: You know, he also talked about security champions programs. I was actually recently visiting a large global bank in Paris, and I heard from our customer there how they've been really successful with their application security champions program, and how that has really helped. And what was interesting - what he told me is that, when he asked, you know, his development teams who would like to be in this application security champions program, he was really surprised at how many developers were interested because he's like, you know, they really do understand the importance of security. They take pride in their code. And, by the way, you know, having something security related on your resume or your LinkedIn profile is actually, you know, quite a remarkable thing for them - for developers to have. So, you know, they're - the story that I heard from him, as well as from Ean and other customers, is that building a security champions program is just another way to help reduce that tension.
Dave Bittner: How do you measure success with something like this? When, you know, you've implemented it and it seems as though it's - everything is running well, what are some of the metrics by which you know if you're doing the right thing?
Sonali Shah: I think it's like anything else - that when you're going through a transformation, start small. Get the buy-in, prove success and then expand, right? So we actually help our customers roll out their application security programs. And what we always recommend is, you know, if you have a thousand applications, let's start with one or two groups. Let's start with, you know, a couple hundred, and let's do the automation. Let's do the integration. Let's prove success there. Show that you can build security into your toolchain, that you can automate it. And really, the key thing is about risk reduction, right? So some of the metrics you can look at is number of applications scanned, number of vulnerabilities found. But really, the metric here is risk reduction.
Sonali Shah: So the measure of success is how many vulnerabilities and of what severity are being released in production, and that is your risk reduction. And then, of course, there's also the efficiency metrics. At what cost did you do this? And, you know, those are the two things, but I would always suggest our customers that they start with a subset of their organization, prove it out and then expand. But the ultimate metric is always risk reduction, and then at what cost?
Dave Bittner: What are some of the stories that you hear back from folks who have successfully implemented this sort of thing? What's their experience like on the other side of it?
Sonali Shah: Yeah. It's a journey. No one gets there immediately. It takes time, and there's steps forward and backwards. One of our - another large global customer that we deal with - they actually had all their developers come together for a two-day virtual offsite and had security training as part of that, and we were sharing some of the stories. I think what good looks like is when security - it's like air. It's just something you breathe. You don't think about it. You know, when you develop code - right? - you want to develop quality code that does what is expected - right? - that does something that meets a customer's need. And what success looks like is when security is built into that. So you can't possibly deliver a new feature that meets the customer's needs if it's not secure because that has to be part of the criteria. And when security is just built into it, and it becomes part of the developer's job, that's what success looks like.
Rick Howard: We'd like to thank Rick Doten, the VP and CISO for health care enterprises at Centene, Steve Winterfeld, the advisory CISO for Akamai, and Sonali Shah, the chief product officer at Invicti, for providing some clarity to us around the idea of shifting left. "CyberWire-X" is a production of the CyberWire and is proudly produced in Maryland at the startup studios of DataTribe, where they are co-building the next generation of cybersecurity startups and technologies. Our senior producer is Jennifer Eiben. Our executive editor is Peter Kilpe. And on behalf of my colleague, Dave Bittner, this is Rick Howard signing off. Thanks for listening.
