Cloud configuration security: Breaking the endless cycle.
Rick Howard: Hello everyone and welcome to CyberWire-X, a series of specials where we highlight important security topics affecting security professionals worldwide. I'm Rick Howard, the Chief Security Officer, Chief Analyst and Senior Fellow at the CyberWire and today's episode is titled "Cloud Configuration Security; Breaking the Endless Cycle." Moving to the cloud creates a tremendous opportunity to get security right and reduce the risk of a data breach. But most cloud security initiatives get underway after services are deployed in the cloud. It's frustrating when major breaches resulting from basic mistakes, like S3 buckets left unsecure, or secrets exposed. Continually checking for risky configurations and unusual behavior in cloud logs, is a requirement. But there is an opportunity to be proactive.
What if you could configure your security and access controls, as you set up your cloud infrastructure. In this show we're going to discuss how security teams are adopting infrastructure as code, or IAC security, as the cool kids call it. As part of their overall cloud security strategy to reduce risk. A program note; each CyberWire X special features two segments. In the first part of the show we will hear from industry experts on the topic at hand. In the second part we will hear from our show's sponsor for their point of view. And since I brought it up, here is a word from today's sponsor, Sysdig.
Rick Howard: To start things off I've invited Kevin Ford, the CICO for the State of North Dakota, to the CyberWire hash table, to talk about his mature DevSecOps program. For those of you that don't know, North Dakota has one of the larger state networks, managing and securing some 250,000 devices at any given time. As the CICO he has the responsibility for the state network, that includes government entities at the state, founding and city levels, plus all of the schools K through 12. I've talked to a lot of CICO's in my career, Kevin has deployed one of the most mature DevSecOps environments that I have ever seen. I started off by asking him to describe the current state of the program.
Kevin Ford: We have multiple DevSecOps initiatives. One is our named DevSecOps initiative, which is the DevSecOps initiative of the state IQ organization, and so as the security organization within the state IT department, we are that big capitalesque sec within DevSecOps. So we will help deploy and help write standards around CICD pipeline, as well as implement cloud technology and cloud security technologies. We also do have internal to the security team, a mini DevSecOps team that is an agile team that is comprised of members of our infrastructure team, our security automation engineers, as well as GRC and a number of other talent sets mixed in there; all working in an agile way in order to deploy a DevSecOps approach to creating automations within the security team.
Rick Howard: There's two pieces to this DevOps thing. There's the traditional DevOps approach where you're automating the infrastructure. You're inserted into the continuous integration, continuous deployment pipeline; the CICD pipeline. Then there's this other part where you're in charge of the security stats for the state. You're experimenting with using the DevOps philosophy to automate the maintenance of the security stat. So are you using traditional DevOps tools to do that, like Ansible? Or are you trying to do it with traditionally security specific tools, like your Soar tool.
Kevin Ford: So for the larger portion of that, the enterprise wide DevSecOps approach, yes, we're doing it with more traditional tools, but the security side, actually our tools are surface to us as software as a service. So there's not a whole lot of the underlying configuration that Kubernetes the containerization and those sorts of things that we really worry a whole ton about; that's generally done on the vendors side.
Rick Howard: So you use a next generation firewall that you guys are Palo Alto Networks customers, right?
Kevin Ford: Yes.
Rick Howard: So you're using the Soar tool from that vendors, from Palo Alto Networks, my old stomping grounds, to update the nextgen firewall, wherever it's deployed.
Kevin Ford: Yes. We use the Soar tools that Palo Alto provides as kind of... the philosophy we're approaching is that we have the Palo Alto soar tools which are really the brain. So that's gonna be the central brain for a lot of the security automations that will be firing off based on different conditions. The Palo Alto soar tool is also we're hoping, going to be enriched by a security development effort, which again is kind of our internal security DevSecOps process, around connecting it to different API's; around firemouth different scripts based on different conditions. So we're leaning very heavily into that soar tool as a centralized brain, but one of the other things that I think I should observe here is, we are also ingesting all the sassy tools from Palo Alto.
So what that means is a lot of the integrations with the soar tool are actually pretty easy, because a lot of our stuff is the same vendors. That makes it very easy, when we're talking about integrating with systems outside of that, we're really right now only looking at our vulnerability management suite, as well as our cloud suite; our Microsoft Dasher.
Rick Howard: Can you give some advice to any organization that's thinking about this. They haven't really started yet. Do you bring in a consultant first so they can help you think your way through it? Or do you start breaking things and then once you learn a little bit you can bring in some consultants to help you undo it all [LAUGHS].
Kevin Ford: We call that fast fail, not breaking things. [LAUGHS] I think my biggest advice would be to be first look at the organization. DevSecOps is an organizational approach first and foremost, so look at your organization, see how it's structured. Do some research around how successful DevSecOps organizations are structured, and try to come up with a hypothesis around which one would be the best for you. Once you have that hypothesis absolutely do not say, okay, this is the way we're changing the entire organization. You want to take an evolution, not revolution approach here, so that you're not breaking things. What I would suggest is put together a little tiger team and start looking at the applications or the products that you have, and seeing where the contenders for, for instance, decomposing a monolithic system into maybe a micro-services system, R.
Then look at how you can facilitate that from the cloud perspective, meaning are we solid with Kubernetes and containerization or femoral environments and so forth. From a security perspective do we have sufficient application analysis in our CICD pipeline, and is that automated, so that we can make these continuous builds throughout the day without introducing new and exotic security loopholes. Then also, at the application developers, do they have the skills or the right tools in order to create these kind of non-monolithic systems? It requires a whole different perspective than a lot of developers currently have these days. So it's a team sport; everyone in the organization needs to participate, and everyone in the organization needs to understand where you're going.
Start small and don't try to reinvent your entire organization around these principals. Start small and disrupt yourself.
Rick Howard: Are your security people learning how to code? I first started thinking about this back in 2015 or so. I thought that security people would become developers. That's clearly not happened, but is that happening at North Dakota; are your sock people coding in this way? Or are you leaning on the IT developers to do security things?
Kevin Ford: So, I will say coding has become more important to us as we've gone down this journey. In fact a lot of the new employees that we're hiring here, we're looking for coding skill-sets. Familiarity with Python, being able to script, so on and so forth, and that's been very important to us with our ability to set up the soar tool, being able to integrate with the different API's and systems, or write those scripts. That's been really important, but another interesting thing is, the soar tools are becoming more and more approachable. Maybe if the soar tools become more approachable, and the CACI solutions become bigger, and as we migrate more and more off our data-center on the grounds here and into the cloud, potentially we may not need that capability.
I think ultimately what I would like to see is my position as CISO become less of a security; a vertical where I've a whole ton of security people reporting up to me, and one which is where I am more of a leader of a community of practice within the state. So the security people, once we achieve full cloud and full CACI, then get mixed into the agile teams creating all the different products. Then I become more of the leader of that community, but they report to the product people, rather than to a CISO. When you're on those teams one of the principals of DevSecOps is trying to create poly-skilled teams, and so it would be great if when we make that move eventually, years from now, they would all be able to code. The developers would all be able to understand security, and both would be able to understand operations and what they need to do to roll out a server in the cloud, or something like that.
Rick Howard: That was Kevin Ford, the CISO of the State of North Dakota, and a regular contributor to the CyberWire hash table.
Rick Howard: One short diversion; I've been having this running argument with a good friend of mine, Steve Winterfeld, the Advisory CISO for Akamai, and another regular contributor to the CyberWire hash table, about my obsession with pursuing DevSecOps with all vigor. He has a different take. Instead of focusing on a green strategy of infrastructure is code, something that would take years to deploy, he thinks we all would get more bang to the buck if we focused actively on securing API's. Here's my short conversation with him.
Rick Howard: So for the past few months you and I have been talking past each other [LAUGHS], regarding concepts of security orchestration and DevSecOps. You make the distinction that we shouldn't be focused on DevOps to do orchestration of our security seg. Instead we should be focusing on API's, providing them to our internal and external customers, and monitoring them as an often forgotten attack service. I say that you can't do security orchestration of DevSecOps without API's; that you can't implement the infrastructure as code philosophy without them. Is that not your proposition? Is that a distinction without merit? Tell me why I've got this wrong, because you and I have been arguing about this forever.
Steve Winterfeld: I think where I'm frustrated is, you've made it an absolute, and I keep coming back and saying, that's the norm, that's the majority. It's not an absolute. You can do DevSecOps without API's, you can do API's without DevSecOps, and consequently from a security point of view I want to talk about how the API's are working, not where we're hooked into the pipeline. How are we protecting API's? How are they thinking about API discovery? It's just like with web pages; we have people deploying API's without bringing in the security team. Some of those traditional guard rails we have on things getting published, a server internet faced thing, may not have those same guard rails around an API internet facing. And so I tend to want to talk about the API rather than the larger, cultural process of DevOps.
Rick Howard: Why? What's the reason for it. How does that help you narrow the problem, how does that help a CISO?
Steve Winterfeld: I think the CISO is addressing the infrastructure. The API is what we need to protect. And the DevSecOps is behind it so, I can protect API's holistically, or I can talk about, did you hook in these things? And ultimately you need to do both, but it's just about where you want to focus your discussion for me.
Rick Howard: Steve, I know you're a huge fan of watching historical trends in cyber security, so have you seen any here? Have we anything to learn from here? Give us your wisdom.
Steve Winterfeld: I know you love the history, but I come from a heavily audited background. PCI, finance, and if you're going to be regulated, if you're going to be audited, you want to build a program that has best practices, industry standards, leveraging the regulations, and this is an interesting area because you talk about history. We're so young here that when you look at NIST, which is one of my foundational standards I like to look at, they have a glossary definition for DevOps, but DevSecOps is a sub-line, and there is no definition; it's just a one line within DevOps. Even Wikipedia, which we often go to since there is no clause to refer terms. There is no DevSecOps for Wikipedia, it's a sub-line within it. This kind of goes to me that we have this thought process that we need DevSecOps, but ultimately do we need a Dev Quality Ops? Do we need a Dev testing Ops?
It's a component of it so I like to think that DevOps includes testing it, includes quality assurance, and a mature program does all that. It's real interesting to see a lot of the regulations and industry best practices have caught up with that.
Rick Howard: Clearly if you read from history you would know the answer to this perplexing problem that you have. [LAUGHS] When we say DevSecOps we don't say, we're just doing the security piece. No, that's not what it is at all. What we're trying to do is say we want to include security, and all that infrastructure is code stuff that you guys have been doing for a decade, that we haven't taken advantage of yet. That's what I think it needs, even though this one line in the NIST definition.
Steve Winterfeld: That's why I said, a mature program includes all of those.
Rick Howard: That was Steve Winterfeld, the Advisory CICO for Akamai.
Rick Howard: Next up is my conversation with Omer Azaria, from Sysdig, our show's sponsor. He is the VP of research and development at Sysdig; a graduate of Ben-Gurion University with a computer science degree, and has been a software engineer for well over a decade. I started out by asking him to describe the DevOps philosophy; or as some people call it, the infrastructure as code philosophy.
Omer Azaria: Maybe taking a step back before jumping into infrastructure as code, what we see in the industry and amongst companies, is that as people are transitioning to the cloud, and starting to deploy more things into the cloud, usually the first set of steps that are taking is just playing with the constant UI, or using a bunch of scripts to speed up services, the applications. Then they realize that that methodology can not really be replicated to other environments, in case they want to go into other regions; maybe other cloud vendors, or even just for the ability to replicate what they have right now in case the environment goes down. Then they start realizing that they need something different, and they come across infrastructure as code.
So infrastructure as code is the ability to go and express your infrastructure as something that more resembles code, or programming languages that are out there. The most common one I would say is Terraform but we do see other types of infrastructure as code out there. But again the goal is to go and express how you deploy things, all the way from networks accessibility to your compute and storage, just using the serve instructions that you can go and apply, and will generate the same type of environment that you have right now. So this is infrastructure as code. When we're looking at securely baked into infrastructure as code, that obviously creates a huge opportunity to go and shift security even further left than just doing it as part of the CICD. Again, maybe I'm getting ahead of myself but the ability to go and look at infrastructure and code and identify security issues, right there when you're running it, or when you're pushing it into some git repo is a huge advantage in the emerging cloud domain.
Rick Howard: I totally agree with you, and ever since I read Gene Kim's book, the “Phoenix Project,” I've been a fan of this philosophy. He published that in 2013, I think I first read it in 2016, and I know that the cyber security Canon Hall of Fame committee inducted it in 2017. I thought back then that this concept was perhaps the most important innovation that has happened to the IT sector, and the security sector since the invention of the personal computer. That's how important this thing was going to be, because it forced us to think of the security apparatus as a whole; as a system of systems, and not individual terms of the crank for each specific tool in the security sect. I just knew that it was just going to be a matter of a few short years until all of us had switched over to this new way of doing things. But nothing happened. It's eight years since the book was published, and my experience is that security organizations of the world are still maintaining their security stack manually. They're not even close to infrastructure as code.
I'm wondering if that's your experience too, and what are the barriers preventing security organizations from being further along this path than we are right now.
Omer Azaria: Absolutely. I'm seeing a very small fraction of organizations that are actually embedded or embraced by infrastructure as code. Again I think it's just how slowly we're transitioning to the cloud, and doing it the right way, which more of us agree that using infrastructure as code. For me one of the main barriers is the complexity that is involved in the migration, the transition; and people are taking baby steps and slow steps, it's really slow. We're getting to a point in which just the heavy lifting, the famous shift and lift from en-point applications to the cloud takes so much time that the current processes that are being used for security and for operations are still in place. They're not changing as well, or moving to something that is better. That is what we're current seeing. Even when we get to a point in which people are operating in the cloud on a day to bay basis, just realizing there is a better way to do that.
Just thinking that they need to go to another migration of how they do things in terms of deploying their software, and expressing their infrastructure, it is such a new thing that there is a really slow adoption of best practices, like infrastructure and code. What I hope is going to happen is that in the next year, or three years, we're going to see a more exponential adoption of that, and that more companies are going to shift to using that and as a result most processes and security is going to improve dramatically.
Rick Howard: What you said before is absolutely right. One of the reasons we wanted to go to the cloud is that we were going to be able to automate a bunch of stuff and take some of the complexity out of configuring environmental boxes to do things for us. Once you've got it in code it would be consistent, and there wouldn't be as many mistakes and you know, blah, blah, blah. But what you said is true; most people are just going very slow. There's a number of ways that any organization can try and do this, so let me ask you this, for your customers that are trying to adopt this model for DevSecOps kinds of things, or call it infrastructure as code for security type things, do they wrap themselves into the larger organizations DevOps movement, or are they trying to do it themselves within the security organization. What's going on out there in the community?
Omer Azaria: We see two types of things that are happening. Some companies do stay with the traditional; we have security team versus development team, and then security teams trying to shift left. So within a security team you see more people trying to identify those configuration issues, or work more like DevOps people that are looking at their run time environment and trying to fix things like the code. Nobody is actually fixing them but the pro-team developers with the suggested fix. Another type of organization who are really shifting security left, and this is where you see DevSecOps team and DevOps teams working side by side as part of engineering or broader engineering groups. For both of them those types of submissions are really necessary, A, from a visability standpoint, B, for ease of use. They do have their own day jobs, so they need to try and do less on that front, or at least have teams that can help them fix those issues.
Again, to your question, we do see those two models, but they're reusing the same tooling, or at least from use case perspective. There is a convergence of the use cases though.
Rick Howard: If they're trying to wrap themselves into the DevOps model, they're part of the team that designs the infrastructure. So for DevSecOps fans, you get to talk to a bunch of them clearly, are they using traditional DevOps tools to automate everything? Like Puppet and Ansible or Nagios or Jenkins? Or are they trying to use traditional sock tools like soar tools and seam tools? How are most people doing that?
Omer Azaria: I don't think there is most at this point. Everyone is using their own tool and it's all over the place, which makes it slightly more difficult. You step into an environment and you see the different type of seams and different type of soars that are being deployed and so on. This is one of the challenges for us as a company, because the expectation is that we will be able to integrate and work with all of them. Or, very smoothly transition from what they're using right now into Sysdig. So it's definitely a challenge, but at the same time the fact that they have those tools in place means that they're aware of the problem, and they are trying to do better at fixing it. This is is usually the reason the two are there.
Rick Howard: If I was going to start this journey today Omer, can you give a single piece of advice? What's the first thing we should be thinking about? How do I organize my thoughts? How do I make sure I can demonstrate success so I can continue to pursue this interesting model?
Omer Azaria: That's an interesting question. I would say the first thing I would do is get great visibility into my runtime environment. Most likely I have something running, and most likely I don't have good enough visibility into what is out there. From a security perspective, what is my security posture? Where am I exposed. So getting what is the current status, from a security perspective, my environment is the number one thing that I would do. Then mapping the risks. The second thing you want to do is to prioritize. Out of all those finding that I got, what is the most important one, or what are the five most important ones. I can do two things; A, go and present that to my executive team, or to my manager and B, start addressing them so I can see real progress in this journey for a better security hygiene. This is what I believe everyone is after, just a better hygiene overall; for me this is security. This is the way that I would go about it.
Rick Howard: If I was doing it I don't know if I would go after the most important one, because if it failed, if we had some trouble with it that would reduce the confidence in the leadership team on this new idea. I would pick something important, but if some aspect of infrastructure is code that wouldn't bring the business down if we screwed something up. I would find a nice middle ground, what do you think about that idea?
Omer Azaria: You're right. I mean the reality is that people are doing that. When looking at some sensitive application, let’s say you're working for a financial institution and the top item in your security list is changing; modify the configuration of that application, you usually try and stay away from doing that. I agree with you, because you don't want to have the business impacted. You choose a good balance between impact or potential impact of the environment, and security value that you're getting out of it. So yes, you're right.
Rick Howard: It's my impression that organizations that decide to flip the model, this is basically everybody go 180 from manually doing everything themselves hardware wise, to going to a software model; I believe you're not going to get this done in an organization from the bottom up. This has got to come from senior leadership at the top of the company saying, "You know what, we think the company will do much better if it adopts a DevOps model," and then pushing it from the top down and making everybody comply with that. Any thoughts on that one way or the other?
Omer Azaria: I agree with you. I just think that the conversation starts in a different place. I think the adoption of DevOps methodology is the CICB is the first and foremost for speed. Everyone is participating in conventions and seeing the numbers out there. The realization, if you shift to that model you can be better as a business and if you want to stay competitive you have to adopt that model. So that is, I would say, how executive teams and how companies usually start this journey to move to the cloud, towards a more DevOps type approach. Then when you have the DevOps type approach embedded or at least there is a transition into that lead organization, then the question about "Okay, how do a bake security into that?" comes into play. This is where we see the question coming up; "What's the right way to do that? What's the right way to have a secure DevOps approach in my organization that addresses both those issues?" which is speed and security at the same time and I think this is very much achievable as we're seeing in more mature organizations.
Rick Howard: I love the way you said that and I just had an epiphany listening to your explanation. If you're in an organization and the leadership has decided to move to DevOps, and you're in charge of the security part of it; don't wait, jump on that and help them get there, be part of that solution. Don't let it get separated that you have to do security later, after the DevOps guys figure out how to do it. As soon as someone decides we're going to go this way, get security in there early so it's not a weird thing two years down the road, that we're going to try to throw security stuff in there. That's what I got out of that.
Omer Azaria: The earlier you are, the better. There is this gap that is open, it's very hard to go in and catch up, because processors are being established and then go in and bake the securities into the pipeline, into the infrastructure as code, it becomes very very difficult rather than doing it from day one. I completely agree.
Rick Howard: Omer, what's the bottom line here. If you could tell our audience anything what is the one message that you want to give everybody?
Omer Azaria: Just one? Come on. Baking security into the life cycle of the application, actually creates better security which means that if you push security left as much as you can, infrastructure as code, explaining your obligation, your containers and so on, eventually will get you better security overall. It's never too late to go and try and do that. If there is one takeaway is look into infrastructure as code. Look at how you can improve security, [UNSURE OF WORD] shifting to that, that would be it.
Rick Howard: That's Omer Azaria, the BP of research and development at Sysdig. Omer, thanks for coming on the show.
Omer Azaria: Thank you, Rick.
Rick Howard: We'd like to thank Kevin Ford, the CICO from the State of North Dakota, and Steve Winterfeld, the Akamai Advisory CICO, both our own CyberWire Hash table [UNSURE OF WORD] experts and Sysdig's Omer Azaria for joining us. CyberWire X is a production of the Cyber Wire, and is proudly produced in Maryland at the start up studios of DataTribe, where they are co-building the next generation of Cybersecurity start-ups and technology. Our Senior Producer is Jennifer Eiben, our Executive Editor is Peter Kilpe and I am Rick Howard. Thanks for listening.