Securing the Unsecurable: Inside the Black Hat NOC and Zero-Hour Resilience

James Holland: You need context. Context says if this attack has come from a training class to a benign destination or a destination that is part of the training class we don't want to block that. That's the whole point. That's the part of the training class. If this is someone from somewhere else on the internet attacking the registration server we're definitely blocking it and we're putting your IP on a block list straight away.

David Moulton: Welcome to "Threat Vector," the Palo Alto Network's podcast where we discuss pressing cybersecurity threats and resilience and uncover insights in to the latest industry trends. I'm your host David Moulton, senior director of thought leadership for Unit 42. [ Music ] Today I'm speaking with Jason Reverri, manager of technical product engineering at Cortex at Palo Alto Networks, and James Holland, distinguished engineer for technology innovation at Palo Alto Networks. Jason has spent years bridging technical product engineering with front line cybersecurity defense particularly in threat prevention in Cortex product development. James leads technology innovation at Palo Alto Networks and is a long time expert in automation, security architecture, and real world incident response, especially under pressure. Today we're going to talk about what it takes to run the network operations center or NOC at Black Hat, one of the most uniquely complex cybersecurity events in the world. With a bespoke network, global visibility, and close to a billion threat events logged annually, defending Black Hat is a real world exercise in zero hour resilience, collaboration, and automation. From live fire threats to misconfigured VPNs Palo Alto Networks and its partners including Istoria, Cisco, Corelight, and others must protect infrastructure, analyze traffic, and support learning in a high trust high stake environment. Today we're going to explore the role of Cortex XSOAR, XSIAM cloud delivered security services and high availability firewalls in that environment, and we'll talk about how automation and AI driven playbooks are changing what's possible not just here at Black Hat, but in enterprise SOCs around the world. Jason, your article, "Not a Knock Knock Joke" which, by the way, fantastic security joke, you've highlighted how the NOC team had to defend against real on the fly exploitation of vulnerabilities that were disclosed moments earlier during talks. Can you share one of those moments and how your team responded to something that literally just hit the floor?

Jason Reverri: Yeah. Absolutely. So thank you for taking the time to chat with us today. We really appreciate it. You know, Black Hat's really unique because, you know, we are -- we are here with some of the most advanced trainers and researchers in the world. Right? And so on a regular basis somebody will go up on stage and talk about something that is brand new that has never been seen before and, you know, we're on the network looking at the data, protecting the infrastructure, and we'll see people start to mess around with those sorts of activities. Right? One top of mind one that I certainly can talk about is I believe it was at Black Hat London right before the start of the show. A really well known security researcher talked about some SSH vulnerabilities and you know the team had to move very quickly to find detections for that, and it was great because it was really a partnership, and that's what Black Hat's all about. Right? Partnerships with folks that, you know, out on the street might be our competitors, but we're partnering together to service our customer Black Hat. Right? So, you know, the researchers from Palo Alto Networks' Unit 42 were there. We had our partner Corelight there. And they had developed detections that we then implemented in to Cortex XSIAM so that we could understand if we were actually seeing that type of traffic on the network and how we can pretend -- protect our infrastructure that's, you know, one of the main charges that we have here at Black Hat. All right. So the infrastructure being, you know, the registration server world, attending PIIs, maintain, that sort of thing. So.

David Moulton: James, let me take it over to you. Why does Black Hat need its own network?

James Holland: It's a very good question, David. And the reality is following on from what Jason just mentioned the environment of this conference is lots of people with extensive research, extensive experience, people who develop exploits for vulnerabilities, find vulnerabilities for fun. It's quite a hostile situation out there. You've got people being trained in the training classes at all levels, to be fair, from basic to advanced. Then you have people coming in to deliver briefings to talk about the latest threat actors at their -- in the big bad world. You put all that together and it's a network that you can't just leave wide open. You need visibility. You need control to be able to manage that situation. And you can't have people on the network attacking each other, for example. You couldn't have people from one training class attacking people in another training class. And I mention things like that because in the distant past that's what happened. I think there was times where the network was down more than it was up, people attacking the infrastructure itself. So they were able to go down to the business hall at Black Hat and, you know, talk to some of the vendors at their, you know -- talking about their products and services and well we need some of this. We need vendor X, Y, Z. You know, you deliver network security, you deliver end point security, you deliver identity. We need some of this. Do you fancy helping us? And unsurprisingly every vendor pretty much said, "Yes. We would love to. We would love to be involved. How can we be involved?" So they had the fortunate situation where they could pick from the best of the best because everyone was saying, "Yes. We'd love to help you. How can we help? We'd love to be involved in Black Hat and be in partnership with you." And so that's how the last few years evolved, you know. From Palo Alto Networks' perspective we've been involved eight years in a row now as a network security partner helping them deliver the network segmentation and control they need, the visibility they need. And, you know, more recently in the last year or two from a security operations perspective Cortex platform is coming in to help them, to help us all, with all the partner vendors, you know, have a single place to do threat detection, threat hunting, and incident response. So yeah. That's the backstory as to why they need this and, you know, how they got to where they are today.

David Moulton: Talk to me about Palo Alto Networks' next gen firewalls. Like the PA 5280 or the PA 5430, are those deployed in the NOC?

James Holland: The PI 5430s we use now we move from the 5700 series to the 5400 series. And they are the -- they are effectively the core of the entire network that is delivered for the Black Hat conferences. The network is extremely highly segmented. Wherever you are at any point in the network you can't really do much without going across the firewall. So it gives us pretty much ultimate visibility and control. They are -- as you mentioned, they're delivered as a pair, the high availability. This is the Black Hat NOC that we're talking about, not the SOC, which is quite a talking point sometimes because people go, "It's a security conference. Why isn't this a SOC?" Well, the number one priority for the NOC leads, for the NOC management, is still performance and availability. You know, people are paid a lot of money to exhibit here, to do training classes here, and so on and so forth. And they deserve an environment that works. They deserve to be able to do what they're learning without worrying about slow speed or without having outages or anything like that. Or if someone from another class tried to take their class infrastructure over they deserve to be protected from that as well. You know. So the firewall is to avoid that. We can deploy them. We're here a few days before the conference starts for real. We get to go and install the firewalls down in the coms room here at the Mandalay Bay conference center. We get amazing access and partnership with the venue to be able to do that. So we go and we have a pretty much best practices design of HA firewalls, high segmentation. Gives us great visibility, great control. We work with the Black Hat leadership team to deliver a policy that helps them enforce the rules that they want to see. So no training classes can fight with each other, but everyone can get out to the internet. And if people are in a part of the network where they should be able to deliver a demonstration of something that looks malicious, we won't block that. We'll actually let it through. We'll see. We'll detect it. But we shouldn't be blocking that because we don't want to break their demo. But if you try and attack something like the part of the network where the registration data is, so us as NOC staff and all the attendees of the conference, all of that personal information is in a big database, there's a web wrap in front of it, we detect that and we guard that. That's our crown jewels, the cliche of security. Right? So we do detect that. We don't just alert. We block on that. And we regularly see people take -- take a punt at that infrastructure, see if they can get in. And obviously pretty much all of it I don't think we've ever really had a successful attack. We've seen people try, particularly because it's open to the internet as well on purpose. It has to be by design. So we will get people scanning it from the internet and poking it and prodding it and it's interesting to see them have a go. You come to Black Hat and you do a training class. That activity is still as it would be anywhere else in the world. You know, if you try and exploit a retail organization's website that's still illegal whether you're at Black Hat training class or not.

David Moulton: Right.

James Holland: But if it's something that is expected from the training class and they are delivering an exercise in their training class against some test infrastructure designed for that purpose we call that a Black Hat positive which is a subtle twist on a true positive. It is a real attack and it's not a false positive because well the CDSS signatures I've seen said, "Hey, we've seen a SQL injection. We expect to see that." As long as it's not against the real live website that's okay. So we have that extra sort of code as you will for an incident which is Black Hat positive. It's a real attack, but we're okay with it.

David Moulton: Jason, what are the common threats that you see and maybe how do you go about prioritizing them?

Jason Reverri: So top priority is protecting the infrastructure and, you know, the registration, all sort of stuff. Right? Anything that could in any way make the network experience not optimal for attendees or, you know, in some way compromise the crown jewels. Those certainly would be top priority. After that, you know, from a network forensic standpoint a lot of the data out there is encrypted now. So what boils up pretty quickly after that is unencrypted data. And, you know, you're starting to see like obviously since we own the infrastructure we can decrypt the traffic on our infrastructure, but as a security conference there's no way that somebody's going to come to a security conference and allow us to put, you know, a certificate on their computer, their device, so that we can decrypt the traffic and see it fully. Right? But we still see clear text traffic all the time. We see clear text traffic that has very sensitive data and when -- again when we can establish, you know, a change of ownership of that, we will reach out to the person and let them know, you know, "Hey, you don't want to pay your mortgage once you're at Black Hat because we're seeing that payment in clear text. That system's not as secure as you think it is. And that's kind of what we're seeing. It's not like people are maliciously doing things. It's just maybe an application's poorly written and it encrypts the login maybe, but then the data that it's sending for that app is in the clip. And so a lot of times we'll try to forensically understand what that application is and then reach out to the application developer and be like, "Hey, do you know you're sending your customer data in the clip?" So there was definitely a situation where that happened very recently with a weather app at Black Hat Singapore, that you know the app was just in the clear leaking the person's location data, where they were, so that it could do weather tracking. And it was a very, very popular Android weather app. So.

David Moulton: James, I see you shaking your head. I assume that you knew about that story. What goes through your head when you see that kind of data being broadcast out?

James Holland: I felt sorry for the users number one because I don't think in a lot of cases that they realize this, you know. I think people use applications like that from an app store, from a trusted source, believing in good faith that the developers will have made sure that the transport of credentials, the transport of data, is encrypted and everything is done to best practices. And it's hard as an individual to really go and audit that. So you have to put some trust in some things sometimes. And, you know, ideally we'd all be in zero trust mode for everything, but sometimes it's not entirely possible for one's entire life personal or corporate. As Jason said, when it's possible, particularly if people are transferring data with personal information in the clear, it's actually very easy to identify them ironically. We can see their name. We can see their email address. It's actually quite easy to get in touch with them and try and help them remediate that. And so they leave better than when they came. And so yeah. You just -- we do try as much as possible, and that's the culture that the Black Hat team have kind of instilled and that we've all inherited and worked upon as partners with them to try and help people leave in a better state than they came if they do have unfortunate scenarios like that. [ Music ]

David Moulton: I want to talk about the combination of AI and humans, the artificial intelligence and the actual intelligence. Cortex XSIAM, you mentioned that earlier, is now automating the majority of the initial triage. How has that changed your operations and what are the analysts freed up to do and focus on now that AI's handling maybe that first chunk, that first 80%?

Jason Reverri: Yeah. That's a fantastic example because, you know, we have some really really great partners in the other vendors, and everybody has their own threat intelligence sources that they're bringing. Right? And so a great example of automation that we've had for a couple of years now at Black Hat when there is an incident we're enriching data around that incident with all of this threat intelligence that our peers are bringing. Not just our Unit 42 data, but then also, you know, Cisco Talos data is coming in. You know, we're seeing that. You know, Corelight's threat intelligence team is providing data. And so as an analyst you sit down. You want to work an incident. You want to get information. All of that is prepopulated for you. So you understand the context around, you know, the type of traffic, you know, the source destination, what we know about that immediately. So immediately if there's an incident that data's pulled out and enriched in there so that, you know, as an analyst everything that you would want to possibly ask of it we're trying to prepopulate and have that immediately in the incident for you to work. So we're talking hours' worth of time saved just in that alone.

David Moulton: Cortex XSOAR is central to the NOC orchestration and integration. Can you walk us through maybe a real world automation, you know a DNS reroute or a firewall rule change that took place during a live incident?

Jason Reverri: So yeah. I mean XSOAR's actually on the autonomous platform that Black Hat shows and it was how the whole Cortex platform came in and was selected by Black Hat. We really established ourselves from an automation standpoint, and that's, you know -- that's really the reason I want to bring it up is, you know, it was a -- it was some very talented engineering people that we continue to have in the NOC with us representing Palo Alto Networks that were able to do that. Right? You know, the Black Hat staff saw a need for automation and they're like, "We just can't scale." Right? And the engineering team that we had, that we were using as support, were like, "Well, you know, we've got a SOAR product that we can do that with." You know? And so then you know that show -- they showed them what we could do and they're like, "Great. What else can you do?" You know, so the next one we're building more. We're building more. So yeah. It's been very very critical from an automation standpoint. One of the greatest use cases that we have, I think personally, because it benefits the Palo Alto Network's team directly, is we have like our morning checklist. Right? You go in in the morning. We want to make sure everything's working. Right? You want to make sure the firewall's are up. You want to make sure the registration's up. You want to make sure there were no outages over the evening, you know, because when the conference closes for the day everybody goes out and has like a beer, whatever, you know, and relaxes. And we don't think about the NOC. Right? But in the morning at 6 AM when we're here before registration opens we want to make sure everything's dialed in. And, you know, in years passed we would go through that checklist. We were very very operationally focused, you know, making sure things are up. You know each day we, you know, check at green. Okay. We're good. You know? And we've automated all that now. And so, you know, at 6 AM, you know, when we're walking in we get a notification in our collective chat app and then it's like, "Here's our report for the morning." You know? And all the data sources are still coming in to XSIAM. Everybody's happy. All the agents are happy that we have deployed. You know, regis is up, you know. Or if it's not then, you know, maybe we're running to the NOC to find out what --

David Moulton: Well, yeah. Jason as you were describing that my thought was if everything's green for go, great, but if it's not you've given yourself a little bit more time to figure out how to mitigate that problem. You know, how to shift to plan B, whatever it is. And you're not getting through your checklist and getting to item number 17 40 minutes later and going, "Oh no."

James Holland: Oh yeah. I mean I absolutely agree. We've worked hard over the last few years to build in a lot of proactive monitoring. So we want to know about something being an issue before any attendee or anyone else in the conference comes to complain to the NOC. We want to already know about it. And so there's a lot of proactive monitoring, a lot of proactive check in, and you know again six/seven years ago we started off doing this manually and now we have the automation platform to be able to just deliver it in an automated fashion and it tells us not -- we don't check it. It tells us. So and we even check in that everything is operationally fine first thing in the mornings. Brilliant. We take -- constantly take, for example, backups of the file configuration on the hour every hour because if we're making changes as things are happening we want to be able to roll back easily. And even just things like a little ping saying on the top of the hour yep the file backup was successful. Great. We've built a lot of that in. And to your early question about the automation of file changes and things like that, the DNS rewrite, we've built a lot of rules that are dynamic such that if we do detect something through the SecOps platform it could automatically without any human intervention put a block in place. It can only do that with the context that it has that Jason and the team have built out because you need context. Context says, "If this attack is coming from a training class to a benign destination or a destination that is part of the training class, we don't want to block that. That's the whole point. That's part of the training class." If this is someone from somewhere else on the internet attacking the registration server, we're definitely blocking it and we're putting your IP on a block list straight away. And so that's where the combination of the alert for that attempted exploit or whatever it is that this person on the internet has done or machine or script or whatever is has attempted to do something that is obviously malicious. It's flagged up. It's blocked. But then the SecOps platform with Cortex analyzes that, agrees with it. It knows it's from the internet, not from a training class. So we do want to put a block in place and the block goes in to the firewall and no humans are involved.

David Moulton: Black Hat's NOC is a really collaborative space. Lots of different vendors working together. What makes this multi vendor integration work in real time?

Jason Reverri: So first and foremost Black Hat is the customer and this environment I think is a practical representation of what a lot of our customers are experiencing. You know, either it's mergers and acquisitions or, you know, there -- you know, business decisions were made, whatever it is. Not everybody has the ability to just execute on a single platform. I mean yeah our products work really really well together. We have a fantastic platform. But whatever situation not every customer is able to do that. So this is a real world example of how partners and integrations and systems can all work together to service our customer Black Hat. And so it's a -- it's the charge. It's why we're here. You know, it's the expectation from Black Hat that we are all going to work together. Any beef we have out on the street we leave it out there. And in there, you know, we're friends. We're partners. I mean, you know, the team lead from Cisco is a good friend of mine. You know? And I consider a good friend outside of here. Right? You know. And we work great together, the teams. The collaboration's great. You know, a lot of times, you know, somebody has a new toy they're working on and they're like, "Hey. Look at what we can do." And, you know, then you see two engineers just working together on that. Right? Yeah. You know, it's great. And that's -- that's what I love about this. It has a very community feel, but also you know there's that level, that expectation that we're going to achieve and execute for our customer Black Hat.

James Holland: Yeah. If you looked at the chat app and you looked at my direct messages the majority of them are not to other Palo Alto Network staff in the NOC. They're to the other partner vendors. They're to Cisco, to Corelight, to Arista, to Lumen. We do work collaboratively. We have collaborative spaces where we can share information and documents and things like that. And to Jason's point that his -- that's real world. No one has a single vendor for their entire IT staff. That's not even possible realistically. So this is -- this is the real world. And we deliver similarly to how our customers and partners would look to be doing themselves in that you have a partner like Arista for wired and wireless networking. We're delivering network security and security operations platforms as Palo Alto Networks, Cisco doing their device management for the iPads that do registration. They've got their threat intelligence tools and things like that. Corelight are doing the NTA and NDR. All these things have to work together in order to get the value out of them. And for us to deliver that for Black Hat we do some of the things the customers would do. We sit down in meetings. We meet weekly, especially in the lead up to the conference. We have -- we have an off site every year where we think more strategically long term about where we're going. Things like the rise of encrypted data over time, encrypted DNS, things like that pose a risk to visibility while they enhance privacy. They do mean that in the future, for example, we might lose a bit of visibility. That's by design. So we talk about how we can do things, how we can protect users, how we can help them with that, about how we can still deliver the infrastructure for Black Hat with the security and visibility that they need for this conference to operate.

David Moulton: So at past events I know the teams had to pivot. There were shipping delays and I'm sure other things that have gotten in the way. How do you keep the NOC agile enough to handle those like on the fly changes, but keep that network secure?

James Holland: It's a tough challenge. The great thing that we have is that if something like that happens in flight everyone is there in the same room from all the partners including Black Hat management team. So we have all the right people in place with great minds and great experience to then back ourselves I guess to deal with most things. I can't say everything obviously, but there was that one incident in particular where there was a shipping delay. There was an airplane that was outright frozen to a runway in central Europe. And so some of the server infrastructure upon which the conference relies straight up didn't make it to the conference in time. And what do you do? This is largely an on prem deployment because everything is on site. The classes -- the training classes are on site. The briefings are on site. The talks are on site. The arsenal is on site. So normally there's not much in the way of cloud infrastructure. There's cloud delivered security services, but the infrastructure itself is all on site. But when the infrastructure isn't there to be on site you have to find another way. The cloud was our backup plan effectively. We didn't know it at the time, but that's how we pivoted quickly. We're like, "Well, we need somewhere to host some servers because the servers aren't here. Where could we?" We need them realistically within 24 hours. So we're not procuring servers within 24 hours. This particular event the build happens over a weekend. So you're even less likely to get anything at a weekend, but it's not the kind of thing you can go and buy out of a store either. So yeah. We're going cloud. We have to now build some servers in the cloud. The team that looks after the registration service are there. So they're like, "Okay. Yeah. We can -- we can pivot what would have been on those servers and put them in to some virtual service in a cloud provider." As Palo Alto Networks we can then pivot and instead of having PA series physical firewalls on the site we have virtual firewalls in the cloud. We can then link the two. We can bring up a secure VPN to our traffic to flow between them. And then hey presto we have the scenario where an attendee would arrive at the conference with their QR code ready to get their badge printed. It gets scanned by an iPad which is on site. Data goes up to the cloud, gets processed, checks that the user is a valid user and what needs to be printed on the badge by the servers in the cloud. Comes all the way back on site to the printer and hey presto we have a badge. And we've saved the day from a weather incident. But we could only -- we could only do that because everyone is on site. Everyone's very experienced, very talented, to be able to think quickly for alternative solutions and then be able to actually deliver that solution, and in that case it was within a few hours that -- from the point at which we realized the servers weren't going to be on site to almost a working solution was a few hours. And then we had to finesse it a little bit. Not something we'd done before in the history of Black Hat using partner vendors, but it was yeah. We got there. We got there in time. That's the main thing.

Jason Reverri: Yeah.

David Moulton: Well, you get there before the plane that was frozen to the runway.

James Holland: We did that [inaudible 00:29:51].

David Moulton: So how does Cortex XDR, unit 42, and the integrations with the other tools help surface high risk incidents like attacks on that registration system or even the internal infrastructure?

Jason Reverri: I mean that's what the systems' designed to do. Right? You know, and so we have agents deployed on the infrastructure that we own. So we see agent operating system level data and then of course we have our network firewalls. We get that log data in there and the system's designed to stitch that and create a story. And then run machine learning AI on that data to then boil that up to ultimately what's called an incident. Right? And so then as a responder we're able to see that incident and say, "Okay. You know, this is -- this is important enough that it -- the system has created an incident from this data." And, you know, now that we're the official sim for Black Hat we're getting data from Corelight. We're getting data from Arista. We're getting data from Cisco umbrella. And we're seeing all of that and that data's then being normalized, stitched together, and used to, you know, boil up these incidents and create it so that an analyst is able to get eyes on it and take a look at it.

David Moulton: Guys, as we wrap up here, can you talk to me about the top three lessons learned from operating the NOC that enterprise security leaders should apply? And, you know, maybe that's network segmentation. Maybe that's automation. Or even this cross functional coordination that you've described.

James Holland: I think from a network security perspective the visibility and control you get from a highly segmented network is so valuable. And it doesn't have to be painful to the extent where we would say you have to go and micro segment your network with rules from everywhere to everywhere because that would be incredibly painful operationally. But just by having next generation firewalls in the right place in your network even to start with with a fairly relaxed rule base the visibility you'll get before you even think about putting more rules in will give you so much more insight in to what's going on in your network and you will find so much more out of that that you probably never knew about. And that's something that I think everyone should be trying to aspire to, and that on its own is valuable enough for me to recommend it as really high up on someone's priorities. But then the fact that that alone is great, it has its own value, but it feeds in to what you would do from a security operations perspective because that visibility feeds in to SecOps which can now no longer, let's be honest, be done at human scale. Black Hat is the same as most other organizations in that the volume of alerts, the sophistication of the attacks, the speed at which adversaries are able to develop with AI just like everyone else is developing with AI so the adversaries so they're faster and more agile and nimble than they've ever been, and so if you want to be able to do SecOps properly you're going to have to be automated. You're going to have to be AI powered. And to do all that you need data. And from a network perspective if you're not in the right places in your network to get the visibility even before you try and do micro segmentation control having that visibility from the network is crucial in my opinion.

Jason Reverri: I would just add, you know, meet your analysts where they work on a regular basis. So one of the automation pieces that we built with XSOAR now, XIM, is a chat bot. And so you're in your chat tool communicating with other team members constantly. Right? And so being able to just make a query directly in to XIM from that chat tool like, "Hey, what do you know about this IP address?" essentially, right? And then it takes all the data it has and it kicks it right back to you in that conversation that maybe you're having with two other analysts investigating something. I think that's a great place to be. And it doesn't require people necessarily to be subject matter experts on a specific technology because it's using more let's say regular language or, you know, like maybe a special command to initiate that. I think that's something that we've gotten a lot of value out of. The other thing is you know when somebody sits down at an investigation kind of like what I was saying earlier have all the data there for them. Have automation in place, you know, bots go -- you know, AI bots going out and getting that information and getting it all back in to the system, scripts, whatever you want to call it. Right? You know, we're getting that information in the system so as soon as somebody's sitting down they have everything they need to do their job. And let the expert be the expert. Let them figure out what happened there and triage it appropriately. So.

David Moulton: Jason, James, thank you for coming in and sharing this snapshot of sort of a day in the life of a security professional in the NOC at Black Hat, sharing some of the insights, the learnings. I really appreciate you taking the time away from the conference today and sharing this with the "Threat Vector" audience. So, guys, we've just scratched the surface today and for those listening I want you to keep an eye out for a NOCumentary that's going to be coming in October, part of our cybersecurity awareness month where we do a deep dive with Jason and James and those that are in the NOC here at Black Hat protecting the infrastructure, protecting the client. And as somebody who's had a little bit of a lens behind the scenes, I can tell you that this is a massively fascinating, massively fascinating, NOCumentary. Heard it here first. So that's it for today. If you like what you've heard, please subscribe wherever you listen and leave us a review on Apple podcasts or Spotify. Your reviews and your feedback really do help me understand what you want to hear about. I want to thank our executive producer Michael Heller, our content and production teams which include Kenne Miller, Joe Bettencourt, and Virginia Tran. Elliott Peltzman edits the show and mixes the audio. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now. [ Music ]