Infosec teams assessment: An essential tactic for risk forecasting.
Rick Howard: Hey everybody Rick here. Out of all the capabilities in the infosec community that have improved over the years, the one essential skill that hasn’t moved forward is calculating risk. Specifically, how do we convey risk to senior leadership and to the board?In my early network defender days, whenever somebody asked me to do a risk assessment, I would punt.I would roll out my qualitative heat map, a fancy name for a color coded spreadsheet, where all the risks are listed on the X axis and my three levels of potential impact high, medium, and low are plotted on the Y axis and call it a day.
Rick Howard: Along with many of my peers, I would tell myself that predicting cyber risk with more precision was impossible that there were too many variables that cybersecurity was somehow different from all the other technical and scientific disciplines in the world, like physics and chemistry and orbital, mechanics, and space and it couldn't be done. We were wrong of course. The cybersecurity canon project is full of hall of fame and candidate books that talk about how to calculate cyber risk with precision. Books like "How to Measure Anything in Cybersecurity Risk," by Hubbard and Seiersen, "Measuring and Managing Information Risk: A Fair Approach," by my friends Freund and Jones, "Security Metrics: A Beginner’s Guide," by Caroline Wong newly inducted this year into the cannon hall of fame and you can catch my interview with her on this podcast as part of the bonus material last season, and finally, "Security Metrics: Replacing Fear, Uncertainty, and Doubt," by Jaquith.
Rick Howard: These are all great primers regarding how to think about precision probability forecasting. And I highly recommend them. If this subject is new to you, they will all change your current view of the world. But my problem with all of them is that I kept waiting for the chapter at the end entitled and here's how to do it or better building that risk chart you take to the board. None had it or anything close. That part was always left as an exercise for the reader. I decided it was time for me to roll up my sleeves and figure out how to do it myself. So hold onto your butts
Rick Howard: My name is Rick Howard and I am broadcasting from the CyberWire's secret sanctum sanctorum studios, located underwater, somewhere along the Patapsco River near Baltimore Harbor, Md., in the good, old U.S. of A. And you're listening to CSO Perspectives, my podcast about the ideas, strategies, and technologies that senior security executives wrestle with on a daily basis.
Rick Howard: The book that changed my mind that calculating cyber risk with some precision was possible is called "Superforecasting: The Art and Science of Prediction,” by Philip Tetlock and Dan Gardner, another cybersecurity Canon project hall of fame candidate book, Dr. Tetlock is quite the character. He's one of those scream and shaker raised fist at the TV because they have no idea what they're talking about. People he would watch news programs like CNN, Fox, and MSNBC, where the host would roll out famous pundits to give their opinion on some topic. Because once in their lives they predicted something correctly.
Rick Howard: It didn't matter that all the predictions they made sense were wrong. The news programs would still bring them on is that there were Moses coming down from Mount Sinai to present the tablets as law. Dr. Tetlock thought that they should have to keep score. I always thought that when pundits came on, the viewer should see their batting average only across the Chiron at the bottom of the screen,
Rick Howard: Rick Howard, the predictor from Paramus, the forecaster from falls church, the risk assessor from Rialto has made three correct predictions out of 20 tries this year. His batting average is 15. Maybe we shouldn't listen to closely to what he has to say. My apologies to the late great Vince Scully, the voice of the LA Dodgers for over 50 years, who just recently passed, but we couldn't help but appropriate his famous call of Kurt Gibson's walk off home run in the first game with a 1988 world series between the Los Angeles Dodgers and the Oakland A's. If you've never watched it, do yourself a favor, take 10 minutes and experience one of the greatest sports storytellers of all time. There's a link to it in the show notes, but I digress.
Rick Howard: And then Dr. Tetlock lock decided to test his idea, working with IPA, the intelligence advanced research projects agency. He devised a test using three groups, the intelligence community, the academic community in a group. I call the Geers on the go. Now the Geers on the go were not all old people. They were just regular people with time on their hands who liked to solve puzzles. According to the Washington post Ted lock had them forecast answers to over 500 really hard questions like will the Syrian president still be in power in six months?
Rick Howard: And will there be a military exchange in the south China sea in the next year? And will the number of terrorists attacks sponsored by Iran? Within one year of the removal of sanctions out of the three communities, the Geers on the go outperform the control group by 60%, they beat the academic teams from 30% to 70%, depending on the school MIT and the university of Michigan were two and outperform the intelligence groups who had access to classify information, but Tetlock also discovered a subset of the Geers on the go, the superforecasters. By the end of the four year tournament, these superforecasters had outperformed the Geers on the go by another 60% and could also see further out than the control group. Quote, superforecasters looking out 300 days were more accurate than regular forecasters.
Rick Howard: Looking out 100 days in quote and these superforecasters don't have extreme muttin abilities either. They are intelligent for sure, but not overly. So this isn't the collection of professor X's from the Xmen comic book. They aren't all card carrying members of Mensa and they're not math nerds either. Most of them only perform rudimentary math calculations when they make their forecast.
Rick Howard: But by following a few guidelines, they can outperform random Kentucky windage guesses by normal people like me. Like for example, number one forecast in terms of quantitative probabilities, not qualitative, high, medium, and low. In other words, get rid of the heat maps, embrace the idea that probabilities are nothing more than a measure of uncertainty, but also understand that just because the probability that something will happen is 70% doesn't mean it's a lock.
Rick Howard: See secretary Clinton in the 2016, us presidential campaign. Number two, practice, do a lot of forecast and keep score using something called the Brier score invented by Glenn Breyer in 1950, the score is on two, a axis calibration and resolution calibration is how close to the line your forecast is. Are you overconfident or under? Resolution is when you predict something is going to happen, it does. Number three, embrace firming estimates outside in first, and then inside out forecast outside in is looking at the general case. Before you look at the specific situation in terms of cybersecurity.
Rick Howard: That means the outside in considers the probability that any organization would get hit by say a ransomware attack inside out considers the probability that ransomware criminals will cause the material impact to your organization. See the difference both have merit, but Tetlock says to start with the outside in forecast and then adjust up or down from there with the inside out forecast. For example, if your outside in forecast says that there is a 20% chance of material impact due to a ransomware attack this year for all us companies, that's the baseline.
Rick Howard: Then when you do the inside out assessment by looking how well your organization is deployed against our first principle strategies, you might move the forecast up or down depending. So how do you make those outside in assessments? Well, the Italian American physicist in Rico Firmi was a central figure in the invention of the atomic bomb.
Rick Howard: And he was renowned for his back of the envelope, estimates with little or no information at his disposal. He would often calculate a number that subsequent measurement revealed to be impressively. He would famously ask his students. Things like estimate the number of square inches of pizza consumed by all the students at the university of Maryland during one semester. And he forbade his students from looking up any information. He encouraged them to make back of the envelope. Assumptions first, he understood that by breaking down the big intractable question, like how many inches of pizza consumed into a series of much simpler answerable questions? Like how many students, how many pizza joints, how many inches in a slice, et cetera, we can better separate the knowable and the unknowable, the surprises, how often good probability estimates arise from a remarkably crude series of assumptions and guesstimate more on this in a bit.
Rick Howard: Frederick Mosteller a groundbreaking imminence statistician in the 1950s through the 1970s said that quote, it is the experience of statisticians that when fairly crude measurements are refined, that change more often than not turns out to be small statisticians would wholeheartedly say make better measurements, but that would often give a low probability to the prospect that finer measures would lead to a different policy in quote, number four, check your assumptions, adjust, tweak, abandon, seek new ones and adjust your forecast from there.
Rick Howard: Number five, dragon. Consume evidence from multiple sources, construct a unified vision of it. Describe your judgment about it as clearly and concisely as you can, being as granular as you. And finally, number six forecast at a 90% confidence level. As you adjust your forecast, remember that you want to be 90% confident about it.
Rick Howard: If you're not, then you need to adjust up or down until you are the point to all. This is that it's possible to forecast the probability of some future and mind numbingly complex event with enough precision to make decisions. If the Geiers on the go can accurately predict the future of the Syrian president. Surely a bunch of no math CISOs like me can forecast the probability of a material impact due to a cyber event for their organizations. That's cybersecurity, risk forecasting. Tetlock, spends time talking about how the U.S government hasn't done this kind of thinking in the past. You and I would call them massive intelligence failures like WMD in 20 years of war on the slam dunk, CIA assertion that these weapons existed in Iraq when they didn't like the Vietnam war, 10 years of war on the widely held belief that if south Vietnam fell, the entire world would fall to communism, like dominoes leaders didn't just think there was a chance this would happen. They thought it was a sure thing. Like the bay of pigs, president Kennedy's political disaster. When the planners didn't consider the probability of success, when the plan changed at the last minute, and finally is Osama bin Laden in the bunker. Ted lock describes a scene in one of my favorite movies, 2012's Zero Dark 30, starring Jessica Chasteen the CIA director, Leon Pineta played by the late great James Gini is in a conference room asking his staff for a recommendation on whether or not Osama bin Laden is in the bunker. He's looking for a yes or no answer.
Rick Howard: One of his guys says that he fronted the bad recommendation about w MD in Iraq. And because of that failure, they don't deal uncertainties anymore. They deal in probabilities, which is the right answer, by the way, just not a very satisfying one. They go around the room and get a range of probabilities from 60% to 80% Chastain breaks into the conversation and says that the probability is 100%. Okay, fine, 95% she says, because I know certainty freaks you guys out, but it's a hundred percent, which is the wrong answer by the way. The probability was never a hundred percent, no matter how sure she was with her evidence.
Rick Howard: One note of caution. This next clip has some strong language. If you have sensitive ears about it's best to pause for a bit.
Male speaker: I'm about to go look the president in the eye, then what I'd like to know, no expletive . It's where everyone stands on this thing. Now, very simply, is he there or is he not expletive there?
Male speaker 2: We all come at this through the filter of our own past experiences. Now I remember Iraq WMD very clearly. I fronted that and I can tell you the case for that was much stronger than this case.
Male speaker: A expletive yes or a no.
Male speaker 3: We don't deal uncertainty. We deal in probability and I'd say there's a 60% probability. He's there.
Male speaker 4: I couldn't care. 60%.
Male speaker 5: I'm an 80%. There OPSEC is what convinces me.
Male speaker 6: You guys ever agree on anything?
Male speaker 7: Well, I agree with Mike, we're basing this mostly on DTE reporting, and I spent a bunch of time in those rooms.
Male speaker 2: I'd say it's a soft 60 sir. I'm virtually certain there's some high value target there. I'm just not sure it's Bin Laden.
Male speaker: Well, this is a little bit of a cluster expletive, isn't it.
Male speaker 2: I'd like to know what Maya thinks.
Male speaker 3: We're all incorporating her assessment DARS.
Maya: A hundred percent he is there. Okay. Fine. 95%. Cause I know certainty freaks you guys out, but it's 100.
Rick Howard: It's clear that as humans in our everyday lives, we don't really understand probabilities. Even if we do claim to understand them, they aren't satisfying. We prefer a yes or no answer. Will the company have a material breach this year telling the CEO yes or no is much more palatable to her than saying there's a 15% chance.
Rick Howard: What does she do with a 15% chance? Anyway? That answer is harder to deal with demands, an effort to parse and requires thinking strategy and flexibility. A yes, no answer. On the other hand is nothing more than an if than else clause like in a programming language. If we're gonna get breached this year, then spend resources to mitigate the damage else. Spend that money on making the product better, easy.
Rick Howard: Unfortunately, no matter how much we desire to live in a fantasy world, full of binary answers, yes. Or nos, the real world doesn't work that way and Neil Stevenson signs fiction novel seven Eves his Neil Degrass Tyson character. Dr. Boy explains how he calculates rock directories through a debris field quote, it's a statistical problem on about day one, it stopped being a Newtonian mechanics problem and turned into statistics. It has been statistics ever since end quote, exactly. Calculating cyber risk has never been Newtonian either. It's always been stacasic, no matter how much we desire to simplify the calculation into easy to read heat maps, we just didn't treat it that way and by the way, heat maps are just bad science. There are reams of scientific papers that make the case. So don't use them.
Rick Howard: There's a chart in the accompanying essay for this show that lists some of those papers you can find a link to it in the show notes, it might be more useful to reframe how we think about probabilities. If you're like me, your own statistics experience came from guessing what color marble will fall out of an urn in that probability in stats 101 course we all had to take in college and, yes, that's a great introduction to the concept, but that coursework only represents a small sliver of what probabilities really are a more useful and broader description in the cybersecurity context comes from Dr. Ron Howard, the father of decision analysis theory. No relation his entire field of study is based on the idea that probabilities represent uncertainty when making a decision, not the number of marbles in our earned collection, probability is not necessarily found in the data,
Rick Howard: Meaning, you don't have to count all the things in order to make an uncertainty forecast using probability. He says that quote, only a person can assign a probability taking into account any data or other knowledge available in quote, counting marbles, tumbling out of earned is one way to take accounted data. But Howard's great insight is that quote, a probability reflects a person's knowledge or equivalently ignorance about some uncertain distinction. He says, don't think of probability or uncertainties as the lack of knowledge. Think of them instead as a very detailed description of exactly what you know, in quote,
Rick Howard: Tetlock interviewed the real Leon Pineta about that internal CIA meeting and the subsequent meeting Pineta had with president Obama about the decision to spend special forces into Pakistan, to get Osama bin Laden. When the president went around the room with his staff, he also got a range of probabilities. His conclusion though, after reviewing those recommendations, was that his staff didn't know for sure. Therefore, it was simply a 50, 50 chance a toss up on whether or not Osama bin Laden was in the bunker, which is the wrong conclusion. By the way, it was probably much stronger.
Rick Howard: He ultimately made the right call, but he could just as easily aired on the side of caution. Tetlock also describes criticism of his superforecasting approach from his colleague, NA Caleb, the author of the black Swan, the impact of the highly probable published in 2007. Caleb says that forecasting is impossible because history is controlled by quote the tyranny of the singular, the accidental, the unseen, and the unpredicted in. According to New York times, journalist Greg Easterbrook, Taylor argues that quote experts are charlatans who believe in bell curves in which most distribution is toward the center, ordinary and knowable
Rick Howard: Far more powerful. Caleb argues are the wild outcomes of fractal geometry in which anything can happen overnight. In quote, caleb says that quote, what matters can't be forecast and what can be forecast doesn't matter, believing otherwise LUS us into a false sense of security in acknowledging the. Tetlock says that quote, the black Swan is therefore a brilliant metaphor for an event so far outside experience. We can't even imagine it until it happens in case in point, if we do some first order back of the envelope calculations, some Firmi estimates, we know that in 2021, the press reported on some 5,000 successful cyber attacks to us companies.
Rick Howard: We also know that there are approximately 6 million commercial companies in the country doing the outside in forecast. There was a 5,000 over 6 million chance of a us company getting breached in 2021, approximately 0.0008. That's a really small number. I'm gonna refine that forecast later, but for now, just go with me on it. By definition though, the experience of those 5,000 companies were black Swan events, significant, impactful events on something that was not very likely to happen at all.
Rick Howard: Tetlock's response to tib is that there are probably a set of estimate problems that are too hard to forecast, but he says that they are largely due to the fact that the forecasting horizon is too long. For example, it's tough to forecast who will win the us presidential election in 20 28, 6 years from the time of this writing. But you could do well with the us congressional elections in 2022, in three months. That said Caleb solution to black Swan events is to not attempt to prevent them, but to try to survive them.
Rick Howard: He says, resilience is the key. For example, instead of trying to prevent a giant media from hitting the earth, the question is how would you survive one in the cybersecurity context instead of preventing Panda bear from breaching your organization, what would you do to ensure that your organization continues to deliver its service during and after the attack? And that sounds an awful lot, like our cyber security first principle strategy resilience.
Rick Howard: I've been trying to get my hands around how to do risk assessment with more precision for over five years now, I've read the books, written book reviews for the cannon project interviewed many of the associated authors published a couple of papers and even presented those papers in consecutive years at the same security conference, one with Richardson, an author of one of the books.
Rick Howard: My initial thought when I started all of this, was that the main reason calculating risk was so hard for the InfoSec community. Was that it involved some high order math, a skill that was beyond most senior security practitioners. I became convinced though that in order to have enough precision to convince senior leadership, that my risk calculation was valid.
Rick Howard: I was gonna have to demonstrate my prowess with things like Monte Carlo simulations and by Asian, a. And then I was gonna have to explain what Monte Carlo simulations and by Asian algorithms were to these same senior leaders who are having a hard enough time understanding why our annual firewall subscription was so expensive. This seems like a bridge too far. So after five years of looking into how to do that, I've become a fan of firming. And Mosteller
Rick Howard: According to NA guest Maudi of the right attitude's website firmly believed that the ability to guesstimate was an essential skill for physicists in quote, I would say that the skill applies to any decision maker, but especially decision makers in the tech and security worlds, where the scales of the encounter problems are so enormous. Getting a precise estimate is hard and time consuming, but getting an estimate that's in the right ballpark in terms of order of magnitude is relatively easier and will probably be sufficient for most decisions. And even if it's not, you can always decide to do the more precise estimate later case in point here at the cyber wire, we did an inside out evaluation of our internal first principles, cybersecurity posture.
Rick Howard: In 2022, we evaluated our defenses in terms of zero trust, intrusion, kill chain prevention, resilience, automation, and compliance. Once complete, we briefed the boss on our findings and gave him our estimated probability of material impact due to some cyber event in the next. I then asked him for permission to do a deeper dive on the issue in order to get a more precise answer. His answer to me was spot on. He looked at the level of effort. This deep dive was gonna take, not only for the internal security team, but the entire company. And especially for him, frankly, it was gonna be high. And then he asked this question, what do you think the difference is gonna be between the initial inside out estimate and the deeper dive I had to admit, I didn't think the deeper dive estimate was gonna be that far away from the inside out estimate.
Rick Howard: Maybe a couple of percentage points up or down. He then said that if that was the case, he didn't need the deeper dive in order to make decisions about any future resource investment of the cyber wire's defensive posture. The initial estimate was good enough, quite so. So in the next couple of episodes, I'm gonna cover how to do an outside-in estimate for the cybersecurity community and discuss how to adjust it for your specific situation.
Rick Howard: In other words, we're gonna start with the general outside in estimate and adjust it based on the size of your organization, small, medium, and fortune 500 and type of organization, government, academic and commercial. I will then discuss how to get an inside out estimate based on how well your organization deploys our first principle strategy. So stay tuned in the meantime, check out Dr. Tetlock lock's superforecasting book. If you haven't already, I think it will be an eyeopener for you.
Rick Howard: And that's a wrap as always, if you agree or disagree with anything I have said, hit me up on LinkedIn or Twitter, and we can continue the conversation there. Or if you prefer email drop a line to firstname.lastname@example.org. That's CSOP, the at sign, the CyberWire, all one word .com. And if you have any questions, you would like us to answer here at CSO Perspectives, send a note to the same email address, and we will try to address them in the show. Last week I got a note from listener, Joe Nath. He's a senior engineer for DevOps infrastructure at NSF International, hey Joe. He suggested a future CSO prospective episode on how practitioners like him can become CSOs like me. I thought that was a great idea for our show and we put it into the rotation for December. Thanks for the suggestion, Joe. For next week's show though. As I said, we're gonna do some back-of-the-envelope cybersecurity Fermi estimations. I can hardly wait.
Rick Howard: One special note, this is the hundredth episode of the CSO perspectives podcast and I can't believe that we've reached that milestone. Where did the time go? But I have to say there are a lot of people here at the CyberWire busy behind the scenes that make this thing go and we don't usually take the time to give everybody credit. But for this special episode, I thought we would make an exception. For audio engineering, we have Elliott Peltzman and Tre Hester. On the biz side we have Bennett Moe, Jennifer Eiben, Brandon Karpf, Eliana White, Gina Johnson, Liz Irvin and Nick Veliky. On the IT and security side, Chris Russell and Puru Prakash. For editors, we have John Petrik, Tim Nodar, Rachel Gelfand, Ladzer Blumenfeld and Katie Aulenbacher for hosting and contributors, we have our own Dave Bitner, Joe Carrigan, Carole Theriault, and Ben Yelin and let's not forget about the volunteers, those senior security executives that come to the CyberWire hash table to share their wisdom. There are too many to list here, but you can see them all on the CyberWire website and last but not least, let's not forget about el hefe, the CyberWire CEO and executive producer, Peter Kilpe. Thank you all for helping to put this show together. I can't wait to see what we're gonna do with the next 100 episodes.