Cyber CEOs Decoded 8.25.22
Ep 5 | 8.25.22

Dror Davidoff: Aqua Security Co-founder and CEO


Rick Howard: Hey, everybody. Rick here. Out of all the capabilities in the infosec community that have improved over the years, the one essential skill that hasn't moved forward is calculating risk - specifically, how do we convey risk to senior leadership and to the board? In my early network defender days, whenever somebody asked me to do a risk assessment, I would punt. I would roll out my qualitative heat map, a fancy name for a color-coded spreadsheet where all the risks are listed on the X axis, and my three levels of potential impact - high, medium and low - are plotted on the Y axis and call it a day. Along with many of my peers, I would tell myself that predicting cyber risk with any more precision was impossible, that there were too many variables, that cybersecurity was somehow different from all the other technical and scientific disciplines in the world, like physics and chemistry and orbital mechanics and space, and it couldn't be done.

Rick Howard: We were wrong, of course. The Cybersecurity Canon project is full of hall-of-fame and candidate books that talk about how to calculate cyber risk with a precision, books like "How to Measure Anything in Cybersecurity Risk" by Hubbard and Seiersen, "Measuring and Managing Information Risk: A FAIR Approach" by my friends Freund and Jones, "Security Metrics, A Beginner's Guide" by Caroline Wong, newly inducted this year into the canon hall of fame. And you can catch my interview with her on this podcast as part of the bonus material last season. And finally, "Security Metrics: Replacing Fear, Uncertainty, and Doubt" by Jaquith. These are all great primers regarding how to think about precision probability forecasting, and I highly recommend them. If this subject is new to you, they will all change your current view of the world.

Rick Howard: But my problem with all of them is that I kept waiting for the chapter at the end entitled, And Here's How to Do It, or better, Building That Risk Chart You Take to the Board. None had it or anything close. That part was always left as an exercise for the reader. I decided it was time for me to roll up my sleeves and figure out how to do it myself. So hold on to your butts.


Samuel L Jackson: (As Ray Arnold) Hold on to your butts. 

Rick Howard: My name is Rick Howard, and I am broadcasting from the CyberWire's Secret Sanctum Sanctorum Studios, located underwater somewhere along the Patapsco River near Baltimore Harbor, Md., in the good old U.S. of A. And you're listening to "CSO Perspectives," my podcast about the ideas, strategies and technologies that senior security executives wrestle with on a daily basis. 

Rick Howard: The book that changed my mind that calculating cyber risk with some precision was possible is called "Superforecasting: The Art and Science of Prediction" by Philip Tetlock and Dan Gardner, another Cybersecurity Canon project hall-of-fame candidate book. Dr. Tetlock is quite the character. He's one of those scream-and-shake-your-raised-fist-at-the-TV-because-they-have-no-idea-what-they're-talking-about people. He would watch news programs like CNN, Fox and MSNBC, where the host would roll out famous pundits to give their opinion on some topic because, once in their lives, they predicted something correctly. It didn't matter that all the predictions they've made since were wrong. The news programs would still bring them on as if they were Moses coming down from Mount Sinai to present the tablets as law. Dr. Tetlock thought that they should have to keep score. I always thought that when pundits came on, the viewer should see their batting average really across the chyron at the bottom of the screen. 


Vin Scully: And look who's coming out. 


Rick Howard: Rick Howard, the predictor from Paramus, the forecaster from Falls Church, the risk assessor from Rialto, has made three correct predictions out of 20 tries this year. His batting average is 15. Maybe we shouldn't listen too closely what he has to say. 


Rick Howard: My apologies to the late, great Vin Scully, the voice of the LA Dodgers for over 50 years, who just recently passed. But we couldn't help but appropriate his famous call of Kirk Gibson's walk-off home run in the first game of the 1988 World Series between the Los Angeles Dodgers and the Oakland A's. If you've never watched it, do yourself a favor. Take 10 minutes and experience one of the greatest sports storytellers of all time. There's a link to it in the show notes. But I digress. 

Rick Howard: And then Dr. Tetlock decided to test his idea, working with IARPA, the Intelligence Advanced Research Projects Agency, he devised a test using three groups - the intelligence community, the academic community and a group I call the geezers on the go. Now, the geezers on the go were not all old people; they were just regular people with time on their hands who liked to solve puzzles. According to The Washington Post, Tetlock had them forecast answers to over 500 really hard questions, like will the Syrian president still be in power in six months? And will there be a military exchange in the South China Sea in the next year? And will the number of terrorist attacks sponsored by Iran increase within one year of the removal of sanctions? Out of the three communities, the geezers on the go outperformed the control group by 60%. They beat the academic teams from 30% to 70% depending on the school - MIT and the University of Michigan were two - and outperformed the intelligence groups who had access to classified information. But Tetlock also discovered a subset of the geezers on the go - the superforecasters. By the end of the four-year tournament, these superforecasters had outperformed the geezers on the go by another 60% and could also see further out than the control group. Quote, "superforecasters looking out 300 days were more accurate than regular forecasters looking at 100 days," end quote. 

Rick Howard: And these superforecasters don't have extreme mutant abilities either. They are intelligent for sure, but not overly so. This isn't a collection of Professor X's from the "X-Men" comic book. They aren't all card-carrying members of Mensa, and they're not math nerds either. Most of them only perform rudimentary math calculations when they make their forecasts. But by following a few guidelines, they can outperform random Kentucky windage guesses by normal people like me - like, for example, No. 1 - forecast in terms of quantitative probabilities, not qualitative high, medium and lows. In other words, get rid of the heat map. Embrace the idea that probabilities are nothing more than a measure of uncertainty. But also, understand that just because the probability that something will happen is 70% doesn't mean it's a lock - see Secretary Clinton in the 2016 U.S. presidential campaign. No. 2 - practice. Do a lot of forecasting and keep score using something called the Brier score invented by Glenn Brier in 1950. The score is on two axes - calibration and resolution. Calibration is how close to the line your forecast is - are you overconfident or under? Resolution is when you predict something is going to happen, it does. No. 3 - embrace Fermi estimates - outside-in first and then inside-out forecasts. Outside-in is looking at the general case before you look at the specific situation. In terms of cybersecurity, that means the outside-in considers the probability that any organization would get hit by, say, a ransomware attack. Inside-out considers the probability that ransomware criminals will cause a material impact to your organization. 

Rick Howard: See the difference? Both have merit, but Tetlock says to start with the outside-in forecast and then adjust up or down from there with the inside-out forecast. For example, if your outside-in forecast says that there is a 20% chance of material impact due to a ransomware attack this year for all U.S. companies, that's the baseline. Then, when you do the inside-out assessment by looking at how well your organization is deployed against our first principal strategies, you might move the forecast up or down depending. So how do you make those outside-in assessments? Well, the Italian American physicist Enrico Fermi was a central figure in the invention of the atomic bomb, and he was renowned for his back-of-the-envelope estimates. With little or no information at his disposal, he would often calculate a number that subsequent measurement revealed to be impressively accurate. He would famously ask his students things like estimate the number of square inches of pizza consumed by all the students at the University of Maryland during one semester. And he forbade the students from looking up any information. He encouraged them to make back-of-the-envelope assumptions first. He understood that by breaking down the big, intractable question - like how many inches of pizza consumed - into a series of much simpler, answerable questions - like how many students, how many pizza joints, how many inches in a slice, etc. - we can better separate the knowable and the unknowable. The surprise is how often good probability estimates arise from a remarkably crude series of assumptions and guesstimates. More on this in a bit. 

Rick Howard: Frederick Mosteller, a groundbreaking eminent statistician in the 1950s through the 1970s, said that, quote, "it is the experience of statisticians that when fairly crude measurements are refined, the change, more often than not, turns out to be small. Statisticians would wholeheartedly say, make better measurements. But they would often give a low probability to the prospect that finer measures would lead to a different policy," end quote. No. 4, check your assumptions. Adjust, tweak, abandon, seek new ones and adjust your forecast from there. No. 5 - dragon eyes (ph). Consume evidence from multiple sources. Construct a unified vision of it. Describe your judgment about it as clearly and concisely as you can being as granular as you can. And finally, No. 6 - forecast at a 90% confidence level. As you adjust your forecast, remember that you want to be 90% confident about it. If you're not, then you need to adjust up or down until you are. The point to all this is that it's possible to forecast the probability of some future and mind-numbingly complex event with enough precision to make decisions with. If the geezers on the go can accurately predict the future of the Syrian president, surely, a bunch of no-math CISOs like me can forecast the probability of a material impact due to a cyber event for their organizations. That's cybersecurity risk forecasting. 

Rick Howard: Tetlock spends time talking about how the U.S. government hasn't done this kind of thinking in the past. You and I would call them massive intelligence failures, like WMD in Iraq - 20 years of war on a slam-dunk CIA assertion that these weapons existed in Iraq when they didn't. Like the Vietnam War - 10 years of war on the widely held belief that if South Vietnam fell, the entire world would fall to communism like dominos. Leaders didn't just think there was a chance this would happen. They thought it was a sure thing. Like the Bay of Pigs - President Kennedy's political disaster when the planners didn't consider the probability of success when the plan changed at the last minute. And finally, is Osama bin Laden in the bunker? 

Rick Howard: Tetlock describes a scene in one of my favorite movies, 2012's "Zero Dark Thirty," starring Jessica Chastain. The CIA director, Leon Panetta, played by the late, great James Gandolfini, is in a conference room asking his staff for a recommendation on whether or not Osama bin Laden is in the bunker. He's looking for a yes-or-no answer. One of his guys says that he fronted the bad recommendation about WMD in Iraq. And because of that failure, they don't deal in certainties anymore. They deal in probabilities, which is the right answer, by the way, just not a very satisfying one. They go around the room and get a range of probabilities from 60% to 80%. Chastain breaks into the conversation and says that the probability is a hundred percent. OK, fine, 95%, she says, because I know certainty freaks you guys out. But it's a hundred percent, which is the wrong answer, by the way. The probability was never a hundred percent, no matter how sure she was with her evidence. One note of caution, this next clip has some strong language. If you have sensitive ears about, it's best to pause for a bit. 


James Gandolfini: (As CIA Director) I'm about to go look the president in the eye. And what I'd like to know, no fucking bullshit, is where everyone stands on this thing. Now, very simply, is he there or is he not fucking there? 

Jeff Mash: (As Deputy Director of CIA) We all come at this through the filter of our own past experiences. Now, I remember Iraq WMD very clearly. I fronted that, and I can tell you the case for that was much stronger than this case. 

James Gandolfini: (As CIA Director) A fucking yes or a no. 

Jeff Mash: (As Deputy Director of CIA) We don't deal in certainty. We deal in probability. And I'd say there's a 60% probability he's there. 

Frederic Lehne: (As The Wolf) I concur - 60%. 

Mark Strong: (As George) I'm at 80%. Their OPSEC is what convinces me. 

James Gandolfini: (As CIA Director) You guys ever agree on anything? 

Jason Clarke: (As Dan) Well, I agree with Mike. Probation is mostly on detainee reporting, and I spent a bunch of time in those rooms. I'd say it's a soft 60, sir. I'm virtually certain there's some high-value target there. I'm just not sure it's bin Laden. 

James Gandolfini: (As CIA Director) Well, this is a little bit of a clusterfuck, isn't it? 

John Barrowman: (As Jeremy) I'd like to know what Maya thinks. 

Jeff Mash: (As Deputy Director of CIA) We're all incorporating her assessment into ours. 

Jessica Chastain: (As Maya) A hundred percent he's there. OK, fine, 95% 'cause I know certainty freaks you guys out. But it's a hundred. 

Rick Howard: It's clear that as humans in our everyday lives, we don't really understand probabilities. And even if we do claim to understand them, they aren't satisfying. We prefer a yes-or-no answer. Will the company have a material breach this year? Telling the CEO yes or no is much more palatable to her than saying there's a 15% chance. What does she do with a 15% chance anyway? That answer is harder to deal with, demands an effort to parse and requires thinking, strategy and flexibility. A yes-no answer, on the other hand, is nothing more than an if-then-else clause, like in a programming language. If we're going to get breached this year, then spend resources to mitigate the damage, else spend that money on making the product better. Easy. 


Unidentified Person: That was easy. 

Rick Howard: Unfortunately, no matter how much we desire to live in a fantasy world full of binary answers, yes or noes, the real world doesn't work that way. In Neal Stephenson's science-fiction novel "Seveneves," his Neil deGrasse Tyson character, Doc DuBois, explains how he calculates rocket trajectories through a debris field. Quote, "It's a statistical problem. On about day one, it stopped being a Newtonian mechanics problem and turned into statistics. It has been statistics ever since," end quote. Exactly. Calculating cyber risk has never been Newtonian either. It's always been stochastic, no matter how much we desire to simplify the calculation into easy-to-read heat maps. We just didn't treat it that way. And by the way, heat maps are just bad science. There are reams of scientific papers that make the case, so don't use them. There's a chart in the accompanying essay for this show that lists some of those papers. You can find a link to it in the show notes. 

Rick Howard: It might be more useful to reframe how we think about probabilities. If you're like me, your own statistics experience came from guessing what color marble will fall out of an urn in that probability and stats 101 course we all had to take in college. And, yes, that's a great introduction to the concept. But that coursework only represents a small sliver of what probabilities really are. A more useful and broader description in the cybersecurity context comes from Dr. Ron Howard, the father of decision analysis theory - no relation. His entire field of study is based on the idea that probabilities represent uncertainty when making a decision, not the number of marbles in our urn collection. Probability is not necessarily found in the data, meaning you don't have to count all the things in order to make an uncertainty forecast using probability. He says that, quote, "only a person can assign a probability taking into account any data or other knowledge available," end quote. Counting marbles tumbling out of urns is one way to take account of data. But Howard's great insight is that, quote, "a probability reflects a person's knowledge or equivalently ignorance about some uncertain distinction." He says, "don't think of probability or uncertainties as the lack of knowledge. Think of them instead as a very detailed description of exactly what you know," end quote. 

Rick Howard: Tetlock interviewed the real Leon Panetta about that internal CIA meeting and the subsequent meeting Panetta had with President Obama about the decision to spend special forces into Pakistan to get Osama bin Laden. When the president went around the room with his staff, he also got a range of probabilities. His conclusion, though, after reviewing those recommendations, was that his staff didn't know for sure. Therefore, it was simply a 50-50 chance, a toss-up on whether or not Osama bin Laden was in the bunker - which is the wrong conclusion, by the way. It was probably much stronger. He ultimately made the right call, but he could just as easily erred on the side of caution. 

Rick Howard: Tetlock also describes criticism of his superforecasting approach from his colleague, Nassim Taleb, the author of "The Black Swan: The Impact of the Highly Improbable," published in 2007. Taleb says that forecasting is impossible because history is controlled by, quote, "the tyranny of the singular, the accidental, the unseen and the unpredicted," end quote. According to New York Times journalist Gregg Easterbrook, Taleb argues that, quote, "experts are charlatans who believe in bell curves in which most distribution is toward the center, ordinary and knowable. "Far more powerful," Taleb argues, "are the wild outcomes of fractal geometry in which anything can happen overnight," end quote. Taleb says that, quote, "what matters can't be forecast, and what can be forecast doesn't matter. Believing otherwise lulls us into a false sense of security," end quote. 

Rick Howard: Acknowledging the argument, Tetlock says that, quote, "The Black Swan" is therefore a brilliant metaphor for an event so far outside experience, we can't even imagine it until it happens," end quote. Case in point, if we do some first order back-of-the-envelope calculations, some Fermi estimates, we know that in 2021, the press reported on some 5,000 successful cyberattacks to U.S. companies. We also know that there are approximately 6 million commercial companies in the country. Doing the outside-in forecasts, there was a 5,000 over 6 million chance of a U.S. company getting breached in 2021, approximately .0008. That's a really small number. I'm going to refine that forecast later, but for now, just go with me on it. By definition, though, the experience of those 5,000 companies were black swan events - significant, impactful events on something that was not very likely to happen at all. 

Rick Howard: Tetlock's response to Taleb is that there are probably a set of estimate problems that are too hard to forecast, but he says that they are largely due to the fact that the forecasting horizon is too long. For example, it's tough to forecast who will win the U.S. presidential election in 2028, six years from the time of this writing. But you could do well with the U.S. congressional elections in 2022, in three months. That said, Taleb's solution to black swan events is to not attempt to prevent them, but to try to survive them. He says resilience is the key. For example, instead of trying to prevent a giant meteor from hitting the earth, the question is, how would you survive one? In the cybersecurity context, instead of preventing Panda Bear from breaching your organization, what would you do to ensure that your organization continues to deliver its service during and after the attack? And that sounds an awful lot like our cybersecurity first principle strategy - resilience. 

Rick Howard: I've been trying to get my hands around how to do risk assessment with more precision for over five years now. I've read the books, written book reviews for the Canon Project, interviewed many of the associated authors, published a couple of papers and even presented those papers in consecutive years at the same security conference, one with Richard Seiersen, an author of one of the books. My initial thought when I started all of this was that the main reason calculating risk was so hard for the infosec community was that it involves some high order math, a skill that was beyond most senior security practitioners. I became convinced, though, that in order to have enough precision to convince senior leadership that my risk calculation was valid, I was going to have to demonstrate my prowess with things like Monte Carlo simulations and Bayesian algorithms. And then I was going to have to explain 

Rick Howard: what Monte Carlo simulations and Bayesian algorithms were to these same senior leaders who are having a hard enough time understanding why our annual firewall subscription was so expensive. This seems like a bridge too far. 

Rick Howard: So after five years of looking into how to do that, I've become a fan of Fermi and Mosteller. According to Nagesh Belludi of the Right Attitudes website, quote, "Fermi believed that the ability to guesstimate was an essential skill for physicists," end quote. I would say that the skill applies to any decision-maker, but especially decision-makers in the tech and security world, where the scales of the encountered problems are so enormous. Getting a precise estimate is hard and time consuming but getting an estimate that's in the right ballpark in terms of order of magnitude is relatively easier and will probably be sufficient for most decisions. And even if it's not, you can always decide to do the more precise estimate later. 

Rick Howard: Case in point, here at the CyberWire, we did an inside-out evaluation of our internal first principle cybersecurity posture in 2022. We evaluated our defenses in terms of zero trust, intrusion, kill chain, prevention, resilience, automation and compliance. Once complete, we briefed the boss on our findings and gave him our estimated probability of material impact due to some cyber event in the next year. I then asked him for permission to do a deeper dive on the issue in order to get a more precise answer. His answer to me was spot on. He looked at the level of effort this deep dive was going to take not only for the internal security team, but the entire company, and especially for him. Frankly, it was going to be high. And then he asked this question. What do you think the difference is going to be between the initial inside-out estimate and the deeper dive? I had to admit I didn't think the deeper dive, estimate was going to be that far away from the inside-out estimate. Maybe a couple of percentage points up or down. He then said that if that was the case, he didn't need the deeper dive in order to make decisions about any future resource investment of the CyberWire's defensive posture. The initial estimate was good enough. Quite so. 

Rick Howard: So in the next couple of episodes, I'm going to cover how to do an outside-in estimate for the cybersecurity community and discuss how to adjust it for your specific situation. In other words, we're going to start with the general outside-in estimate and adjust it based on the size of your organization - small, medium and Fortune 500 - and type of organization - government, academic and commercial. I will then discuss how to get an inside-out estimate based on how well your organization deploys our first principal strategy. So stay tuned. In the meantime, check out Dr. Tetlock's "Superforecasting" book if you haven't already. I think it will be an eye-opener for you. 

Rick Howard: And that's a wrap. As always, if you agree or disagree with anything I have said, hit me up on LinkedIn or Twitter and we can continue the conversation there. Or if you prefer email, drop a line to That's And if you have any questions you would like us to answer here at "CSO Perspectives," send a note to the same email address and we will try to address them in the show. Last week, I got a note from listener Joe Knaff (ph). He's a senior engineer for DevOps infrastructure at NSF International. Hey, Joe. He suggested a future "CSO Perspectives" episode on how practitioners like him can become CSOs like me. I thought that was a great idea for a show and we put it into the rotation for December. Thanks for the suggestion, Joe. For next week's show, though, as I said, we're going to do some back of the envelope cybersecurity Fermi estimations. I can hardly wait. 

Rick Howard: One special note. This is the hundredth episode of the "CSO Perspectives" podcast, and I can't believe that we've reached that milestone. Where did the time go? But I have to say, there are a lot of people here at the CyberWire busy behind the scenes that make this thing go, and we don't usually take the time to give everybody credit. But for this special episode, I thought we would make an exception. For audio engineering, we have Elliott Peltzman and Tre Hester. On the business side, we have Bennett Moe, Jennifer Eiben, Brendon Karpf, Eliana White, Gina Johnson, Liz Irvin and Nick Veliky. On the IT and security side, Chris Russell and Puru Prakash. For editors, we have John Petrik, Tim Nodar, Rachel Gelfand, Ladzer Blumenfeld and Katie Aulenbacher. For hosting and contributors, we have our own Dave Bittner, Joe Carrigan, Carole Theriault and Ben Yelin. And let's not forget about the volunteers, those senior security executives that come to the CyberWire Hash Table to share their wisdom. There are too many to list here, but you can see them all on the CyberWire website. And last but not least, let's not forget about el hefe, the CyberWire's CEO and executive producer, Peter Kilpe. Thank you all for helping to put this show together. I can't wait to see what we're going to do in the next 100 episodes.