Weathering the internet storm.

Dave Bittner: Hello, everyone, and welcome to the CyberWire's Research Saturday. I'm Dave, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. [ Music ] It is my pleasure to welcome to the Research Saturday podcast Johannes Ullrich. He is the dean of research at the SANS Technology Institute, and also the host of the daily SANS Internet Storm Center podcast. Let me get that right. It is the ISC (Storm Center) podcast. And Johannes, you and I joke about this because, for whatever reason, I always want to say "ICS," which is wrong.

Johannes Ullrich: Lots of people do that.

Dave Bittner: Right, right.

Johannes Ullrich: It's one of my pet peeves when people say that.

Dave Bittner: Yeah. I want to get it right for you. So welcome, it's great to have you here.

Johannes Ullrich: Yeah, thanks for having me.

Dave Bittner: So we're going to do something a little different than we typically do on Research Saturday today. We're going to dig into some of the history of SANS as a cyber research organization, and then also talk about some of the process that you and your colleagues use there. Where would you like to get started?

Johannes Ullrich: Well, in the beginning, I guess. One reason that I think it's nice that we talk about this is it actually started 25 years ago. And not too many things in information security are that old and have survived for that long. But it actually originally started in '99. And, you know, for the young kids listening, there was something called Y2K that actually sort of sparked it all, where SANS sort of said, hey, you know, we probably should get better at exchanging what we are seeing in our environments. And Y2K sort of gave the spark to it. But then people found it really helpful to have a place where you can report what you're seeing, where you can talk about some of the observation environment. And that sort of then evolved into what's now Internet Storm Center.

Dave Bittner: And what was it like in those early days? Are we talking about, you know, message boards? Are we talking about blog posts? What was it?

Johannes Ullrich: Well, actually, one reason we still call it a diary today what we are writing sort of each day is -- the term "blog" didn't exist back then. And it was sort of a message board. It was emails coming in. We had our handler on duty -- we sort of still use some of that language today -- who would receive all these messages and then sort of compile little digests that would then be posted in this diary format.

Dave Bittner: Was it bidirectional? I mean, could people get feedback?

Johannes Ullrich: Yeah, and actually, that's something that still works quite well today sometimes. And we do post something like, hey, we received email where someone reported something odd in their environment. And then others are some of chiming in and reporting why they may be seeing that or some of the background about that particular software. So that community aspect of it really was developed very early on.

Dave Bittner: Yeah. Well, let's walk through the evolution then. I mean, how have things changed over the years?

Johannes Ullrich: Yeah, so, and actually that '99 was a little bit before I started working with SANS and working with Storm Center. By myself, I sort of started setting up a little bit of similar system but more automated, where I basically, with a couple of friends, started collecting our firewall logs, analyzing them, creating some graphic representations of those logs, which started in 2000, so like a year after SANS started its system. And it came really handy, if your remember, like 2000/2001, when these early worms came out. We really had some great data to then reflect how these worms spread, how fast they spread, where they started. So these firewall logs back then were what we collected, and people liked it. We got a ton of people that were then willing to submit their logs to the system. The nice thing was, the original SANS system was a more manual process I described. People writing in and people analyzing it, then posting about it. That's a slower process. These automated systems allowed us to speed all of that up. And of course it then would start feeding each other. That's when I then started working with SANS and also then we officially named it the Internet Storm Center.

Dave Bittner: You know, my brain short-circuits a little bit when I think about 1999 being 25 years ago. I don't know how you feel about that.

Johannes Ullrich: Yeah.

Dave Bittner: But can you give us an idea of what the community was like back then? I mean, cybersecurity itself was different than it is today.

Johannes Ullrich: It was very different. Like, for example, one parameter we're tracking is what we call the "survival time." And that's how long it takes between sort of unsolicited packets being received by your system, an average home system. We'll call it an attack. And back then that time was about 15 minutes. After the initial worms started sort of in 2001-ish, that shrank down to about five minutes. Later, in particular ones, Ryan, some of these really aggressive scanning bots, started. We are now well below one minute, sort of between unsolicited packets hitting random IP address on the Internet.

Dave Bittner: Are there any specific I guess milestones along the way that stand out to you in terms of either the growth and evolution of the Storm Center, or also the growth and evolution of the Internet itself?

Johannes Ullrich: Yeah, I think evolution of the threats over the years is one of those things. Like I mentioned, initially, we started collecting firewall logs. And that was very interesting back then. Because then we had bots like Linda, if anybody remembers that. It sort of hit the IS on port 80. We had the Blasto worm, which hit port 135 back in the day. Over the years, that changed. These days, much of the attacks we're seeing are web application attacks, which basically hit your standard web ports, like 80, 443, 8000, and so on. So as the initial firewall logs we collected became less telling as to what the actual threat is, we had to adapt, and we adapted to sort of more complete honeypots to collect our data. So where we now set up honeypots that are collecting data from Sage server, from Telnet server, sort of like your Marai style attacks. You have honeypots that are emulating different web applications. So there's all the different web attacks. That really now, first of all, tells us more detail about these attacks, what they're all about, what they're after. But again, we have to sort of keep up with the attacks. These days, many of the interesting attacks, they first check if your system is actually vulnerable. And then sort of about five, 10 years ago, we started experimenting with what we call an "agile honeypot," where the honeypot is able to emulate different applications, different devices. So that's sort of attacks against IoT devices started. That sort of helped us then gain a little more insight into those attacks that we're seeing these days for, a lot of it I mentioned already, Mirai a couple times, are sort of attacking very specific sort of routers or devices. I always joke that, hey, we can turn our honeypots into toasters if that's what's being attacked today. [ Laughter ] [ Music ]

Dave Bittner: Was there much thought given in those early days about scalability? Like were people imagining that the Internet would be so interleaved into our lives the way it is?

Johannes Ullrich: I think some people were sort of imaging it. I certainly kind of believed in that, you know, that sort of got me stuck with it. But I think overall, I would say, you know, back in the early days, Internet was a much nicer place kind of, where people helped each other a little bit more. And that in some ways got me into security. Like, you know, one of the early instants that I sort of had to deal with in my personal system was setting up a Linux system. Which back then, again, we're talking like in the late '90s, had an open mail relay by default. And that's just how we rolled back then.

Dave Bittner: Right [laughing].

Johannes Ullrich: You set up mail servers just for everybody to send email with.

Dave Bittner: Right.

Johannes Ullrich: And of course that was then when spammers started coming up and started using those mail servers. So then I think the security community was also smaller in that sense. I think there was more trust then there is now, more collaboration. Obviously, collaboration, a lot depends on sort of, you know, people collaborate with other organizations collaborating with each other. That personal connection I think happened probably more back than that it does now.

Dave Bittner: Yeah. Can you speak to the transformation where it has kind of become corporatized these days? So you have the big players, there are still individuals who are known by name. And I would put you in that category. But so much is, you know, Mandiant says, or Microsoft says, or, you know, the big names, come out with their research.

Johannes Ullrich: Correct. And I think at the Internet Storm Center, we try to sort of still follow that old model. Like all of our honeypots are run by volunteers. You have some individuals at corporations sort of donate significant resources like IP address space and such to our honeypots and to the effort overall. Also, a lot of the analysis we do is sort of done by volunteers.

Dave Bittner: Well, let's fast forward to today. I mean, what does it look like nowadays? What sort of processes do you all have in place?

Johannes Ullrich: So these days, we heavily rely on our web application logs in some sense on some of the logs, but not as much we should these days. But all of these logs are being reported by these honeypots, which usually on Raspberry Pies, that's our preferred platform, we have virtual systems that people are using to send out these honeypots, some like in various cloud providers. They send all of these logs to our database. We add them to the database. But one of the unique things we offer is essentially real time these logs are being turned around via our website. Everybody can look at them, can see what's new, what's interesting. We do have actually now some interns that help us from our undergraduate program that also run honeypots, help us develop the software and test it and also alert us of some new attacks that they see.

Dave Bittner: Can you give us some examples of some of the more interesting items that you and your colleagues there have been researching lately?

Johannes Ullrich: Yeah, just today, earlier, I was working on Alation, Alation, Confluence. They patched a vulnerability last week. On Monday, I saw in our -- we have a report that you can also see it on our website, called the First Seen URL Report. It basically lists, hey, these are web application attacks that we saw today that we haven't seen before. And one of the URLs that sort of popped up there was related to the Alation attack. Then I was able to actually emulate that particular software in a subset of our honeypots, that's sort of where the agile part comes in. And then sort of collect more data about these attacks, what people are trying to do with those servers. And then again, sort of immediately turn it around, publish something about it, put up a quick summary about what we are seeing. But again, the data was already there for everybody else to see. So our diaries, as we call them, these blog posts, are really just summarizing the data that we have, that's at least part of what we're doing there.

Dave Bittner: What goes into being an effective researcher here? The folks that you work with, yourself included, what are those personality elements, the areas of curiosity that seem to work out?

Johannes Ullrich: I think curiosity is really it, kind of, and then being willing to experiment, being willing to be wrong sometimes. And that's of course I think something I think we have changed from the early days. But the social media environment these days can be a little bit unforgiving in that respect. But, you know, being wrong in a sense, hey, if you're wrong, someone else will tell you why you're wrong and what the real answer is. Also, being willing to listen to those people that tell you that you're wrong, I think that's important. It sometimes helps to have a little bit of memory of what happened before, surely not remembering all of the different attacks that I've seen over the years. But I see a lot of re-reporting of attacks too, that's a little bit annoying.

Dave Bittner: Oh, that's interesting, yeah. And sometimes I would imagine, you know, speaking to that memory component, you probably just get a funny feeling, like something is amiss here, but you can't quite put your finger on it.

Johannes Ullrich: That's correct. And also seeing like the what's different, what's new, that's really some of the important thing and the difficult part to figure out. Also, being willing to just plain experiment, being wrong. A boss once told me in a prior job that the important part is to make the right number of mistakes, kind of. If you don't make mistakes, you just aren't really brave enough to try something new, try something different, I think that's important as a researcher to make those mistakes, learn from it.

Dave Bittner: What are your recommendations for somebody who's coming up in the industry? You know, either a student or maybe somebody considering a career change, the types of things that they can do to prepare themselves if this sort of research is something they think they're going to be interested in?

Johannes Ullrich: This may be a little bit very specific advice, but setting up a honeypot. We had really great success with our undergraduate students who did it and then realized, hey, these are actual attacks I'm seeing here. Because when you're reading about it, even when you're studying about it in the classroom environment, maybe you're running some exercise around an attack, it's all sort of fairly sterile and artificial. If you actually see a simple worm kind of, you know, hitting your honeypot, exploiting some of these vulnerabilities they had talked about in class, I think that makes it much more real and brings it really home to people. And it's relatively easy from a technical point of view to sort of get started with that. Of course, I'm biased here, but I thought that I saw really a lot of people's eyes light up sort of the first time they really saw these attacks hitting their systems.

Dave Bittner: Do you find that folks can be kind of intimidated by that, you know, sort of playing with live fire, if you will?

Johannes Ullrich: Yeah, that certainly happens. And maybe that's also important for them to realize that how frequent these attacks are, also, how many of these attacks really don't matter. We had recently this famous statement from some bank executives and how they're being attacked like a billion times a day and such. And some security people made sort of fun of that statement. It's real. They're being attacked that many times. But most of these attacks don't matter, they don't cause any damage. And that's in particular if you're sort of starting out from a defensive side, from like a software developer or network administrator point of view, your goal is these five nines, or, you know, this high reliability, everything has to work. You somewhat have to switch mindsets when you're talking about attacks, where you're just saying that, hey, you know, for an attacker, it's perfectly fine if 99.9% of their attacks don't work. If the one attack works that breaks into the Fortune 500s and the research department and gets to all their secrets, it was a good attack.

Dave Bittner: It kind of reminds me of, you know, our own immune systems, where, you know, it most of the time is just running there, fending things off, and we don't think twice about it, it just takes care of its business on its own and we don't even notice. But then every now and then something gets through and, you know, you could get a cold or you get something more serious.

Johannes Ullrich: Yeah, and that's sort of the important task of the researcher, to find those new and different things, where you have to adjust your immune system, where you actually have to build these new capabilities to defend against this new attack. And the danger is of course from sort of, you know, someone is in the business like me for a while, to sort of get a little bit, you know, dull over time or you sort of stop caring really to some extent [laughing].

Dave Bittner: Right.

Johannes Ullrich: And balance that with the new person who is getting excited about every little attack that's coming in. And I've seen both work. Really that's why you sort of need that diversity also in your security teams, where you still have, you know, someone that's new to it, that still gets excited about some attacks. Because sometimes they find some interesting things because they do that research, they do actually dig in and see, hey, what is this attack doing?

Dave Bittner: Yeah. [ Music ] All right. Well, Johannes Ullrich is the dean of research at the SANS Technology Institute, and he is also the host of the ISC Storm Cast podcast. Johannes, thank you so much for joining us today.

Johannes Ullrich: Yeah, thank you. [ Music ]

Dave Bittner: The CyberWire Research Saturday podcast is a production of N2K Networks. N2K strategic workforce intelligence optimizes the value of your biggest investment, your people. We make you smarter about your team while making your team smarter. Learn more at N2K.com. This episode was produced by Liz Stokes. Our mixer is Elliot Peltzman. Our executive producers are Jennifer Eiben and Brandon Karpf. Our executive editor is Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next time. [ Music ]