That first CVE was a fun find, for sure.
Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.
Larry Cashdollar: One of the PR guys, Tim, he thought it would be interesting if I spoke about my first CVE since I had been at Akamai for twenty years. It had more of a story behind it than I think Tim was expecting so it was actually caused some mayhem...
Dave Bittner: (Laughs)
Larry Cashdollar: ...So I wrote it up, and here I am.
Dave Bittner: That's Larry Cashdollar. He's a Senior Security Response Engineer at Akamai Technologies. Today, we're discussing his recent blog post, "Music to Hack to: My First CVE and 20 Years of Vulnerability Research."
Dave Bittner: All right, well, let's dig into it. Can you sort of set the scene for us? What era are we talking about, and where were you in your career at the time?
Larry Cashdollar: So, I was studying computer science in the University of Southern Maine back in 1994-95-era. And I was working for this small company in southern Maine – in Portland, Maine, actually – and I was there as an Internet analyst. And I ended up getting a position at Computer Sciences Corporation. Now, for folks who don't know, Computer Sciences Corporation is a – it's a large consulting company where they hire folks to take care – or CSC – to take care of their infrastructure. So at the time, Bath Iron Works was contracted – or had contracted CSC to handle their IT infrastructure. And I was looking to get more of a broad view of the Internet world and the computing world, and CSC also paid more.
Larry Cashdollar: So I got hired at CSC, and the playground I had – is was what I called it – my team managed over three-thousand Unix systems. There was anything from SGI machines to IBM AIX machines to HP-UX machines. So I had this enormous pool of Unix systems to play with. And the first day on the job, my manager was giving me a tour of various buildings around Bath Iron Works, because Bath Iron Works has a large campus. And he took me into one of these rooms where they had – I can't remember if it was a dozen – SGI Indi boxes. And they were all sitting there humming along and they were working on a new submarine program back then. And this was the room that they were going to start doing this development work with 3D images.
Larry Cashdollar: My manager sat down at one of the consoles and he says, you know, someday, when you prove your worth, I'll give you a login on this. And me coming from a security background, I knew that IRIX Systems had LP account and that LP account had no password. It had a valid shell, but you didn't need a password to log in so you could simply walk up to one of these machines, type "LP," and hit enter. So what I did was I strolled up to one of the machines, and I typed "LP" and I hit enter and logged in, and I looked at it and I said, thanks, I don't need one. And he looked at me and his mouth dropped, and he's like, how did you do that? And I said, by default, IRIX 6.x machines don't have a password set for LP. You know, none of these machines have passwords set on that account, so you can literally log in anyone as the LP user and get a full desktop. So he looked at me and he said, would you do security for us? And I said, I was hoping you'd say that.
Larry Cashdollar: So from then on, I was sort of like the penetration tester/hacker kid that was doing security testing. And one of the – the story goes is that we had this SGI Onyx – for folks who don't know, this thing's about the size of a refrigerator...
Dave Bittner: Yeah.
Larry Cashdollar: ...Back then, it would cost between anywhere from two-hundred-and-fifty-thousand dollars to half a million dollars, depending on how it was configured. This one that we had had its own private room that it sat in, with raised floors and the air conditioning, and it had a punch key code on the wall. You had to punch in a number that only a handful of sysadmins knew. None of my team knew because we were only sysadmin level one, and only the sysadmin level threes had access to it. So if you needed something from that room, you had to ask them to go get it for you. They wouldn't give you the code. You had to ask someone, hey, can you go in this room and get me a new hard disk or whatever you needed, and they'd go in and get it for you.
Dave Bittner: Yeah.
Larry Cashdollar: And the sysadmin who – Dave, I think his name was – who had access to this room, would taunt the rest of us, you know, the guys in my group, like, hey, you know, someday maybe you guys will get root on the Onyx machine, or you'll get an account on the machine...
Dave Bittner: (Laughs)
Larry Cashdollar: ...And I'll give it to you once you're maybe at my level of sysadmin. So this sort of bothered me...
Dave Bittner: Just – let me interject here, Larry, because for folks who aren't from this era, who may be unfamiliar with these Silicon Graphics Onyx machines and the types of machines that SGI was putting out at the time, you know, if you were working in graphics or 3D animation or really any of these things that you could apply one of these machines to, the type of processing power that you would need for one of these things, these machines were objects of desire. Not only were they extremely powerful, but they were beautiful machines as well. If you go look up a picture of one today, you'll probably say, oh, it looks like a computer from the late nineties or the early 2000s. So they look a little dated by today's standards, but take my word for it – at the time, these were extraordinarily sexy devices.
Larry Cashdollar: Yeah, they were the creme de la creme back then.
Dave Bittner: Yeah.
Larry Cashdollar: So, to have access to the Onyx/2 was like the Holy Grail for my team and mostly me – I don't think other guys really cared, but I just didn't like being taunted. So I thought to myself, you know, I already know that I can log into that system with the LP account and get a shell on it because I know they didn't secure it. But I want to get root on this thing. So how can I get root on the Onyx? So I'm like, well, I have another SGI system that I have access to that's near me in my lab, in my office that I had. I can log into that system and just look around the operating system.
Larry Cashdollar: And back in the nineties, you know, it was you know, when you attacked the system, you were trying to get hacked into the server and get root. You know, you were trying to get root so you can wipe your logs and your access showing that you had logged in and hide your tracks, and that was a big thing to get root on a system, for black hats.
Larry Cashdollar: And so I knew what I should do is look for setuid binaries, which is binaries on systems that when you execute them as a normal user, they execute with root privileges. And I figured if I could find a binary with setuid root permissions that I could somehow abuse to get it to do something that it wasn't meant to do, like write to a system file or execute shell as root, I can get a root prompt on the machine and I would be all happy and have hacked into the system.
Larry Cashdollar: So I saw this looking around /usr/sbin/, where most of the setuid binaries lived back then. I saw this file called "Midikeys." Now, midikeys had the setuid bit set on it, and I had never really heard of it before and was curious what this binary was that it needed root access. So I ran it, and on my screen pops up this little keyboard, looks like a piano, and when you click the keys, it plays these little tunes, these little MIDI tunes, and you can save them. You could compose a little MIDI song and save it to disk. And I thought to myself, well, can I edit files with this? Can I save to a file as root and possibly edit the password file? So I opened up the password file, and I put a zero in for my user ID, for "Larry," and then saved it and then logged in again. And I was like, I found a way to get root on the Onyx.
Larry Cashdollar: So, I quickly logged into the Onyx's LP, forward an x window back to my machine and forward back a – execute the Midikeys, bring it up, open up /etc/passwd, create a "Larry" account, make my user id "0" and then save it, log back into the Onyx, and I have root. And I'm like, OK, I've hacked into the Onyx, I have a root prompt now. I should set the password on the Onyx so it's not sitting there with a "Larry" login with no password, super insecure. So I type "password" to change the password, and I'm like, oh no, my user id is 0. I'm changing the password for root, not for Larry. So I go to back out, and I hit CTRL-D, CTRL-D, thinking that I would back out of the password program. And instead, the IRIX system saved my password as CTRL-D.
Dave Bittner: Oh my.
Larry Cashdollar: So, I changed the root password on the SGI Onyx/2 to CTRL-D. At this point I'm like dreading it. I'm like – I turned white and I'm starting to sweat and I'm like, shoot, I'm like, I got to go tell Dave the sysadmin, who doesn't like me very much, that I just changed a password on the Onyx to CTRL-D and he's not going to able to log into it. So I asked my friend Donovan who is much bigger than I am – I'm small and five-foot-six, he's six-foot-three. And I'm like Donovan, can you go tell Dave that I changed the password on the Onyx system to CTRL-D, and if he could fix it? And so Donovan was like, OK, sure...
Dave Bittner: (Laughs)
Larry Cashdollar: And Donovan came back like fifteen minutes later and he's like, dude, they are so mad at you. And I'm like, what, what happened? He's like, when you changed the password, they were in the middle of giving the Navy a demo of the 3D modeling on the Onyx/2. And he said there was an admiral standing there, there was upper management standing there, and they couldn't log in to the Onyx. And I'm like, oh my God...
Dave Bittner: Oh boy.
Larry Cashdollar: ...I'm like, I'm going to get fired. I'm like, son of a – I'm like, that's it, I'm going to lose my job. So, you know, Donovan came back a little while later. He's like, Dave managed to change the password, he had a prompt or a login open on his desktop to the Onyx, so he changed the password back to the original. And I'm like, OK, my manager's going to be here in a half an hour and tell me I'm going to get fired. And lo and behold, half an hour passes and my manager shows up, and he's like, I need to meet with you with my manager in his office. And I'm like, ah, crud, here we go.
Dave Bittner: Mm-hmm.
Larry Cashdollar: So, I go to my to his manager's office, and I notice on the wall it has a SANS security poster. And I'm like, this guy, like, knows about security. He's into you know, he's aware of the SANS Security Conference, and has all of the posters all decked out with all of these, like, security procedures and things that can harden your system and stuff like that, and the newsletters that you can get from SANS. And I'm like, cool, OK, well, this guy's aware of security, but I'm still gonna get fired.
Larry Cashdollar: So I sit down, and he's like, OK, first thing I need to tell you is that, before you do anything like this again, you need to tell me, your manager, and the person – the sysadmin whose machine you're testing, that you're going to be doing it and when. He's like, we can't have an incident like this happen again – example, the Onyx/2. And he says you disrupted an important demo, but I understand security is important. He's like, I've been talking to these guys about security, and he's like, I'm glad that you're actually out championing it, but I don't want you to actually break systems and get, you know, bring systems down. He's like, we can't have systems break because you're testing security, you have to do it in a safe manner. So he was like, you need to let me know.
Dave Bittner: And at this point, are you thinking, he said "again," he said "again." He's referring to the future.
Larry Cashdollar: Yeah...
Dave Bittner: (Laughs)
Larry Cashdollar: ...My brain is going, you still have a job, you still have a job. He said "again," that means you're here. And then he's like, so from now on, I want you to email me with a report of what you're testing, when you're planning to test it, CC your boss, CC the people that you're going to be asking to test. And he's like, if they give you permission to test, you know, you can go ahead and test, but you have to let them know and they have to say it's OK, because you can't test when they're trying to do something and you mess it up. And I'm like, OK. And he's like, you know, have a nice day. And I'm like, OK.
Larry Cashdollar: So I walked out of there and I'm like, great. I'm like, I have legitimacy now, you know, he gave me a framework that I can work in to tell people, hey, you know, your systems are getting a penetration test this week and you need to pick a day and a time when it's convenient for you for me to go and test your systems. Now, what this led to, most likely, is that folks would go ahead and try and lock their system down before then, but it did get them to secure the system before I tested it. So it was a win-win. They actually took security more serious, and then I still got to test the system. Now, it may not have been easy to get into, but at least it was fixed. So it was a – I lucked out. It was a good opportunity and thing that had happened after I'd gotten into that trouble.
Dave Bittner: Wow.
Larry Cashdollar: So, it was fun.
Dave Bittner: So, where did that ultimately lead? I mean, what's the sort of – the connecting dots between that incident and where you are today, and this bigger conversation about CVEs in general?
Larry Cashdollar: That sort of – that was like my first taste of blood. You know, it was like I really enjoyed finding a vulnerability and finding a way to exploit it and then getting access that I wasn't supposed to have. And it made me like – I like to solve puzzles like that. So, I started doing more – like, before I had gotten that or found that vulnerability, I didn't know how folks found vulnerabilities. It was – I was studying them, you know, the folks that I worked with at the company would always study these new vulnerabilities that came out and wonder, how did this person know where to look? How did they find this? And, you know, and then I realized, you just have to look. You know, if you're not looking, you're not going to find anything. If you're looking, eventually you're going to find something. You just have to know what to look for.
Larry Cashdollar: Now, I knew what to attack, I knew to look at setuid root binaries, so I managed to find something that somebody had looked at or found before. So that sort of gave me the mindset and the idea that if I look at other software and look for common programming problems or mistakes, I might find more vulnerabilities. And then that's what I've been doing for the last twenty years, when I can.
Dave Bittner: Did you ever have any more insights into that little Midikeys program itself? I mean, I think just for our listeners, MIDI is a protocol that musical instruments use to communicate with each other. And back in the late nineties, it was not as ubiquitous on computers as it is today – I'd guess that most systems today come with some sort of MIDI implementation just as a matter of course, because it's all over the place. But I'm just trying to imagine, you know, the folks who made this SGI the size of a refrigerator, you know, did there just happen to be a developer on that team who had a side hobby of hooking up, you know, playing with electronic keyboards at the time, you know, a Yamaha DX7 or something, and so that's why this was on this machine. Like, what was it doing there?
Larry Cashdollar: That I don't know. I assume it was just another little bell-and-whistle to add to the IRIX portfolio of neat things you can do with the system. And, you know, it just never occurred to me that there would be this music program on this operating system that was pretty much meant for 3D modeling. I didn't expect it, which is why I had to see what it was, because it just seemed so out of place.
Dave Bittner: Which is an interesting lesson in itself, right?
Larry Cashdollar: Right. You know, and then I later found out on Bugtraq after I published my findings to Bugtraq, that some of the folks on there said if you change the editor in your environment, your environment variable editor from "vi" to bin/sh, and you launched the editor from Midikeys, it just spits out a root prompt. So, you didn't even need to edit the password file. You could just pop out a root shell because it executes the vi editor as root. So it was – there was even a quicker way to get root. So it was – yeah, and I'm not sure why it was there and then why it was setuid root, but it was a fun find for sure.
Dave Bittner: Yeah. Well, let's fast forward to today. I mean, you and I have spoken on this program and other places many times. You have a number of CVEs under your belt. What's the latest from you?
Larry Cashdollar: I've been told it's three-hundred-and-five is what I've been told. I only have – I think I have only about two-hundred-and-something documented. But there are folks – like, I have a friend who had started at IBM X-Force and he has been tracking my vulnerabilities from day one. He actually mentioned to me over Twitter, he's like, I remember when you found that vulnerability, and it's like, oh yeah, you were running the database over at IBM tracking all these vulnerabilities, so of course you remember so.
Larry Cashdollar: So, it was just something that I had done. And then now it's – well, I guess I got a bunch from WordPress plugins because I had done a little experiment there. But, you know, and then now it's more web application stuff, is what I've been attacking more. Web application, and then I still have a soft spot in my heart for temp vulnerabilities. I recently found a /temp race condition vulnerability for – it was for Solaris 11 x86, where one of their utilities would create a file in /temp, and then it would chmod that file to be world writable. So you could use that to chmod /etc/shadow and change the password and get yourself to root. So I got a CVE for that like two months ago. And then I have two more, I think temp vulnerabilities coming out next month for Oracle Solaris 11. So, yeah, I don't know, it's just fun.
Dave Bittner: Yeah. Well, you know, when you look back on this incident in particular, you know, as a way to sort of, you know, you got your start, it kind of set the hook for you, you know...
Larry Cashdollar: Exactly.
Dave Bittner: ...Set you on this path, right? How has that informed – the way that this played out and how it informed the way that you look for these sorts of things? I mean, do you think there are dots to connect all the way back to that, that that really set you on this path and you still approach things in a similar way?
Larry Cashdollar: Yeah, I think that – I don't think if I had ever found that vulnerability, I don't believe I would be where I am today, because that was a spark for me to say, OK, this is – you've done this once. You now know where to look and what to look for, and you were able to find something just from looking. You can keep looking and find more things to find security vulnerabilities in. And from then on, I was always interested in finding security holes.
Larry Cashdollar: And I remember in early 2000s I would open up trade magazines and look through them for software that ran on Linux that you could get a free demo. And I would download the demo and I would look for vulnerabilities. And I would find, you know, either a temp vulnerability, where it was a race condition and you could elevate privileges to root, or there were some where they had – there was code execution errors and things like that. So that was how I used to find vulnerabilities then.
Larry Cashdollar: And then working in the Akamai CERT in the last couple of years, when I had joined there, my then-boss was like, you know, if you like breaking stuff, you should really look at WordPress plugins – those things are full of holes. And I'm like, OK, so I started looking at those and found lots of vulnerabilities in those because there are just so many that didn't really have a process for checking the security or the coding requirements for those plugins. So I ended up finding a lot of stuff with that. And I did a bunch with Ruby Gems. So it's sort of like a – I don't know, like a prairie fire for me, that that was an initial match.
Dave Bittner: (Laughs) You know, I think something you and I have in common is, you know, those of us who remember this era of computing, I think most of us have a certain amount of affection for it. You know, there were things that were harder than they are today, but I think it's easy to look back on those days fondly. And I'm wondering, you know, for folks who are getting started today, who are – the folks who are, you know, perhaps we have students listening to us, or folks just starting out in their career, who knows what the next twenty years is going to hold for them in terms of the changes that they're going to see, similar to the changes from twenty years ago. And I'm wondering, do you have any insights there on the, you know, how those things from long ago are still informing the work you do today? Any tips or words of wisdom for the folks who are just starting out?
Larry Cashdollar: Just, you know, don't be afraid to fail, and really just realize that you're going to learn from your failures and you really want to just learn how to just Google stuff that you don't know. Search for things, and just, you know, if you find something that you're interested in, try and find out as much as you can about it and try and look for things that could benefit your – if you're looking to find a vulnerability in it, I guess is what I'm trying to say – learn as much as you can about that thing and then don't lose your traction for learning new stuff. I'm always learning new stuff and it gets more tiring as you get older, but in your twenties and thirties, you know, you should just try to absorb as much as you can.
Dave Bittner: And always check with the boss to make sure there isn't an admiral on site getting a demo before you do your stuff, right?
Larry Cashdollar: Yeah, don't do anything illegal...
Dave Bittner: (Laughs)
Larry Cashdollar: ...So make sure you got permission to do stuff.
Dave Bittner: Our thanks to Larry Cashdollar from Akamai for joining us. His blog post is titled, "Music to Hack to: My First CVE and 20 Years of Vulnerability Research." We'll have a link in the show notes.
Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.
