Middleboxes may be meddling with TLS connections.
Dave Bittner: [00:00:03] Hello everyone, and welcome to the CyberWire's Research Saturday, presented by Juniper Networks. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, and solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.
Dave Bittner: [00:00:26] And now a word about our sponsor, Juniper Networks. Organizations are constantly evolving and increasingly turning to multicloud to transform IT. Juniper's connected security gives organizations the ability to safeguard users, applications, and infrastructure by extending security to all points of connection across the network. Helping defend you against advanced threats, Juniper's connected security is also open, so you can build on the security solutions and infrastructure you already have. Secure your entire business, from your endpoints to your edge, and every cloud in between, with Juniper's connected security. Connect with Juniper on Twitter or Facebook. And we thank Juniper for making it possible to bring you Research Saturday.
Dave Bittner: [00:01:13] And thanks also to our sponsor, Enveil, whose revolutionary ZeroReveal solution closes the last gap in data security: protecting data in use. It's the industry's first and only scalable commercial solution enabling data to remain encrypted throughout the entire processing lifecycle. Imagine being able to analyze, search, and perform calculations on sensitive data, all without ever decrypting anything - all without the risks of theft or inadvertent exposure. What was once only theoretical is now possible with Enveil. Learn more at enveil.com.
Nick Sullivan: [00:01:53] Several years ago, I worked with researchers from the University of Michigan and a bunch of other institutions to help study what's going on on the Internet.
Dave Bittner: [00:02:02] That's Nick Sullivan. He's the head of cryptography at Cloudflare. The research we're discussing today is titled, "Monsters in the Middleboxes: Introducing Two New Tools for Detecting HTTPS Interception."
Nick Sullivan: [00:02:14] Cloudflare is in a great position to be able to see a large percentage of what traffic is flowing over the Internet, and help study what clients are actually making these requests and what traffic on the Internet actually looks like. And during that period, we were developing new technologies for encrypting the Internet. So, Cloudflare was one of the first proponents of a protocol called TLS 1.3, which is the latest web encryption standard for - anytime you see that nice little lock icon in your browser and HTTPS, TLS is the protocol that encrypts it.
Dave Bittner: [00:02:50] Hmm.
Nick Sullivan: [00:02:51] And so, when we were developing TLS 1.3, we're really changing what the protocol looks like on the wire. So, we were bumping the version number, we were changing a couple of fields around in there. And when actually deploying this protocol, there were some unexpected failures and unexpected issues, and it turned out that these were caused by the way that the architecture of the Internet is built for some users. So, it turned out that some users in corporate environments or in managed infrastructures, the encryption between the browser and website was not actually fully end-to-end. In fact, this is something that we knew, but how widespread it was wasn't really well-known to us.
Nick Sullivan: [00:03:35] So, it turns out that there's several companies and organizations who provide services to allow the corporation or whatever the managed service it is that's providing Internet for people in a corporate environment, or a high school, or a university, or things like this, to be able to inspect encrypted traffic, and make editorial decisions, or block malware. There's many different reasons that you'd want to look inside of encrypted connections. And so, it turned out that these middleboxes, which - these were often actually pieces of hardware that were shipped to corporations to help inspect this traffic - were causing some issues with TLS 1.3. They were not necessarily compatible with some of the new changes that were happening.
Dave Bittner: [00:04:18] Hmm.
Nick Sullivan: [00:04:18] And so, in this study, we tried to figure out what it was that was going on, and in what way is traffic between what a user sees - with the nice lock icon and typing in the actual HTTPS URL on the site - and how that traffic transmits from the browser to the server and back, whether or not this is end-to-end encrypted, and if not, what is actually going on. This is the kernel of the research project that we did, and this was published in NDSS, which is a computer security journal. And this kind of led us down the trail of seeing how we could actually take this kind of one-time study that was done with these researchers and turn it into a continuous dashboard that people could see what's happening on the Internet is evolving.
Dave Bittner: [00:05:05] Let's walk through together some of the basics here for folks who may not be up on it. Can you walk us through how TLS works? If I'm not in a corporate environment, it's just me and my computer and I'm trying to connect to someone else, and we have that secure connection, what's going on there?
Nick Sullivan: [00:05:20] If you're typing in, say, www.securesite[.]com into your browser, there's several steps. The first is DNS, which is the Domain Name System lookup. This is what your computer uses to translate a name to an IP address. So, your computer does the request to a DNS server, which then tells you an IP address, whether IPv4 or IPv6. And then your machine has the address of the server that it's connecting to, and then it begins what's called the TLS handshake.
Nick Sullivan: [00:05:53] And the TLS handshake is a back-and-forth that the browser and the web server do in order to establish shared cryptographic keys that can be used to encrypt and authenticate any subsequent requests and response that comes between your browser and the website. And part of this handshake is used for the server to prove ownership over that hostname. So, the server provides a certificate, and the certificate was issued by a trusted certificate authority, and it says, you know, "my securesite[.]com" in it, and the certificate is associated with the key that is used to sign the handshake.
Nick Sullivan: [00:06:31] So, the way the handshake goes is the client sends a first message called the "client hello," which indicates which cryptographic algorithms it supports, and the server will respond with a server hello that says, great, of the algorithms that you advertise support for, here's the ones that I choose, and also here's my certificate, and here's the proof that I actually control the key associated with this certificate.
Dave Bittner: [00:06:56] Hmm.
Nick Sullivan: [00:06:56] And then, from that point onwards, the communication between the client and the server is encrypted and authenticated. So anybody seeing this traffic along the Internet really only sees encrypted packets, so they can't really tell what you're requesting, what the content of the website is, or anything you're sending from that point on.
Dave Bittner: [00:07:14] I see. And then, so, in these corporate environments that you describe, where someone has the opportunity to inject themselves into this stream, what's happening there?
Nick Sullivan: [00:07:22] In environments in which your machine is managed, as a user - you may have a phone that has a mobile device management profile, this is a system in which the corporation installs some additional software to help onboard you onto the system - what happens is, rather than the server using its certificate to complete this connection, when the client connects outward from the network, it actually goes to this piece of hardware called a TLS intercepting middlebox. And this TLS intercepting middlebox has the ability to basically print out any certificate it likes for any website at all. So, this certificate itself is not by a publicly trusted certificate authority, but it's by a specific certificate authority that's specific to that corporation home. So, your device, you get enrolled into the system, and typically there's a set of trusted local certificate authorities that your browser trusts, and you will only trust websites that have certificates from those certificate authorities.
Nick Sullivan: [00:08:28] When you're enrolled into one of these systems, they install a corporate certificate authority, and this corporate certificate authority is something that the middlebox is able to issue certificates from. And so, when you think that you're connecting to the actual website itself, in fact, you're getting a certificate that is created by this middlebox. And so your connection is end-to-end encrypted between your browser and this middlebox. And then, the middlebox decrypts the traffic and then creates its own TLS connection from the middlebox out to the website. And so, effectively, you are getting an encrypted connection to the middlebox and then the middlebox has an encrypted connection to the website itself.
Dave Bittner: [00:09:09] And the main reasons that this goes on, the things that folks want to look at are what?
Nick Sullivan: [00:09:14] Well, oftentimes, there are restrictions with regard to what type content that people want to allow people to view at work. There's also restrictions with respect to what type of data can be transmitted outside of the enterprise. So, DLP or data loss protection is a requirement in several compliance regimes to make sure that the folks that are inside the company aren't sharing private corporate information outside of their corporate bounds, and so these middleboxes can sometimes detect this and report it, and can kind of catch people leaking private information. Oftentimes, it's also used to see if you have - if there's malware, if there's some sort of virus or something on the user's computer that is attempting to reach out to some kind of command-and-control server or some orchestrator that lives outside the corporate bounds. So, you're looking for people leaking data, you're looking to restrict the types of websites and content that people can view at work, and you're looking to try to prevent malware or detect malware inside a corporate network.
Dave Bittner: [00:10:25] And so, your research was looking at different types of interception of HTTPS traffic. What were you looking at here?
Nick Sullivan: [00:10:33] We knew that these connections, for a large percentage of the Internet, were not end-to-end. And so, we knew there were corporate middleboxes. We knew that there are services like antiviruses that effectively do the same thing, where the antivirus software is installed on your machine and acts as a software middlebox on your machine to look for different malware signatures. And we didn't know exactly how widespread these were or which specific technologies were used, and which technologies were used to what extent. And so, this is really what we're looking for, is what is going on?
Nick Sullivan: [00:11:14] For the percentage of the Internet that is not actually the browser connecting to the server directly, from the servers perspective, we can tell that it's not actually the browser, and in TLS, there's certain fingerprintable characteristics that are innate to the type of TLS client software you're using. So I mentioned, in the client hello you list the cryptographic parameters that you support. Well, some of these middleboxes don't necessarily support the same ciphers as a browser, so you can tell whether or not the request to the server is actually being sent from the middlebox or from the antivirus provider or directly from the browser, because its - there's a unique fingerprint for each of these pieces of software.
Nick Sullivan: [00:11:54] And so, we were trying to figure out what these are, and from that, whether or not these connections are secure. The browsers invest a lot of time trying to use the best possible encryption algorithms, and use different techniques like certificate transparency and various ways to make sure that these connections are actually safe and secure. And browsers invest a lot of time and have big security teams to do so. And we were trying to see if these middleboxes, or these antivirus services, whatever these man-in-the-middle, as you might call these middleware pieces of software, whether or not they were secure - essentially give a letter grade to whether or not they're following the best practices with respect to what you would expect the browser to do to have a secure TLS connection.
Dave Bittner: [00:12:44] And what did you discover? How did they rate?
Nick Sullivan: [00:12:46] Well, some of them were quite good, but some of them failed quite miserably. In fact, some antivirus services that we discovered did things like not even validating the certificate at all. So, if you imagine you're going to a website and your browser connects to the local antivirus middlebox and then that antivirus middlebox would connect out to the website, some attacker in between could potentially hijack that connection, because the antivirus itself is not checking that certificate. Other things we found were using weak and outdated ciphers. Some of them were using ciphers like RC4, which has had known vulnerabilities for the last five years or so, but they were still using these old technologies.
Nick Sullivan: [00:13:31] So, the letter grades rated from - some of them had a really good rating of A that we sort of mark as equivalent to how a browser would do their TLS encryption, to some that were a straight F, which actually destroyed all semblance of security that you would want to have from an end-to-end encrypted connection. So, the lock kind of itself is supposed to indicate trust, and in some cases these middleboxes, they had bugs and they were violating this trust.
Dave Bittner: [00:14:00] Would there be any way that the user would know that they had a problem?
Nick Sullivan: [00:14:04] In fact, not really. Uh, (Laughs), which is one of the disturbing pieces. In corporate modes, if you will, where your browser is connecting to a site and it's trusting a non-publicly trusted CA - some corporate CA or some antivirus CA - there's no real indication to the client that you're using one of these other CAs, or that the certificate provided by the website is not a real Web publicly trusted certificate. And furthermore, there's no indication to the user what type of TLS the second hop of the connection is using. So, users really have no way to know.
Dave Bittner: [00:14:41] And so, what are the ramifications of that?
Nick Sullivan: [00:14:43] There's several ramifications. First of which is user security for a lot of these situations is decreased. And this is actually a surprising situation, because oftentimes these middleboxes and antivirus programs are implemented in order to increase user security. You know, protect them from malware, protect them from other things that are malicious on the network. But in the end, it turns out to be weakening their encryption.
Nick Sullivan: [00:15:12] So, this is something that we did report, in the original paper, to a lot of these providers, the ones that we could identify. And they managed to, in many cases, update their software, do things more safely. At the very least, it exposed this as a potential problem, and it exposed the fact that if you are going to be doing this type of inspection for compliance reasons, or for whatever reason that you'd want to be doing it, that you really should take very, very good care of how you do it. Because building a secure TLS implementation is challenging to do, and browser companies invest a lot of money and time and expertise in this, and sometimes these middlebox vendors don't necessarily have the resources to do so. So it became something that was kind of a call to action to the industry to improve their best practices.
Dave Bittner: [00:16:02] And were you able to observe any instances where these were actually being exploited out in the wild?
Nick Sullivan: [00:16:08] It's not something that we particularly looked for in this study. We were more looking for the signatures of the connections and trying to figure out what actually was happening.
Dave Bittner: [00:16:18] I see.
Nick Sullivan: [00:16:20] This is more of a measurement exploration study itself. And we didn't necessarily look for specific instances of exploitation, but you can't necessarily rule it out.
Dave Bittner: [00:16:31] So, based on what you've learned here, what are your recommendations? How do people best protect themselves?
Nick Sullivan: [00:16:37] Well, if you're a corporation, ask your vendor how strong the TLS connection is with respect to connecting from the middlebox up to the server. So, we created a service called MITMEngine, which is something that can - on the server side, you can detect which of your clients are actually using one of these inspection services, and you can kind of provide recommendations. So, as somebody who's operating a server, you can inform users that potentially whatever their corporation is using for middleboxes is not doing the right thing. This is something that our open-source library is to do, to provide to people.
Nick Sullivan: [00:17:19] And overall, it's also indicating that there is room for a philosophy change with respect to how corporations deal with internal security. So, the numbers that we see - we have this dashboard, it's called malcolm.cloudflare.com, where we take the daily statistics and aggregate them, and look at the fingerprints, and say, is this really coming from a browser? Do we really think that this is coming from the browser that is sending the request? And our rates of interception that we're seeing are somewhere around 30 percent. Around a third of Internet users are potentially vulnerable to someone doing a bad job on one of these middleboxes. And so, it becomes very important for these providers to do a really good job and put them through thorough testing.
Nick Sullivan: [00:18:03] What I was mentioning in terms of a philosophy, it does bring you to this opinion, or this possibility, that doing something that is pro-security - in this case, having an inspection looking for malware trying to protect users - is actually putting them at risk, and it might be doing less of a service than what you really think it is.
Dave Bittner: [00:18:25] Is there any sense that this is kind of a set-it-and-forget-it kind of thing, where people set up these systems in their organizations, as long as it keeps running and it's working, it's not something that they go back and check in on to make sure that it's doing what it should be doing?
Nick Sullivan: [00:18:39] Yeah, exactly. Browsers have been at the forefront of computer security with respect to auto updates, and so whenever a browser has a vulnerability, they're on a pretty frequent schedule of auto-updating...
Dave Bittner: [00:18:52] Mm-hmm.
Nick Sullivan: [00:18:51] ...There's something of a six-week cadence for something like Chrome. So, they're constantly scouting the worlds, looking for vulnerabilities, deciding to change which ciphers they're using, implementing new things like TLS 1.3, which is a much more secure protocol, and update. This is not the norm for all security products. I would wish it would be, and I think most of us in the security community think auto update is a great idea. But browsers are really at the forefront, and some of these other technologies are, as you said, kind of set-it-and-forget-it, and often they don't get patched automatically, or if they do get patched, it's something that's manual and causes downtime. And so, having your TLS endpoints be something that's more frequently updated is likely going to result in better security than having it be something that's some appliance in the back room of the data center that you don't necessarily update.
Nick Sullivan: [00:19:49] Some of these middleboxes, they do attempt to replicate the signature of the browser underneath it, and those can be detected as well. So, we've built several different techniques here to detect this, and it's based on just the variety of things you can do with TLS. There's different extensions you can support, there's different ciphers you can support, and then there's different quirks that involve which order different extensions are, or which order different ciphers are in. And so, if you have the time to dig into the MALCOLM dashboard, you can take a look at what these different appliances and pieces of software do. It's interesting to explore which types of devices are more likely man-in-the-middled than others. This can give anybody who's generally interested some insight into what types of corporate environments are more likely to do interception towards in or out.
Dave Bittner: [00:20:47] Our thanks to Nick Sullivan from Cloudflare for joining us. The research is titled, "Monsters in the Middleboxes: Introducing Two New Tools for Detecting HTTPS Interception." We'll have a link in the show notes.
Dave Bittner: [00:20:59] Thanks to Juniper Networks for sponsoring our show. You can learn more at juniper.net/security, or connect with them on Twitter or Facebook.
Dave Bittner: [00:21:09] And thanks to Enveil for their sponsorship. You can find out how they're closing the last gap in data security at enveil.com.
Dave Bittner: [00:21:17] The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technology. The coordinating producer is Jennifer Eiben. Our CyberWire editor is John Petrik. Technical editor, Chris Russell. Our staff writer is Tim Nodar. Executive Editor, Peter Kilpe. And I'm Dave Bittner. Thanks for listening.
