Detecting Twitter bots in real time.

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Dave Bittner: Thanks for listening to the CyberWire's Research Saturday podcast. Today, I want to reach out to those members of our audience who are students or serve in the military. Did you know that the CyberWire has special CyberWire Pro subscription offers just for you? Well, you do now. Because of your student or military status – that's active or reserve military status – you are able to subscribe to CyberWire Pro or CyberWire Pro+ at a significant discount. That means you can unlock access to our Focus Briefings, exclusive podcasts, quarterly analyst calls, premium articles, and much more. To learn more, visit thecyberwire.com/pro, and click on the "Contact Us" button in the "Academic" or "Government & Military" box. That's thecyberwire.com/pro, and then click "Contact us" in the box that applies to you, and we'll hook you up. Thanks again for listening to Research Saturday.

Dave Bittner: Thanks to our sponsor, Reservoir Labs. Reservoir knows that cybersecurity teams need full network visibility to discover new threats, tactics, and behaviors. This is true today more than ever. Reservoir Labs provides solutions based on rock-solid, enterprise-class network sensing, and spectral hypergraph analytics, using advanced algorithms and mathematics to deliver for your team and your network. Contact Reservoir to learn how you can gain comprehensive threat visibility in minutes. Learn more at reservoir.com/cyberwire. That's reservoir.com/cyberwire. And we thank Reservoir Labs for sponsoring Research Saturday.

Daniel Kats: I started looking at disinformation campaigns by foreign actors early last year.

Dave Bittner: That's Daniel Kats. He's a Senior Principal Researcher in the NortonLifeLock Research Group. The research we're discussing today is titled, "Introducing BotSight: A New Tool to Detect Bots on Twitter in Real-Time."

Daniel Kats: NortonLifeLock – previously Symantec – has a history of looking at state-sponsored campaigns in the security space. So, we look at advanced persistent threats, malware that's distributed by state actors. And we had this conversation within the Research Group that disinformation is very similar to these kinds of threats, but it's operating in a space that doesn't have as much scrutiny from dedicated professionals. So, we started looking at the data and this was the end result after a lot of work and a lot of back and forth about what the best way to tackle that problem is.

Dave Bittner: Well, let's walk through it together. I guess, can you describe to us, first of all, what is the tool that you all have released?

Daniel Kats: We released a tool called BotSight. The idea of the tool is that it can flag accounts which are behaving in such a way that they're very similar to social bots. So, these are accounts that state-sponsored groups use to spread disinformation. So, when you install BotSight, which is available as a browser plugin for all major browsers or as an app for iOS, you can use this tool to see percentages right in your Twitter feed of the likelihood that a given account is acting as a social bot.

Dave Bittner: Well, take us through what's going on under the hood here. I mean, how did your team come at this problem and analyze the data to be able to come up with these percentages?

Daniel Kats: So, there's a big reservoir of data that we analyzed. We actually analyzed four terabytes of past tweets that we got our hands on through various sources. There's actually been a lot of academic work in this area, but it's been focused on older datasets. So, while we took a lot of cues from the academic world in terms of our approach, the data that we used is a little bit newer.

Daniel Kats: So, in the background, we have a machine-learning model that takes in approximately twenty different features from a given account and calculates what is the probability that this account is a social bot. These features are based on historical examples of social bots in the past. And this percentage is what we call "calibrated." So, a lot of the time machine-learning models, they return just numbers between zero and one. But these don't really correspond to percentages. But we have calibrated it so that whatever percentage you see in the feed, that is the actual likelihood that that account is a social bot.

Dave Bittner: And how do you check against yourself in an ongoing way? I mean, how are you making sure – I guess maintaining the integrity of these evaluations over time?

Daniel Kats: That's a great question. So the short answer is testing. We have a lot of people now who are using this tool and who had been using this tool internally at NortonLifeLock before we released it for about five months. And people were constantly coming back with feedback. As you know, different people use Twitter in a variety of different ways. There are so many accounts and a lot of languages. NortonLifeLock is a global company who has employees from all over the world, and they're telling us that they're getting unexpected results here and there, and so we're constantly tuning the model for months and months before we go on something that we were really, really happy with. And this kind of highlights the difference between academic research data sets that we initially started using versus a real-life deployment. And the real world is much more messy than the small academic datasets that researchers tend to use.

Dave Bittner: Can you give us some insights on the types of things that you and your team had to do to fine-tune the results?

Daniel Kats: Absolutely. So, one of the things that we found was that a lot of celebrities kind of behave like bots. And the reason for this is they use social media management tools in order to coordinate their posts. They release the same post, for example, on Instagram and on Facebook and on Twitter and on other platforms. And it looks a lot like the coordinated activity that we see. And so we had to make some adjustments for that kind of behavior. Another example are ad campaigns by corporations which kind of behave in the same way.

Dave Bittner: Can you give us an idea – I mean, what is a typical behavior that differentiates a bot from a real human?

Daniel Kats: So, there are a few different behaviors. One is we find that groups of accounts that act together in concert over a prolonged period of time, they tend to belong to a single bot network. This is very rare for humans, because humans are fickle. Some days you tweet a lot, other days you don't tweet at all, and you don't see this long-term, collaborative behavior. That's one thing that we've observed.

Daniel Kats: Another feature that we've observed is that social bots are very bad at coming up with their own organic content. They generally try to amplify, so they do a lot of retweeting. And social bots – they're they're really not normal Twitter users. So, on Twitter, you might engage with the content. You might reply to some people, you might like posts. But that kind of what we might say "passive and active engagement" is actually very rare for bots. Bots tend to either retweet – that's the most common behavior – or they tend to generate kind of generic tweets. So they'll tweet a news story, for example.

Dave Bittner: Now, I've installed the plugin here for myself and I've been playing with it with Twitter, and first of all, I have to say it is a lot of fun (laughs). So there's that element of it. But it's also fascinating to see these numbers scroll by. And I'm curious, you know, what do you anticipate being a typical use case of this? How do you – ideally how would you like it to contribute to how people use the platform?

Daniel Kats: That's a great question. I think that there has been a lot of discussion in the media about bots and fake news, and I think that a lot of people are aware of the problem – that there exists these social bots on social media. But I think most people don't really have a sense of where these bots live. And this can be a little bit toxic because every time there's some odd opinion that's a little bit different, you might see on Twitter that people will call this person out as being, for example, a bot or a troll or something like that. And so we really wanted to contextualize where are you most likely to find bots? What are the typical bot behaviors? And we wanted to do it in a way that is very clear to the average user. You know, we can always release a paper and talk about these bot behaviors. But I think this really helps educate a person in a way that is interesting, is obvious.

Daniel Kats: Yeah, it's almost like you have an expert sitting over your shoulder while you're scrolling through things – you say to yourself, hmm, that seems a little bot-like, you can then look at the results from the plugin and say, yep, yep, it likely is. You know, someone else agrees. Or I guess the other direction where you can say, no, you know what, that is probably a real person.

Daniel Kats: Exactly. And the other angle of this is we wanted to create a sense of critical thinking about where are tweets coming from? So, one of the key instigating ideas behind this tool was in early March, I think, back when the US Democratic primary was in full swing – I don't know if you remember it because it feels so long ago – but it was a contest between Bernie Sanders and Joe Biden. And right when Joe Biden won the South Carolina primary, there was some coverage by major publications that talked about, you know, there was a lot of anger online from Bernie, people that were tweeting with certain hashtags, like #riggedDNC, for example. But when my colleagues and I looked deeper into these trends, we found that, specifically for that one, there was actually a lot of non-organic activity. So it didn't really appear that these were legitimate Bernie supporters, but were actually outside actors. And so we felt like BotSight, going forward, might help journalists to think about the possibility that these trends that they're seeing – they actually might not be organic.

Dave Bittner: What sort of insights have you gained on Twitter itself? As you've been going through this process and gathering this data and fine-tuning the tool, do you have a sense for where we stand when it comes to Twitter and the ubiquity of bots on the platform?

Daniel Kats: I think that we actually stand in a really good place right now, contrary to some of the coverage that you may see, because Twitter has gone after the bots – at least certain types of bots – very aggressively. And so, from our own research, we see a marked decrease in the overall amount of bots – we call it the "background radiation" of bot activity on the platform – from almost twenty percent in 2016 to around five percent currently. Five percent is still a high number, but it's a lot better than where we used to be.

Daniel Kats: However, this doesn't address other misinformation problems that our tool doesn't tackle. So, for example, people retweeting misleading claims is just something that we don't address. So, for example, there was a picture that people were tweeting that seemed to show an explosion in DC over the past little bit, but it turns out that this was just a still from the show "Designated Survivor." And this is not something that our tool would catch unless this was sponsored by outside groups, which it doesn't look like it was – it was just organic activity.

Dave Bittner: Right. So if the bots latch on to it and start amplifying it, that's something that you would detect, but the truth of the post itself is not something that you're really aiming at.

Daniel Kats: Exactly. Exactly. And this is both an advantage of our approach and a deliberate design decision, but also something that I think that people have to keep in mind when they go on social media. So, on the one hand, just because something is not true doesn't mean it was posted by a bot. And secondly, just because the number of bots on the platform has gone down, doesn't mean that there is less false information.

Dave Bittner: Do you have any advice or things that you learned in this process – I'm thinking for the folks in our audience who may be working with artificial intelligence or perhaps there's students who are learning about this sort of thing – are there any insights that you can share through going through this process, anything that surprised you or was unexpected in using that sort of approach for this sort of information and challenge?

Daniel Kats: I have a few different insights. Well, the first one is what I said before, which is that your training set may not necessarily be representative of real-world data. So, our initial training set were common datasets that are used in the research community, things that are datasets that people have published many academic papers on. But we found that these datasets, if you just use them to train the models, they're actually not really representative of current use of Twitter. And whether this is because people are just using Twitter in different ways or because the datasets themselves are small, I'm not sure. But it speaks to the value of doing some kind of validation before you really deploy your models into the real world, and being able to adjust your datasets accordingly.

Daniel Kats: We also did a variety of what's called "cross-validation" in a very specific way. There's a paper called "DBOT" which talks about the use of cross-validation to make your model more resilient. And the idea there is that you're specifically taking your training set and you're splitting it according to the different types of thing – in our case, bots – and you're training your classifier on – let's say you have five types of bots in your training set. You train on the first four, and then you see if your classifier can recognize the fifth type that it hasn't trained on. And this makes your model more robust, because if you start mixing in the different types of bots together in your training set, then what happens is when you encounter a new type of bot in the wild, you're not as sure that your model is going to detect that kind.

Daniel Kats: And I'd say the final thing that we really learned is that even if you have a false positive rate or a false negative rate of one percent or two percent, that can be still a lot if people are using your classifier and are really relying on it. You know, if you have, for example, one-hundred thousand or one million lookups, one percent of that is a big number. And so you have to think about these percentages in terms of the anticipated volume.

Dave Bittner: How do you and your team protect against your own personal biases sneaking into the various algorithms that you're using here? You know, how do you make sure and guard against that sort of thing?

Daniel Kats: So, when people send us accounts that they think are bots or are not bots, we are very conservative. And so we apply a consensus to these examples before we add them into our training set. So, if we're not all sure that something is a bot or is not a bot based on the examples that someone has sent in, we don't add them. This is one of the reasons why we try to stay away from actually using the content of the tweet, and our classifier really focuses on metadata. It does look at the hashtags, but it looks at not what the hashtags are, but for example, is the hashtag popular? How many hashtags are there? How many mentions are there? But in terms of the actual content of the tweet, whether it's political or whether it's medical, we try not to look at that, partially for this reason.

Dave Bittner: Why was it important for you all to put a tool like this out there, to make it widely available for free?

Daniel Kats: For me, this is deeply personal. I grew up in Russia in the early '90s. And my parents actually met handing out pro-democracy leaflets in the late '80s in the Soviet Union. And so, my family background has a rich history of both experiencing a wide campaign of disinformation and misinformation and knowing the real value of objective truth. And I really latched onto this issue quite strongly when we were looking at these kind of misinformation campaigns, and I felt like this was a real positive good that we can do in the world. You know, it's very rare, as technologists, that we can just put something out there to really help people in an unambiguously good way. And that really made me excited to do something like that.

Daniel Kats: Disinformation – it doesn't just come from one side or the other. It comes from both sides at once. The intent isn't just to deceive you, but it's also to inflame tensions. It's also to divide us. It's also to play to our existing biases. So we have to be especially careful when we are on social media to question things that we see that appeal to us.

Dave Bittner: Our thanks to Daniel Kats from NortonLifeLock Research Group for joining us. The research is titled, "Introducing BotSight: A New Tool to Detect Bots on Twitter in Real-Time." We'll have a link in the show notes.

Dave Bittner: Our thanks to Reservoir Labs for sponsoring this week's Research Saturday. Don't forget, you can learn all about them at reservoir.com/cyberwire.

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team working from home is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.