The supply chain in disarray.

Dave Bittner: [Music] Hello everyone and welcome to the CyberWire's Research Saturday. I'm Dave Bittner and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us. [ Music ]

Elad: So initially we started searching and researching for workflow vulnerabilities and built pipelines attack vectors.

Dave Bittner: That's Elad, a senior security researcher at SciCode. The research we're discussing today is titled SciCode Discovers a Supply Chain Vulnerability in Bazel. [ Music ]

Elad: Meaning, try to map the whole surface, the whole attack vector surface and try to find where could be misconfiguration, where can be vulnerabilities in workflow. One of the vulnerabilities that could be made in workflows are command injections vulnerabilities. So let's try to break it down, what it means from the beginning. So command injections are not new. They exist in web applications and they exist in anywhere that user input is inserted and is not properly being verified and then is being executed somehow. So if the user is inserting some malicious input, it could be executed and will be run in some other system. So the same thing could happen in build workflows. Workflows run automatically. Workflows run based on triggers like push event, issue event or fork event or anything else that, let's say, GitHub offers us. Let's say we have a workflow and that workflow checks if the issue title contains a specific prefix, for example. So every new issue, that pipeline will be automatically triggered. We'll check that if the issue title contains or something like that and we'll return some result. Maybe we'll open a new comment on the issue. So now, as far as we understand workflows and the automatic procedures of them, let's try where it can be tempered and where the attack vector is regarding injections of workflows. So let's say a user opens an issue. And like we said before, there's a workflow that checks the issue title. If the issue title is not treated properly, then the user could insert a malicious payload as the user as the issue title, and then it will be interrupted in the workflow and will cause an injection. Meaning a user input that is not being serialized and treated properly could be interrupted and inject the entire workflow. So those kinds of vulnerabilities are already well known. We published several articles about them, and we thought about creating something else, researching something else. Those kinds of vulnerabilities could be not detected by standard security tooling. Meaning, if we already know how to find command injection vulnerabilities in workflows, let's try to think about something else. So we decided to map GitHub actions and try to map the entire components that users can use inside workflows. And one of those components was, well, composite actions. So composite actions are like functions inside code. You call composite action. It is containing some code or calling another composite actions, so it runs specific procedures and maybe returns a result. Maybe it's not. It inputs some parameters. Maybe it's not. So we decided to look for workflows that wasn't vulnerable to command injections, but use [siren] a composite action that were vulnerable to command injection. So the battle story starts right there, meaning a workflow, but that by itself wasn't vulnerable to any command injection, just used two composite actions. But the command injection was inside the composite actions. So it was way harder to find, way harder to maybe standard security tooling to find this kind of vulnerability because the command injection wasn't in the workflow itself, but in a composite action, in a function, you can say that this workflow called inside the workflow. [ Music ]

Dave Bittner: We'll be right back. [ Music ] So I understand here, I mean, it sounds like the composite actions are kind of code within code. So does that make it so that someone can change something within the composite action and then it might not be completely obvious to the user of that composite action that something has changed within it?

Elad: So you cannot change the content of the composite action because like us in the workflow itself a YAML file that are stored in a repository. In that case, it was another repository also maintained by Bazel. But you can try to find command injections vulnerabilities in those composite actions and try to map what are the workflows that are using this kind of composite actions. So you cannot touch the content of the composite action, but you can try to find vulnerabilities inside that and try to roll it back and see what workflows are using this kind of composite actions. So if a composite action is vulnerable to one kind of vulnerability, if a workflow is using that composite action, the workflow is vulnerable too.

Dave Bittner: Hmm. Well, tell us about this sample attack that you all put together here. What's going on?

Elad: Of course, we tried to map GitHub and GitHub actions and we came across Bazel, the Bazel repository. So let's start from the beginning of the workflow. We saw a workflow that was being triggered on a milestone. A milestone is an event where a maintainer can mark an issue as milestone. So if a maintainer would milestone that issue and composite action, a workflow will be triggered and a composite action will be called. And when investigating that composite action, we can see that that composite action was vulnerable to command injection because it used the issue title and issue body in an unsecured way. They passed directly the issue title to an inline script, meaning if a user would tamper the issue body, for example, it would go to the workflow and then to the composite action and then inject the workflow from the composite action.

Dave Bittner: And so what are your recommendations here for folks to protect themselves? I guess I should say first that you went through a disclosure process here with Google and they awarded you all a bounty.

Elad: Yeah, so let's start with mitigation. It is unrecommended to pass directly the user controlled input into inline scripts in GitHub, in GitHub actions. So the recommended manner here is to first declare those user inputs as environment variables and then pass those environment variables into the build pipeline. So this process mitigates that issue. So Google took two procedures to fix that issue. The first one is the one that I just said. They declared environment variables and passed those environment variables to the inline script. The second manner they took was to lower the permissions as low as possible of the workflow. In that case, they removed completely the workflow permissions. If someone tried to inject code, even if they succeed, the workflow has no permissions and the attack vector will be lower. But of course that we want to remove the scenario of any injection. So they also mitigated that risk in the composite action. So those two completely removed the [inaudible 00:10:43].

Dave Bittner: Have you seen any examples of people using this technique in the wild?

Elad: So, so far, we haven't seen any behavior like that. It all started by we research new attack vectors of workflows. Then we mapped to see if there's any vulnerable workflows. And it came as a research purpose of us for trying to learn and investigate in more attack vectors of workflows and then try to map using Raven and the entire surface and see if something is vulnerable in the area, in the field of GitHub Actions. Bazel wants one of them. But we haven't seen such technique in the wild. And we hope that using the blog about the Bazel, others can take this example and avoid such scenarios using the recommendation and using everything that we suggested.

Dave Bittner: Are you satisfied with the response that you've had from Google?

Elad: Yeah. Google were very quick. So we opened an issue to Google. We wrote a report using and we detailed everything that we found. And we gave examples and everything that they needed. They quickly responded. They were very nice. It was very great to work with them and they acknowledged the vulnerability really quick. They mitigated by add vulnerability, like I said, by lowering the permissions of the workflow and mitigating their command injection vulnerability. [Music] And it was all very nice, very smooth. And like every vulnerability disclosure should be. So we very glad and pleased with the disclosure process we've done. [ Music ]

Dave Bittner: Our thanks to Elad, senior security researcher from SciCode. The research is titled SciCode Discovers a Supply Chain Vulnerability in Bazel. We'll have a link in the show notes. [ Music ] The Cyberwire Research Saturday podcast is a production of N2K Networks. N2K strategic workforce intelligence optimizes the value of your biggest investment, your people. We make you smarter about your team while making your team smarter. Learn more at n2k.com. This episode was produced by Liz Stokes. Our mixer is Elliot Peltzman. Our executive producers are Jennifer Eiben and Brandon Karp. Our executive editor is Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next time. [ Music ]