Exploits Now Beat Patches: Open Source Security in the AI Era - Patrick Smyth, Chainguard

[music] >> So, this is how Frontier AI changed risk tolerance for open source. Kind of wanted to subtitle it, you know, how I learned to love the atomic bomb or whatever but uh decided not to go for that. So, I'm Patrick Smith. I work for Chainguard. When I'm not thinking about the software supply chain or messing around with um you know, Frontier AI, uh I am hanging out in Queens. >> [snorts] >> I'm one of these annoying people who like saunas all the time. If you're out of town, we have some great ones in the city. There's 10th Street. That's one since around since the 19th century, and they have great food. And then there's one at FiDi, too. So, um you know, and then I know there's a lot of finance folks interested in this, so that's the one. 88 Fulton. That's the one if you're if you're down there. >> [laughter] >> Um okay. So, uh this is going to be the scaring you part of the the talk, basically, but um but this year So, here's some just some fun headlines.

So 17-year-old Osakan teenager, he's still in high school, used ChatGPT, wrote some scripts, exfiltrated something like 4.4 and 1/4 million records from the biggest um internet cafe in Japan, and then used one of the cards to to buy some Pokémon cards. Used one of the credit cards to buy Pokémon cards or whatever. So, people can be very smart and very dumb at that age. FunkSec. Somebody hits Thought to be an Algerian semi-technical person, hit 85 targets over the course of the last uh few months. Um one individual who has not been identified used Cloud CoVE to attack 17 organizations from ranging from healthcare to government to you know major corporations. Use cloud for everything from writing the ransom letters, picking the next target, organizing his files, and of course doing the attacks. Kind of a takeaway here is that the way I've been thinking about it is everyone kind of goes a level up in terms of threat. So if you were an actual engineer before, now you're like a team. If you were actually you know a team before, now you're like advanced persistent threat level. Um So here's a little another little

tidbit for you. This is a This is basically whether the patch comes before the exploitation. So the exploit you know obviously is the attack. The patch is remediation of the vulnerability. Back in ancient history in 2018 and 2019, it was 63 days. This is an average, so it's pretty noisy, but it tells you something. Um it's kind of there was this like secular trend that was going on for a long time that I guess is continuing though it's kind of been drowned out of like You could kind of think of it as you know digital application or something like that. But basically like every you know hospitals were using Python code all of a sudden and stuff like that. So that secular trend sort of was responsible for this tightening over the course of you know 21, 22, kind of probably 23. But then you see in 23 and 24 and last year it just kind of left, right? I wonder what could have been causing all of that. We'll talk about it. I think you already know. Um but now it's negative 70 days.

So actually the exploit the on average hits before the patch. That's just from Mandiant. So they have their Mtrends report. Um this is kind of what it used to look like if you were at add organization who uh you know you would have say uh you you have x amount of software in your stack, you run scanners against it um such a you know scanners like Grype, Trivy, etc. Trivy also had a bad year. Um you uh scan against it, you get your x number of CVEs, known vulnerabilities that are rated you know from critical, high, medium, low. Um that goes to a team. They get some amount of things to remediate. All very sensible, all very good. Um the only problem is 30 days may as well be a million years in terms of some of these attacks that have been happening. >> [snorts] >> So this timeline basically you can't you know I mean obviously you have to have some kind of timeline just because you're remediating on some cadence, but it basically doesn't work that well anymore. Um between uh attackers jumping on known vulnerabilities almost immediately, exploiting finding vulnerabilities out of ban, um and now these new sources of zero days cough cough we also know what that is.

Um you know mythos, new frontier models um that this timeline is basically completely out the window. Um this is a whole other area of uh of fun and enjoyment that we had this this last year. So this is um just these I don't know what you'd call it, but like these are I kind of you could think of it as the four horsemen here. They're the met waves and waves of malware. Um every month uh you know I used to do a meme calendar for um supply chain attacks in old presentations where I'd have like one a month one big one every month and then I used to be like a big attack every month guys. It's it's really bad out there. And now it's like it would be crazy. I could do a meme calendar of every day pretty much. And but the you know this is basically just I put um four up here, but three from one month um and then Shai Hulud which of course is just a perennial source of fun. But basically what is Shai Hulud? It's a a worm originated in JavaScript ecosystem. kind of showed everyone that you could do a huge worm and do really well. So, now there's huge numbers of copycats shai huluds, there's sandworm mode, all this kind of stuff. Shai hulud is a worm from Dune. Um so, you know, it's a worm, worm, very clever. And then some of the code of these things, if you look at it, is actually pretty I mean, you you know, shouldn't be too entertained by attackers like comments in their code or whatever, but it they they they name things in amusing ways. Um trivy got attacked, that's a scanner.

Um this is all the other three were all in March this year. Um trivy uh light it all in is a uh kind of rooter for for agents. And um you know, axios was only a supply chain attack, it was only live for 2 hours, 600,000 um downloads, okay? And a lot of these supply chain attacks So, this is malware we're talking about in in specific. A lot of these attacks are chaining one from the other because everyone has everyone else's dependencies, you compromise one person, now you're compromising other people. So, something's wrong in the state of Denmark. Um this is from Hamlet 2000. It's a weird movie. You can go back and watch Hamlet 2000. Um and uh yeah, it's agents, right? It's frontier agents. And I think everyone in the room kind of knew that. Um that they have been accelerating everything, but but you you put that salt and pepper on everything, you know, you're accelerating how you make code, you're accelerating everything. Um but but it also [snorts] attackers are making heavy use of these.

Um and the environments have changed a lot. Um I don't know if folks have heard, you know, maybe I'm old or whatever, or like I kind of I'm enthusiastic of early computing and stuff. There's this a concept called the eternal September. Basically, it was back in the day there would be these uh university networks. People were very like, you know, people probably wearing their bow ties or whatever. People were were very um polite, they had their like no community norms. Um everything would be stable. And then what would happen? September would roll around, students would join those accursed uh things that are always clogging up the university. Students would arrive and start messing up all the the networks and everything like that. They wouldn't know any of the norms. They'd be talking in capital letters. They wouldn't know any of the good emoticons or whatever. Um so, another way of thinking about it is the eternal September has kind of gone on forever. And this is how this happens all the time. Culture It's a cultural thing, but it's also like naturally you have an environment or community that is like high trust and moves to a uh sort of more adverse low-trust situation okay?

Um another way of thinking about this is this And this is just the basic problem of software supply chain security, okay? And it's very simple, but when you think this way, I think it kind of clarifies a lot is that you have users. They're relying on your infrastructure, on your software um to do the things that they do with with your with your your stack, with your business. Um you have a responsibility to them. Then, you turn around and you're actually pulling from upstream. You were pulling from uh places that are, you know, you're trying to vet, you're trying to keep up with it, but it's difficult. It's a And any of those things that you pull in could potentially compromise you and your users, okay? So, it's kind of But we're all in this position, right? We're all between in this very specific rock and hard place. And this is the base This is the fundamental problem of software supply chain security. Um so, again, we're going to move from just random gloo doom and gloom to actually what you have to and should do this year. But I'm just going to throw out a few things, but this is to make a larger point, okay? You have to do um if you're thinking about uh packages, you have to do cool down periods. If you are um you have if you're thinking about see uh CVEs, you have to remediate them faster than ever.

Um you have to sign, right? Um I'll talk about it later, but salsa and sigstore, you know, sign your your uh your artifacts. Um you have to worry about operation security on developer laptops, um including physical keys. Um there's a lot you have to worry about. Uh the point here isn't so much I think I'll dial in on a whole bunch of these as we go on, but the point here is actually like it's time to get serious. Like it's actually time to to take this seriously because right now it's not 2024, it's not 2023. It's a actually significantly a much more adverse environment out there than it was just a few years ago, okay? Um there's a and there's a lot to do and it costs and uh it's painful, but um some some of these you should do first and also I would say if you're still, you know, moving to containerize, if you're still moving to do CI/CD, um great time to finish those projects, right? Um because or I would say you you almost kind of need to because if you don't have some of those um basic fundamental uh pieces in place, it can become very hard to do these other modernization efforts that have become more important over the last couple years. Now luckily now you have AI to do some leapfrogging there, so I wouldn't be, you know, so down, but time to get that skunk works going, time to get those projects going, time to complete um your golden images program, etc., etc., okay? Centralize your containers.

>> [snorts] >> Um who are we? We're Chainguard. Um we've been thinking about this since 2021. Um we have uh four founders. They've created uh projects you have might have heard of them, Kubernetes, uh Sigstore Salsa um uh Google Distroless and we, you know, this is basically we're basically here to just just think about the software sup- supply chain security problem and try to like nuke it from orbit basically. Um you know, we're I'll get into some of the specifics that we do um in a second, but here's some just a few surfaces that I kind of want to dial into, okay? So, first First you've got um and these all interact in different ways, but you can think of them somewhat separately and sometimes it's helpful to think of them separately. Okay? On the one hand, you've got um malware, okay? So, that was like your trivia, your Axioses, everything like that. That's one vector, okay? That's the kind of thing a cooldown period is going to help you with. Um then, you've got known vulnerabilities, okay? So, this is kind of like legacy security. Um yeah, people are often like, "Oh, see you know, it's CVE is kind of the unit of this, right?"

Common vulnerability and exposure. People are like, "CVEs are are CVEs are Yeah, a lot of CVEs are basically noise, but X percentage are actually important, you know? They actually need to remediate. The rest make The rest noise makes it harder to deal with the ones you actually need to deal with. And because they're known vulnerabilities, you have to fix them because they're in a database, right? Like someone could just be like, "Go find Here, look at your software, look at the database, and combine those two things together, right?" So, it's actually very important to remediate known vulnerabilities. We now, even more importantly, have this realm of unknown vulnerabilities or like uh zero days, okay? So, this is becoming more important with the rise of frontier models that have {quote} the juice, like Mythos, um ChatGPT now has Daybreak, um but Astra, also very capable in this in this way. Um basically, frontier models are very good at chaining together lots of smaller vulnerabilities to make exploits, okay? And And so, that realm of zero days is very important. Um then, we have these emerging layers or you could also think of them as like side targets, soft targets, side doors that attackers are entering in through, okay?

Used to be they'd come to the front door and be like, "No admittance. Our production code is very secure." Now, attackers kind of know, "Hey, actually, let's hit your CI/CD cuz that's No one's really [snorts] looking at it that much. It's misconfigured. It's running a lot of dependencies and stuff in there. Uh let's hit the developer workstation cuz the developer is pulling things to prepare. You have a cooldown period, but does your developer workstation's laptop have a cool down period? No. So, let's attack the developer laptop. Let's attack the new agentic layer. It's emerging. It's changing a lot. People don't even know what's going on. And you can poison the models. So, the the that's these new uh side doors that attackers are very conscious are the soft targets. And And the last year has shown that. Mac OS stealers, all this kind of stuff. So, this is kind of how Chainguard's thinking about it. And then we'll dial in on I'll try to also combine with more general ways that these things can be addressed. Okay? For zero days, we have um we've I I'll actually get to Athena last. Um so, our classic um uh a classic sort of thing that we've been providing for a long time is um containers that have no known vulnerability. So, if you migrate onto them, they basically don't have any vulnerabilities at all. I'll talk more about that in a second. Um for malware, we have our Chainguard libraries. We build from source.

Um you pull from us instead of from PyPI, NPM, Maven Central. And makes a lot of attack structurally impossible. Um so, basically replacement repositories. Um for zero days, we've been working with um major banks um and other uh tech companies. And uh we have a large consortium going to Athena's a consortial solution. So, basically we we do have a clearinghouse where we um take uh uh known zero days and then we provide them back to people within the consortium. Um and then we also have our new uh Chainguard actions and Chainguard skill uh Chainguard agent skills that are hardened uh versions of CICD actions or agent skills um that you can kind of drop in. Um in terms of the the sort of that area that I was talking about, that CVEs area, that sort of like legacy area that you still can't ignore, uh the big problem is this like iceberg problem, right? Is that you have to run it in some infra, but that infra is um Um, most of it is beyond your control.

Um, so that basically means that like say 2% of the code that you run is your application code. That's made up number, but I would say it's actually not a bad made up number. It's actually probably pretty close. Could be 1%. Um, is actually your code that your engineers wrote and they can introduce problems in that. That's not We do software supply chain security. Everything else is kind of more our area. So, if you can don't control the OS, um, the application layer, the language runtime, that those are the layers that most of the vulnerabilities are on, okay? And I call this the uh, my boss told me to fix a Ubuntu problem. Um, and it basically is like, well, okay, now we somehow have to fix all these things in systems that we don't control. The way that Tanker thinks about this is that we actually started over with a new distribution of Linux that is called Wolfi and it's basically just completely, um, you know, from the built from the ground up to be to solve this problem. And we may mostly do that by having it be a rolling distro. So, there's kind of like not much of an upstream beyond You're basically running on upstream maintainers, um, code, okay? And we pull in very quickly. Things get pulled in. Images get rebuilt, uh, one to four times a day.

Um, so it's a very different way of approach. We don't sit on stable versions, okay? It's a very different approach. We also try to be minimal and do some other things. Minimal also helps with zero days. >> [snorts] >> This is our SLA. So, note 30 16 90. It's, uh, 1 7 14. One day for a known exploited vulnerability, seven days for criticals, and then 14 days for everything else. So, the 14 for a lot of that 14, that's stuff that no team would ever fix, right? Like a lot on a lot of teams, mediums and lows would never be triaged in under any circumstances. So, um, what else do do we think is really important? This is not a product. This is, uh, something we care about. Um, Salsa and Sigstore, um, basically we think you should sign everything. Whenever an artifact is created, sign it and then pass it off, then check that signature on the other end, okay? That's And what you can use for this for this stack is Sigstore is very useful for this, okay? Um and then Salsa is basically like attach provenance of the artifact, cryptographically sign it, and then check it on any other internal pipeline that you have, okay? So, it's a way of doing like a zero-trust pipelines that's very effective. It's very good in this current environment. A lot of these attacks are basically pipe different kinds of pipeline attacks, messing with your um your your tags, your download pages, um your depths, uh anything earlier between source and where your users get stuff. Um libraries is basically uh we take And this is a new mechanism, so

it's a little hard to it's a little hard sometimes to for people to be like, "Wait, what's the point of this?" Basically, we take clean clean source. Most attacks are after source and then between the source and the distribution of the artifact. So, we take the source, we airlift it out of the maintainers kind of like, you know, situation with their distribution page and their CI/CD and everything. We airlift it out, little helicopter. We put it into our system, we build it all, and then we pass it on to you, okay? So, it's built from source. We also do scanning, another kinds of remediation triage, you know, uh to, you know, beyond that, but you know, the built-from-source is a kind of unique mechanism that we do. And then I guess Ubuntu's not updating. >> [laughter] >> Maybe I should be running on Wolfi. And um and then see our ChainGuard Actions, and we're thinking about more in this space now, but basically trying to remediate this this problem of CI/CD is being this this one of the main soft targets in the modern um environment, okay? So, ChainGuard You can pull You can try it out. It's it's free to try it out.

ChainGuard Actions, um you can drop them in for GitHub Actions. Um we'll also be thinking about other ways where, you know, I think I don't know what exactly what I'm allowed to say. We have Assemble London going on, so you can tune into that this week, okay? Um this is something I've been working a lot on myself, too, is the Chingford agent skills. So, Chingford agent skills is basically we take upstream third-party skills, um we harden them in our pipeline, and then we pass them on to you. Uh so, it goes through a whole bunch of deterministic and non- and in-agentic steps. I actually just released a weird little video with like, you know, like Machinima kind of Minecraft-esque that will kind of explain how the pipeline works. If you want to check that out on the R YouTube channel. But, um but yeah, so, you know, and a lot of agent skills like my own in um recent uh analysis is something like 5% of agent skills on upstream repositories are actively malicious. Um earlier in the year, I think it was like something like 20%, though I think it's changed a little bit. And then another like 30 to 40% have serious security issues or implications um that make them insecure even though they're not intended to be malicious. Um we also can in your own organization, you can upload um uh skills to be hardened and then discovered by folks within your own organization. Um so, it's basically a quick and dirty, you know, a 20 minutes is kind of the new lightning talk, right? And I'm trying to cover a lot of ground. Um hopefully we can maybe answer a question or two. Um but we have the open source finance forum forum coming

up. It's funny when I just just now was looking at the slide and it said open source fiance forum. So, that's not a kind of fun, actually. We should have the open source fiance forum. But, um but it's the open source finance forum on November 5th. Um that's with Dan Lawrence, our CEO, and uh leaders who are leaders from the Athena consortium. Uh check that out. That's the QR code there. And then hope hopefully it works. And then um this is our fi If you're I know there's a lot of finance folks in the room, this is QR code is to our like start here if you're in finance page. Um come do check out our our in um in It's booth 23. We're behind a big column, so we're really hard to see. So, just look behind all the columns in the big expo room and you should be able to find us. And then my name's Patrick Smith. Uh this is my LinkedIn and this is Wolfie. Wolfie is the smallest octopus in the world. He's really tiny. He's minimal. Being minimal helps with zero days. Uh it's he's the size of a of a fingernail. There's not that many videos of him. This is one of the few that are online, but he you know, I'll leave him up for a second and then hopefully we have time for just a couple questions, but thank you.

>> [applause]