0:03We're going to talk to you about how we think about eugenic threat intel and threat response at Cloudflare here. So I think one of the top areas of, Javier and I wake up every day thinking to ourselves, what are we doing with humans that we shouldn't be doing with humans? And how can we make cyber response operations more effective with automation and AI. And so throughout this presentation, we'll cover kind of a couple different areas that Javier will get into. But at the end of the day, we're always thinking to ourselves, like, how do we get faster? How do we make better decisions? How do we become more deterministic in the way we think and create a more regimented playbook in order to have the right response and not make errors? And that's a lot of the focus of this presentation.

0:51Yeah. And so the goal is to have a tier one workforce that we can offload so that our analysts become tier two and tier three. So today we're gonna talk about three agents that we kind of have and they kind of play a different roles in our ecosystem. The first one is a threat intel agent that's responsible for handling requests and leads and then outputs reports. The second one is a managed defense agent that handles our SOC alerts and generates WAF rules and triages things. And the last one's a vulnerability discovery agent that goes through code and outputs WAF rules. So the first agent is our threat intelligence agent. And this one's, there's nothing magical about it really at the base. It's a standard flu agent deployed in a worker. And it uses skills and tools to kind of triage these requests.

1:38The magic is that we kind of have put all of our intelligence into these skill files. And as we get requests, we try to, we use these skills and tools to give us. response and we keep updating these skills and tools as we go through these responses. If something wasn't done correctly, we then update the skills and tools. And overall, it's kind of like putting all our collective knowledge into these things so that we can answer these responses. Now, this is interesting because this is a little bit of an open-ended investigator. It's not just something that we're very restrictive on. It can kind of scour the web for all these leads and pull in to kind of output some context or some information about the lead that we have.

2:25So I would say like, this is one of like the areas that's super interesting, which is this of all the agents we'll discuss today is the most abstract in that we have data we look at with the agent, but we don't necessarily always know what we're going to find. And so as something I always worry about, and I think Javier worries about a lot is how do we ensure determinism and how do we ensure we don't. just get garbage input to garbage output and model hallucination things just making up results that fit a narrative because that's always our concern. And all of our output for these things we try to cite it in concrete evidence whether it's a URL or raw query that we kind of got we try to tie it to the output so that we can gain confidence that there was no hallucination in the process. But again this is a very basic one.

3:11Our next one for manage the fact as a SOC one that handles alerts. And we have to be a little bit more prescribed on how we deal with this agent because it's going to generate WAF rules. This is going to actually cause actioning in your environment. And because of that, it's not open-ended like the first one. It's a little bit more restrictive. So all the recon we do is via code. And we do recon to pull like customer telemetry, global telemetry, threat intel cases. And we do this because we want to to be able to recreate it, um, recreate any of our analysis, um, in the beginning when we're doing collection. After we do collection, we want to be able to either accept the alert or reject it. And to do this, we go through a series of clef logic for each parts of our recon saying, does this recon support the alert or does it reject the alert?

4:00And at this part, we can actually dismiss alert or let it go on to the next phase. And we choose clef, a decision model, or Clef is Cloudflare's version of Jeff. We choose Clef because it's a little bit more, or it's less hallucinogenic. It's more deterministic so that we can create it. We're doing an automatic action. We need to be able to reproduce this when something goes wrong.

4:30After we accept this alert, this will go on to a more traditional agent that then interprets our facts. It interprets the recon, the Clef output, and it will... suggest a WAF rule that is then validated against the customer network. Is this going to disrupt their domain? Is this going to take care of the attack? And that is put in logging mode in the customer's account. And the SOC analyst can either accept or deny it, like a pull request, to take an action in the account. So again, we're being a little bit more prescriptive and how we're doing this because the risk is a little bit... higher we're taking action we're not just doing an open investigation and importantly um that agent like we built that in just what was it like two three four weeks yeah of work that entire workflow we just showed you was basically about four weeks work we couldn't do that without the power of the cloudflare developer stack so we're able to do rapid iteration with the cloudflare developer stack with the tools that you all have access to today through the developer platform and those tools have really changed our ability to build things in rapid succession.

5:39Well, not just that. The first iteration of it was almost the same kind of agent that we had in the first one. But as Clef and Jeff came out, all of a sudden we have to pivot to something that's a little bit more deterministic that helps us. And we're still making these pivots within the ecosystem to kind of improve and be more effective. And so again, in the beginning, we're very focused on getting all of our facts, concrete, doing our recon via code, moving on to the next step and using... uh clef and jeff to be a little bit more reproducible in our automatic action and then we pass all that to the agent for interpretation the next one is our vulnerability scanner so our vulnerability scanner a lot of people have been making these agentic vulnerability scanners they look at the code they try to find vulnerabilities and they output 4 000 vulnerabilities and now it's on the engineering team to like figure out how which of these 4 000 we need to we need to deal with and the agent does its best to like evaluate it but some it's just going off static code and i don't know if anyone's ever done malware analysis before but and analyze like a windows executable when you look at it statically like the disassembly or the decompiled you're kind of you kind of understand what's going on but it's not until you introduce dynamic content you you kind of see the request that it's handling you see the requests that are going out that you really understand what the code's doing and so for this approach we do the same thing cloudflare proxies a lot of our customers don't And then when we have the code, we're able to not just do the vulnerability scan, but we're able to take context.

7:07We're able to take their web traffic and see where the hot paths are. Where are the users going? How are the users interacting with the service? How are other services interacting with the service? And so instead of just pulling dependency modules for pip installed things or node installed things, we're actually pulling the way it's being used so that we can have a better evaluation. Not only that. We're using WAF rules. We're using the WAF rules because this lets us know where the attackers are targeting us. This lets you know where the attackers are near misses to some of the vulnerabilities we might find. And these might be more critical than others if we were just looking at it statically. Now, this harness I won't dive into too much because Dan Jones is going to talk about it tomorrow. But each one of these steps is a subagent that can be configured with different types of models based on how computationally expensive or how.

7:57off when we call these things. But out comes vulnerabilities with a priority. Again, not just based on code, but based on your network traffic. And also what that comes with are suggested WAF rules so that we can apply virtual patches in your tenant instead of just waiting for engineers to patch these 4 ,000 vulnerabilities. Security can be empowered to put these blocks in place to kind of protect yourself now rather than waiting for these cycles later.

8:25And again, this is just an overview again of we use the paths to find the hot paths. We identify the routes. We use a network context, WAF context to prioritize our vulnerabilities. And then again, for the source code validation, it's kind of like the vulnerability scanners that you've seen, the agentic -backed ones. It's bounded by code investigation. We get the vulnerabilities. We validate against the code. And then we give remediations.

8:52So everyone. making agents right everyone's making agents so we're all doing it in different ways and we've kind of had some like lessons learned i feel like that's the the biggest thing that's kind of happened and so the agents we've picked are very scoped we don't have one agent solve everything and that's done purposely because we don't want an agent just to like different agents have different risk in different scenarios again the threat intel one low risk you're just pulling an investigation you're just pulling in information the vd are the vulnerability discovery one and the SOC one, more risk. You got to be a little bit more stringent in how it's going. Also, sometimes the input might be inherently malicious. So in that case, you can be a little bit more aggressive with your agents. We're very big on evidence over instructions. Again, we don't want to tell the agent how to do all this investigation. Sometimes we know the facts we need to collect and we should do that via code if we can.

9:39Everything we output, we should validate. All of the stuff that the models output should be brought back. and have pointers to where it got that information for tracking. And we should always save input and output, all the recon and thinking, because this is going to help us later on when we're trying to change models in and out. And it's going to help us if a WAF was suggested that maybe didn't align with how we felt. And we need to be able to look back and see how it's different. And on the point of model changing, right, it is empirically critical, right, like as cybersecurity professionals, that we have access to everything. model there is. We need to be able to see like how a Chinese model might think about a problem, how a model from Europe might think about a problem, how a model from somewhere else might think about a problem.

10:29Because you have to understand like we have threat actors that have access to those models. They may use that model against you. If your company is afraid to use this Chinese model to test how your business is going to perform, you may get act by that Chinese model. And so we're seeing this throughout. like the fortune 500 where there's a lot of companies that are like well we're scared of the chinese models it's like well that's not right the right answer we need to be able to use all the models and we need to be able to change which model we're calling over time for cost reasons possibly for um safety reasons for just understanding attack surface of those different models because each one does think differently and does have kind of greater strengths and weaknesses and there's also cost efficiencies do you want to talk briefly about like how we about cost efficiency that's actually like a really big thing because like every time again it's always fun in games when you're making models until all of a sudden you're constrained to a budget or you're constrained to resources and we've we bumped up against that a lot we process like 60 ,000 alerts a day and it kind of turns into this traditional problem we used to be developers we used to be developers some of us are still developers some of us are prompters but used to write code in Python right it processes the 10,000 requests Everything's fine and dandy.

11:43And then you get to like 100 million requests. You're like, oh, Python doesn't work. I have to move to C or Rust. And the same thing applies to like these agents and the models we use. So in order to kind of deal with that, we again have to store inputs and outputs in our recon so that we can evaluate and change things as these models change. We have to be able to do benchmarks against them. And so, you know, I mentioned earlier that we went from like a frontier model or like Kimmy or GLM, and then Clef came out and we... we immediately wanted to start switching over to the cheaper model, the quicker model, but we had to do it with confidence. We had to make sure that we could run a score. Again, we can evaluate and see how many changed and evaluate the output. And not only that, when you're hitting these resources, sometimes you have to have a fallback model. You need to fall back to a different model because the other ones might be out of resources.

12:30And you have to be able to do that with confidence. And you have to be able to do things like asynchronous comps, you know, like maybe something this... alert doesn't need to be triaged real time maybe we can offload it to maybe when it's a little bit lower resources or lower lower expense maybe it's at night you're able to offload all the requests maybe talk just briefly about like when we talk about model fallback right like yeah this model becomes available how do we switch to the other another model without and with while ensuring that our efficacy does not change yeah so i mean that goes back to the evaluation of inputs and outputs have we save all of them we can run them against the various models that we have and see which ones are comparable for the step that we're dealing with so each one of those steps in the the vulnerability scanner we can we've compared the models ahead of time to say like okay we can fall back from like a daybreak blue model to a kimmy model on this specific step and we don't lose much efficacy and then with like ai gateway we're able to put the model and just simply have a fallback model if it fails the exponential exponential back off it just falls back to that model and we're a little bit more confident that like this one's not going to perform worse and put us in a bad situation um and then last thing is like with that ai gateway we're already we're getting caching so if you're doing the same request you're getting caching and you're we're able to kind of like again reassess our models to kind of like do what's what's optimal and sometimes it's not even going from llm to clef or whatever new versions out there in the future.

14:02Sometimes it's going back to code and making sure our code can be equivalent to what's going on with some of these LLMs. So we're in a good spot right now. Again, we built this tier one workforce. Our threat intel agent is capable of responding to 80 % of our requests and leads without analyst input. Our analysts review it. They check the siting. They check the data behind it. It looks good. We move on. If it doesn't, they add the information into the skill. file so that we know how to where to pull the information or how to evaluate it given certain contexts. Our alerts that we're dismissing in SOC, we can do under 10 seconds. And if it does make it past that phase and into like our in the SOC hands, our SOC analyst hands, they can do it within three minutes.

14:47Again, 90 % of these actions are done via these agents and from end to end. So I guess the big takeaway is for all of this are when you need to be deterministic and you need to define facts because it's important for auditing it's important for changing models it's it's always good to have that as like your base like everyone should do that it's always fun making the model and the agent like day one day two but there's going to be month three month six and you're going to have to maintain these things the second takeaway is again make it audible make if something happens and the output's done you need to be able to cite the sources And the last one is, again, we're trying to build this tier one source. We're trying to figure out where humans are doing things that they probably shouldn't be doing things because we can automate that.

15:34And that's not just to replace the humans. That's just best to turn the tier one work into automatable work, just like we used to do with code. And this is an important point. We actually had people in our org that were like, oh, you guys are trying to replace us. And it's like, we actually have more work than we do people. So we actually don't need to replace you. We actually just want you to do other work that's more meaningful than the. like false positive, false positive, false positive, false positive. Like this is not an effective use of your time. I actually need you to use your brain cells to analyze more complex attacks than marking things as false positives. So like we're trying to move people to do the right work to have a greater impact as opposed to doing just trivial work that can be done automatically. Yeah. And, and, and that's a good point because we get, there's a lot of ABTs out there.

16:20There's a lot of intersection. cloud first telemetry and these attacks and it overwhelms us you know we have a 60 000 alerts we're not going to be able to go through all of those and so falling back to this stuff really helps us to kind of like level up everyone um and so that they can do like the important work never once in my career have i thought to myself wow if only we had less people there's no work to be done like that just doesn't happen in cyber security and never will happen right there's as good as like ai is there'll always be cyber security issues that come up and you know having the humans being capable of responding to those issues it is the best right and we're just trying to optimize and become more efficient yeah so moving forward we're going to continue like our mission to add ai observability into all of our and into all of our agents not just these three but the others so that we can continue to adapt as new models are coming out and we're going to keep exploring different ways to scale is it using slms is it using fine-tuned model you want to talk about SLMs are in case people don't know.

17:20Small language models, so just not the large ones, just smaller. And maybe talk about also how we think about SLMs and where they might be more applicable. So we're looking at using SLMs probably more in the, again, in that vulnerability discovery region where something's being called like thousands or tens of thousands of times. You don't want to waste like paid resources on that. You want to be, or you want to use a general model. You want to use something more specific. tailored towards that so just we can kind of scale and be more effective with that and i think the slms are interesting because it also like javier's earlier point in one of the earlier slides was about um how we want to have determinism right and so like an slm might be more deterministic in how it inputs or outputs data because it's trained on less volume of data so we might it's just like an additional determinism that we can add to a workflow that gives us more confidence and how to be confident to block something or take out some sort of other evasive action like wiping out someone's hard drive obviously if we're going to do that we want to be very certain before we just start erasing someone's device correct and so that that's how we're using agents at cloudflare that's how we're using them across to kind of like level up our workforce and uh yeah always interested in what everyone else is doing with agents in the same scenario thank you