0:03So, we're going to start with a very simple statement. The number of people using your application can remain the same, but the number of requests can increase exponentially. The work that goes to our origins can grow dramatically. Now, just for a definition, what do we mean when we use the word origin? We're talking about the servers and applications. applications that sit behind Cloudflare. A person may ask one question, they may expect one answer, but the systems that process the response may exceed dramatically. They might check a catalog, retrieve documents, check live information, and even slow down before that answer even appears.

0:49So again, what are we talking about? The user sees one interaction, the origin could see an entire My name is Chad Nachieff, and I'm a product manager here at Cloudflare, and we're going to be talking about a product called SmartShield. We're going to be focusing specifically on how we can protect existing infrastructure in the age of agents. And to do this, we're going to walk through the life of a request as it enters Cloudflare's system and all the different paths it could go, again, specific to agentic requests.

1:29first a case study the fair price group is based in singapore they serve more than a million customers a day and they've built shopping agents that help people plan and make their meals now picture somebody opening up their app they ask the fair price group hey at 6 p .m what can i cook tonight that's a very simple question But the agent may need to search the product catalog, check current inventory, retrieve a recipe, check if something is even in stock, and then assemble a useful answer. Now, if any one of those systems is slow, what is the agent going to do? It's going to retry. And now that one user question has now created work across several systems.

2:15Now, let's think about our environments. If our traffic doubled in... next month, and because of all these agentic requests, where would we notice it first? Would our database slow down? Would connections fill up? Would we see the effect on our cloud bill? There's also one other interesting distinction when it comes to looking at agents.

2:45The pacing between... between a machine demand and human demand. Software doesn't have the natural pacing of a human. If I read something, I probably think about it, I might get distracted, and only then I'll make another request. But a machine can keep retrieving content without stopping. It can also decide that a request has failed while the origin is still working to produce a result. Now, Cloudflare traffic data shows how much this traffic has increased. In a single year, GPT bot, grown over 300%. ChatGPT user, close to 3,000%. And perplexity, 157,000%. Now, these numbers are not meant to be forecast for applications.

3:35Automated traffic includes much more than AI. But the important thing to understand here is that the traffic mix can change very, very quickly. It gets scary, though, when the origin begins to slow down. The original request remains open. The origin is still working and maybe opens up new connections, new workers. There could be a timeout upstream. Another request arrives, even though the first one is still going. And all of this extra work makes the origin slower. This feedback loop is what turns an increase in traffic into an incident. If you heard Polly Lane... pitched yesterday, we learned that on -calls are now spending more than about 37 % of their time managing incidents up from 33 could go up to 50%. This is the problem.

4:23What can we do?

4:31There are three main jobs that can take place when we're looking to control agentic traffic.

4:40stopping repeated origin work, moving requests efficiently, and then controlling access to the origin. Now, many organizations and companies have these jobs, but they're separate. They're all owned by different layers. Caching determines whether the request needs to reach the origin at all. Cloudflare has smart-tiered cache, regional-tiered cache, and cache reserve. The whole goal is to stop traffic to limit. traffic from going to the origin. But what happens when traffic actually does need to go to the origin when the cache can't help? We want those requests to be routed efficiently. We want them to go on the fastest possible path. Connection reuse helps us save setup time. That helps the network. Now, let's say we do get to the origin.

5:26What's going to protect our origin from the influx of requests? Having dedicated CDN. egress IPs. It protects the boundary. Health checks are constantly monitoring the origin. Now, of course, Cloudflare DDoS protection happens at the edge, but right now we're focused on the small amount of requests, the smaller amount of requests that makes it past that, and it's actually a real request going to the origin. Now, these are all different product names that are controls, but what we want to focus on is the job that they do. So now let's look at the path, and the different decisions that it goes through.

6:15So looking at this diagram from left to right, a request arrives at Cloudflare, and the best outcome is that Cloudflare already has a response that's safe to reuse. Now, that saved response might be in a cache near the user in one of the shared cache layers. It could be in cache reserve. And if Cloudflare finds it, great, we can answer the agentic request right away. The application doesn't need to run. We don't touch the origin. But there are some requests that cannot stop at the edge. They need to go all the way to the origin. Think about purchases, checks on a customer account, an inventory check. How should it travel? Well, we want it to be routed through the most efficient path possible. Argo.

7:01improves the network path, connection reuse, avoids building a new secure connection every time Cloudflare needs to talk to the origin. And then once we get routed with Argo, we get to the origin. When it reaches that final boundary, the firewall can recognize a small and dedicated set of Cloudflare IP addresses rather than allow listing the broad set of Cloudflare IPs. Now, the thing to remember here is all of these things, all these layers. affect the same request. We wanna stop and answer a request as early as we can. If we can't stop it, we wanna route it efficiently and quickly and make the remainder of the journey efficient. So now let's dive into each of these jobs. The first job, caching.

7:58so now what we're looking at here is what we call a tiered cache tiered cache solves a problem that comes from having a very large edge network we have locations close to users around the world that's great for performance but that also means each location can have an incomplete view of what we already serve to users imagine the same document becomes popular among different users in different cities around the world if local cache misses independently, each of those locations is going to ask the origin for an update. What do we see? The origin ends up repeating the same identical work. What tiered cache does, what we see on the right, is cache layers so this allows us to check a cache before a request even goes to the origin.

8:49This allows many Cloud4Locations to share what's already been fetched. So the takeaway here, more Cloudflare locations get a chance to share the same saved response before the origin gets involved. Less work to the origin. But you might have noticed there was one other control we mentioned under caching, something called cache reserve. What's that? We know that every o, c, c, c, e, c, c, c is soul, there is a experi cash has limited space, right? It'll hold a document or object for a little bit. If it's not requested, it gets evicted. That is okay for a lot of human traffic, but in the age of agentic requests, agents make this a lot more expensive because they're often searching a large collection of documents.

9:40Maybe they're only searching every few days, which is just the right amount of time for an object to go into cash, get evicted, and then be brought back. again. What does cache reserve do? It's a deeper storage layer. It's a storage layer that holds content longer. So if there's a local cache miss, we can go to cache reserve, which is R2 backed. It still keeps the origin as a source of truth, but what just happened? We don't have to go all the way back to the origin because we have another level of caching available. Now, not every object can belong. in this R2-based cash reserve. There are some eligibility requirements, but the entire goal here is to prevent requests from going all the way back to the origin.

10:28Yeah. Does top here get the request set back on these stairs?

10:36Good question. So cash reserve is not included by default for all customers. If someone is signed up for it, then it will be used. Is that your question? Yeah, of course.

10:53Now, let's imagine we go check local cash. We don't, it's not there. We go to cash reserve. It's not there either. The request really has to go to the origin. So now we want to make that request as efficient as possible.

11:12We want to make that trip. as least wasteful as possible, more efficient. Now in our minds, we might have, we might think there's an obvious route across the internet, a fastest path to the origin, but that's very dynamic. There could be a fiber cut, there could be congestion somewhere, there could be a network outage. What Argo does is constantly send out probes from each of our colos to origin locations around the world to always have a mapping of what the fastest path is. Just because a previous path was taken yesterday, it doesn't mean it's still efficient today because congestions and failures change constantly. So what Argo smart routing does is send out probes, regularly evaluate times. We might even carry the request along or through Cloudflare's own network and exit closer to the origin.

12:02So again, the request, what's the fastest way it can get to the origin for the current situation? Now, there's also work involved in setting up a connection. There's TCP, right, establishes a connection, TLS, the necessary security handshakes. That setup is necessary, but imagine if it has to be done every single time. That's expensive. What connection reuse does is it keeps secure connections open so Cloudflare can send more than one request across that same connection. Now, across customers, we've seen an average improvement of 30 % fewer origin connection, fewer origin connections. Now, what does that mean? This means we now have more connections open for increased traffic, which is what we're seeing from agents.

12:50Now, imagine we've gotten to the origin, but now we've got a lot of requests coming to the origin. How do we make sure that, how do we make sure they're secure, they're safe, and those are the actual requests we want. Now, we get to the third job.

13:12if you use cloudflare you're likely allow listing a long list of ip addresses because that's when you know and trust that cloudflare will talk to you some but those ip addresses those shared ip addresses are talking to every single customer what dcei does in dedicated seating egress ips it gives you a small subset of IP addresses that are just for you. So instead of allow listing hundreds of thousands of IP addresses, that scope becomes much, much smaller and those IP addresses only talk to your origin, doesn't go to any other client. Now you might think, hold up, hold up. If you're saying there's a dedicated egress IP, doesn't that mean it's locked to a certain colo?

13:58What if that data center goes down? Well? the benefit is there are multiple dedicated CDN egress IPs, and they are placed at different locations. So if one is not available, the alternative POP can carry the full weight of that traffic. Your origin firewall is now checking that list of dedicated CDN egress IPs, verifies it as safe, and it moves through. Now, this isn't a replacement for things like MTLS. Very important for security. But what dedicated seed in our egress IPs allows you to do is to have a verified source and you know Cloudflare is only going to talk to my origin from this range of IP addresses.

14:46On the bottom section, we see a bit about health checks. What health checks do is monitor the origin from Cloudflare's perspective. This allows us to see crucial information. It doesn't make any changes if something is unhealthy, but it allows monitoring and alerting based on our view of the origin.

15:08How do we know? Let's say we use all three of these levers, right? We're caching more, we're sending requests along a faster, more efficient path, and we're making sure these requests are secure. How do we know if this actually helps?

15:24The telemetry for... cacheable and dynamic paths are different we know this traffic behaves differently cacheable requests will often stop at the edge or a little bit beyond it if we're going to cache reserve we want to know how much work disappeared before it actually goes to the origin before it goes to the application so what are some metrics that'll help looking at cache status age how many fewer origin requests are there how many bytes did we avoid sending back along the wire. And we can actually look at this, split it by route, split it by traffic type. Just because one entire zone shows as healthy, there still might be some weak points. So these metrics help diagnose that. What about something like a dynamic traffic?

16:11This is traffic that we've purposefully allowed and we really want to get to the origin. For things like this, we care about how long the origin took to begin responding. We're connected new or were they reused? How many requests timed out? Did errors increase? And so when we look at our dashboards, when we're monitoring, keep in mind that looking at cash flow traffic and dynamic traffic should have different telemetry. Let's look at a brief customer example. Docker.

16:48Now, Docker is a really helpful example because to start with, they already looked really, really good. Docker Hub serves more than 500 million downloads a day and 97% of those downloads were already being served from cash. If you're doing a review and someone comes to you and says, hey, my cash hit ratio is 97%, we'd probably say, great, that's fantastic. But look how it got better because at Docker scale, the remaining... 3% is still expensive. That's a large stream of requests. Most of those requests require Docker to read back from storage and pay for egress costs on S3. What Cash Reserve allowed them to do is to move that hit ratio above 99%.

17:40A two percentage point change sounds pretty modest, but if we frame it the stream of requests reaching the origin fell by two -thirds. That sounds pretty impressive. It helps save on costs, and also Docker saw a similar reduction in S3 egress. So if we think about just cache hit ratio, is that enough? Not necessarily, because there's so much more that comes into play. So what have we covered this afternoon? We've covered three controls. goals we've covered how do we prevent requests from going to the origin if they don't need to go there if the requests do need to go to the origin how do they get there on the fastest path possible and once they do get to the origin how do we make sure that those are the safe secure requests we want because what's the whole goal our goal is to make sure that the origin only does work that only the origin can do if it can be repeated the origin doesn't need to do that.

18:42When we look at these lists of products, you might think, well, this is a lot of knobs to control. But not everything has to be enabled at once. For example, when you analyze your own traffic, if many Cloudflare locations repeatedly fetch the same content, tiered cache and cache reserve might be the most helpful. How do you know if cache reserve is specifically going to help you? Well, this is if less popular objects keep getting requested. They're just popular enough not to be in the cool kids club, but you still need them every once in a while. That's where cash reserve helps. Now, if traffic must reach the application, you have an application that does receive a lot of dynamic requests.

19:29That's where Argo smart routing and connection reuse can help. The opportunity there is in the network path and reducing the number of connections. And with DCI, dedicated CD and egress IPs, if we have a lot of requests going to the origin and we want to secure them and we want to guarantee that we're talking to Cloudflare, that's where we can help by controlling the boundary. So with agential requests, all applications are going to be flooded. How can we make the origin do the work it's meant to do? Just because we think we need more capacity, the answer isn't always through. more origins at the problem. It's making origins more efficient at what they already do. Again, my name is Chad Nachiep and thank you for joining us this afternoon.

20:16We'll be around having to take any questions after.