0:03I'm Emily Music. I am the Network Strategy PM here at Cloudflare. I'm joined by David Tuber, a Director of Product Infrastructure, also at Cloudflare. And we're here to talk to you today about how we grew Cloudflare's global network. We're going to focus a little bit on peering, why we've placed our pops where we did, traffic management, and if you're looking. you'll see some scary slides of things you hope you never have to deal with. So, first, Cloudflare's network. You've probably seen this slide a gajillion times, legally required for infrastructure presentations. We're in over 300 cities. We're all over the world. We've got a global backbone, but that's not how we started. So we thought it would be fun to talk to you a little bit today about what things were like 10, 15 years ago.

0:53We just celebrated our 16th birthday. It's a little look. is what you're in for. So a lot of you are familiar with how our network works, but just to remind you, we are an Anycast network. We are focused predominantly on getting as close to our users as possible. Breath is really important to us, and we run almost all of our services everywhere so that, like Stephanie said this morning in the keynote, no matter where you're located, you can get to what you need quickly.

1:21So in the early days... It may surprise you to know we actually didn't set out to build a CDN. We wanted to build a WAF, a powerful, performant WAF. So what we needed to do was filter traffic. We knew that we were going to need to do that on our own servers.

1:43And so becoming sort of a reverse proxy was sort of a prerequisite for us building a WAF. And a CDN was born. We wanted to be performant. We had to add caching. And so it wasn't our goal, but we needed it to get to our goal of security.

1:59So when we started out, we didn't have the, you know, Cloudflare money you see today. We knew that the public clouds were just not in the cards for us financially. And we realized really quickly that if we wanted to build our business, we couldn't sort of take on the fees associated with building on someone else's cloud. So what we decided. to do was take the plunge build our own dedicated edge network and this gave us more control over sort of the fees that we were paying in terms of you know bandwidth avoid compute markups all the fun things you see um so and we also knew that we couldn't really operate and buy all sort of the specialized appliances that we need so we really set out to offer what we could uh in terms of of software um we towards this commodity hardware approach instead of the specialized hardware.

2:52And yeah, we decided to run all of these sort of WAF, DNS, cache, DDoS that you know and love everywhere.

3:02So today, Stephanie said, we launched in five places. We very quickly tried to expand into these other ones. The approach originally was just to get into these major IXs. You might recognize some of these from today. day or long ago. In phase one, we just knew we needed to get where it was going to be easy to interconnect. We knew that there was going to be traffic there for us to serve. And so our initial goal was to get into the hubs. Phase two was getting us closer to the end user. So we have a program internally we call EPP program, Edge Partner Prop Program. And this is where we work with local ISP so that we can get to their network directly. We call locate in their facilities, and make sure that we can serve their networks and interconnect everything, and it gets us closer to the user.

3:53So I know what you're thinking. You just gave us the recipe. We buy commodity hardware. We get into hubs. We may be deploying some ISPs. Why wouldn't we just build a CDN, right? I'm not here to deter you, but...

4:16I don't know if you can see it, but it's under a lot of water there. This is a dramatic reading of one of my favorite emails. There is no security guard on premises. However, there is an aggressive dog that serves as an extra security measure. Whenever we visit the house, the landlord ensures his dog is kept inside before we can access the machine room. This adds an additional layer of security. Now, don't worry. We are a security first company these days. We do require at least three aggressive dogs to guard our infrastructure. And then my personal favorite, dear Cloudflare team, the power outage was caused by a squirrel damaging the fuse, resulting in a power interruption. Resolution, the damaged fuse was replaced. We can confirm the power supply to the rack is now stable and operational.

5:07Squirrel's fine. Don't worry about it. um perfect so now i'm going to pass it off to david tuber he's going to talk to you a little bit about how we are resilient thanks emily so as you can see our network is very heterogeneous so we need to build our network to make sure that no matter where our pops are deployed no matter the network conditions or the number of dogs or the squirrel situation that like our network And so a lot of this is actually not done in hardware, right? Like you can have as many, you know, routers and interconnections, but if the squirrel gets into a data center or a data center catches on fire or something happens, then we can't deliver your traffic. So what do we do?

5:54Well, we got to build resiliently, but also there's a whole bunch of other stuff besides just resilience that we have to care about, right? Let me ask a quick question here. How many here? people here are running AI workloads? Hands up, cool. How many people here are running zero trust SASE workloads? Oh, cool, some of you guys. How many people here are running CDN? Cool, that is a pretty even mix and all of those look completely different on the network and we need to be able to handle those. We need to be able to handle stuff like geopolitical fragmentation. Wesley talked about it at the keynote. I need my to be decrypted here and only here, not over there. How do we handle that? We need to be able to handle supply chain constraints.

6:40Fun fact, who here has tried to buy RAM in the last 12 months? It's pretty hard, right? So imagine buying a lot of RAM for a lot of computers around the world. It's also pretty hard to ship to a bunch of different places like the Middle East. It's a little bit of a challenge right now. So we need to be able to architect our network to be able to handle all of these problems and also the ones that are... not on this list and we'll figure those out when they show up. So one of the ways that we solve this problem is just by architecting our data centers differently. Who here runs compute in US East 1? Show of hands. That is actually far less people than I thought. US East 2. That's actually slightly more. So those sites are pretty important. It would be really bad if something happened to Ashburn, Virginia. It would be really bad if something happened to a Cloudflare.

7:29center in ashburn virginia so let's go and build our data centers differently let's go make sure that each section of computers each group of computers can fail independently of one another let's make sure that we have additional physical redundancies we group our computers into logical groups of capacity called colos and we have many of them in a point of presence so we call them multi -color pops this ensures that if something happens in ashburn virginia we don't take the entire pop down that means we can still surf traffic nashburn virginia even if, you know, a squirrel gets into a power generator, although I would imagine that's probably less likely to happen in Ashburn than other places.

8:06We run this in all of those major cities, and I'm sure that if I asked you to name some of the most important cities on the Internet, you could do that. Ashburn, Frankfurt, Singapore, London, Amsterdam. I could go on, but we don't have that much time. And all of these are running Cloudflare machines. And as Emily pointed out, and as Stephanie pointed out, and anyone who... is probably even a talk who's telling you will point out all of these can run any cloudflare service all the time and that ensures you get single pass protection single pass performance single pass ai stuff um on every single machine and if any one of those machines fails it's okay we just move over to the next one um and that data center architecture allows our single point of failure to be as small as possible if we have fewer single if we have more things that can fail then there's more opportunities to serve your traffic. And that's thinking about things from a resilient level.

8:53And Matthew talked about how, and Matthew and Emily and everybody talked about how we do that. This is why. We want to make sure we can always serve you no matter where you are and no matter what's going on on the internet. And a lot goes on on the internet. For example, cables get cut. The North American fiber -seeking backhoe is a very hungry beast and will strike anywhere in North America and also in other places. like underwater, and that will cause disruptions on the internet. We need to be able to handle these shifts of traffic when they go from place to place because things just break on the internet. And to do that, we've built autonomous traffic management systems that can detect that there is a problem and move traffic somewhere else so that that traffic can be served.

9:39Our principle is we want to serve you as best as possible all the time. That doesn't always happen all the time, but we want to make sure that no matter what's going on, can serve you to the best of our abilities. And that combined with our resilient machine architecture and data center architectures allows you to be resilient. But it's not just about data centers. It's also about the network. You saw all of those connections and all of those long lines on the Cloudflare network map. We need to be able to make sure that for those paths, we can still make sure the traffic gets from point A to point B. So we have this product called Argo Smart Routing. And it's actually very simple. Who here is sort of Waze? I feel like Waze. It's Google Maps now for the younger people.

10:24Google Maps will tell you and Waze will tell you if there's a traffic jam or there's construction or there's an alternate path that will get you there faster. And Argo is essentially the same thing. If there's a problem on the internet somewhere and your traffic is gonna route through there, Argo will essentially tell our network, hey, there's actually a better path if you go this way. And so we'll send traffic that way. And we can detect a whole bunch of different things. We can detect latency increases. We can detect packet loss. And we combine all of that into a globally distributed route optimization engine that ensures that you're getting the best possible performance and the most reliable performance you can get based on the underlying infrastructure that we have. And Argo doesn't just make you more reliable. It also makes you a lot faster. We can make you up to 33% faster and sometimes even more depending on how your network and how you're, you know, cases and where your users are.

11:13And another combination, and we talked about fast, let's also talk about reliable. This is a nice fun little Greek mythology joke for those of you who either haven't seen Hadestown or didn't read mythology. Orpheus was actually the navigator of the Argo. And so Orpheus steers the ship to make sure that you get around the choppy water. So we have a metric internally that we like to call origin reachability, which is how often can Cloudflare reach your origin? And around 2020, that metric was about 99 .9%, which for those of you who run enterprise networks is probably not a very good number. We rolled out our first version of Orpheus in June of 20, June or July? July of 2021. And when we did that, we gained an entire nine.

12:00Who here has worked on enterprise networks before? That is pretty hard to do, right? Gaining an entire nine. So we gained an entire nine just by... this out and we're continuing to improve it and continuing to catch those edge cases in those new failure modes so that we can make sure that we steer around that choppy water so that jason can chop off medusa's head although it's not medusa correctly and all of this makes us not just reliable and fast from you know colo to colo from site to site but it makes us super fast on the internet in general um stephanie gave that net metric 95 of the internet net is within 50 milliseconds of us. That's a really good goal because it means that no matter where you are, you're getting the best possible performance. We're running workers and running applications and agents right next to you.

12:47And that's not like a really big deal if you're in San Francisco and the nearest data center is like five milliseconds away. But if you are, let's say, in South Africa or Jakarta, actually not Jakarta, but if you're in, you know, Bali or in Indonesia, you're in Africa or you're someplace very far away from where the major internet sites are, having a worker and having your stuff run super locally is actually a game changer. And we've heard from a lot of customers that this ability to be close to users is actually a difference maker when choosing a provider. And we're pretty fast. We're actually much closer to our users than anybody else. If you've seen the Cloudflare blog post during any Innovation Week, you've seen a post like this.

13:34where we talk about how fast we are. We talk about how close we get to our users and how that changes our last mile RTT. And we're the fastest network in 60% of the top 1,000 networks. And we can't get this way without our network strategy. Get close to our users as possible through either major data center, major interconnection points, or by embedding in last mile networks. That gets us literally that extra mile. So, summing up. Whether it's snow or sleet or rain or fire or dogs or squirrels, Cloudflare has got you. And we do this for every data center in all of our 400 data centers around the world to make sure that you get the best performance no matter where you are.

14:20And Emily said, you know, just in case you were thinking about building this yourself, like you can go build this yourself. Like none of this is actually anything wildly crazy. you can go and go rent data center space today. You can go get peering. You can go set up a network. You can go buy an ASN. You can go buy IP addresses. You can configure them. You can advertise them. You can negotiate with transit providers. And oh my gosh, that sounds incredibly, insanely hard. And it's so totally different than what you want to be doing. And so our hope is that you look at this and you say, this is hard because it is. But also at Cloudflare, this is our job. And so hopefully we do a pretty good job. good job. And the reason we know this is that data center fire. Literally, we were back up and serving traffic the next day, thanks to the hard work of our data center providers and also all of our network and infrastructure operators.

15:08So we've got a really good team that's working around the clock to make sure that you are getting the best performance you can. Thank you so much. Thank you