0:03and today we're going to talk in terms of like how we've been starting to sync how we're moving beyond IP addresses and specifically we'll dive into like some cryptographic primitive that we're thinking for honest bots before starting I want to ask the room who's operating the bots a bot in the room okay there's a couple of hands who is operating the Cloudflare workers who is running something like curl command like that there's even more hands and like it's always starting to get interesting because what's about is actually a bit fuzzy and so like what we would like is to have like some signal that people would be like willingly provide that would help us to like do some job and help them to like better identify themselves so um i'm tibo i've been working on web bottles for year and a half uh as one of like the main coordinates of draft and we're going to dive without further ado.

0:56So one of the main thing that we're trying to think is like how do we help honest bot identify themselves and the perception of like what is an honest bot, what is like friendly, what is unwanted is really up to origin and Clarflair tries to like convey the intent that the origin has given us in terms of what traffic they want to allow, what traffic they want to block, what traffic they want to rate limit or whatever policy they want to and so like the bots that go here we have like sometimes like really traffic that like you don't want you have some ai agents which maybe traffic that you want maybe traffic that you don't want maybe traffic that you do something else with and you have like some friendly bots and what you really want to help here is focusing on like the friendly bots how can we help to help them make sure that they can identify themselves in method that works for them and that works for us so historically a lot the detection mechanism and like that's what you have on graphic dashboard you rely on ip addresses or you rely on user agent and the ip address maybe because like you're seeing a lot of requests coming from a specific ip address and you say okay like this request it seems like there's a bit too many requests coming from like this ip address i'm looking at like an ip info table i'm looking at like google for instance um documentation and i can infer that this ip address comes from google maybe it comes from like a data center from like GCP, maybe it's associated with OpenAI or similarly we have like the user

2:26agent when we have the user agent which is said to like GPT bot, OpenAI publishes and advertises that like this is a user agent that they use and therefore you can like reasonably think like this is what OpenAI will use when making requests to you. Of course depending on the product each of the company may segment this type of like identifiers like you may have like different egress ranges for like google search or the fun type of crawlers that google uses similar with your user agent google has a whole documentation in terms of like which user agent that they would use and to be fair it's not specific to google cloudflare does the same we have a couple of like user agent that we use should be workers should be like remote browsers or should it even be like traffic that we pass back to the origin thing is this type of signal are quite brittle we can do better.

3:16So for instance, this is a list of IP address which is served on a more or less arbitrary HTTP page. These IP need to rotate because you sell them, because they are not available. There's a lot of management involved. Similarly, with user agent, they are spoofable. I mean, you know that Google agent is like Google bot. And because you know that, you can also set that same shield and that same header as part of your request. There are a couple of other reasons why this doesn't work, but really what we want is something which is protected against spoofing, but which also is easy to manage, to rotate, to extend, and works in an area where we have just a diverse set of participants and parties.

4:08So what we looked into is how can we introduce and leverage some of the knowledge we have on digital signatures. How can we do that for bots? And this way, when you have some bot which is unwanted, well, they won't sign anything, so you won't deal with that. However, when you have people that are willing to help you, similar to what Google does with sharing IP addresses, setting a specific user agent for OpenAI, etc., they're able to have the mean and the method to do so in a safe, secure way, and in a way you can understand them. And in this talk, we're going to dive exactly how we constructed an initial version of web models with HTTP message signatures, what are the properties we were looking for and how we can achieve that. So, as I mentioned, the way web models work is it relies on HTTP message signature.

4:56is an RFC and like it just more or less looks like that. So it's a set and a series of HTTP headers which are formatted in a specific way, which we're going to detail. It's a digital signature. which is carried through the HTTP header. It's using public key cryptography. This is what provides the property that we want, which is like security and authority. And it comes along with every request. So just by looking at the request, I have all the information I need that the other party, which is a client, the crawler, the bot, wanted to communicate with me.

5:32If we look a bit, it's actually like separated in three main components. We have the signature agent, which is a discovery hint, which is provided by the client. Similar to have the user agent attempted to convey what type of product was making the request. When it was Googlebot, you had a hint that it was Google. The signature agent conveys something from the client. In this case, it conveys a URL, like example .com, where the set of keys from the clients are hosted. The second set is all... is standardized in RFC 9421 for HTTP message signature and it's a signature input. It is like the set of parameters that we want the signatures to cover. We have a couple of sets, the authority means we want to sign the domain so that let's imagine I'm a bot, I'm making a request for example .com, I can sign example .com to guarantee that like this request is for example .com, this will of like this will prevent when I'm going to evil .com, evil .com re - using my signatures and impersonating me by replaying it.

6:36We have some timeline, we have some subprint, which is cryptographic details on that side, and we have a tag webbottles, such that when I use webbottles and if I have another application down the line, I already have scored my signature to the specific protocol. There are some parameters which are optional to protect a bit against replay, to have some better signatures, and finally we have the signature, which we will validate. the edge, Cloudflare does it automatically, but the nice property of like these HTTP message signatures is like it's also transitive. Cloudflare passes the signature down to the origin, and so if you have additional set of policies that you want to apply, you can do so.

7:18If I can go on the next slides. The main question is why do we have like these specific mechanism? Because it's great we have three headers and etc, but like are there the right one? Are they the one that like will allow? but that want to identify themselves, do so in a non-spoofable manner, in a secure manner, and in a way that they control it. The property we're after, where request authentication, we want to ensure that like when the request comes in, it's actually coming from the party which we think make the request. And this is why this request is signed proves that the sender holds the key. If we have and we see the same key ID, we know that like it's actually the same key. that made the request, and therefore the same parties that we have validated before. The second property we wanted is we wanted to have some sort of discovery.

8:06It's very inconvenient when you receive requests, and a lot of requests from a specific IP address, that you have to somehow infer that this IP comes from Google, this IP comes from OpenAI, this IP comes from data centers. And it's much more convenient if you have a way to somehow, some ways to discover the material that you... need in order to like make the claim and like have all the validation. In web bottles can be conveyed by like the signature agent, can be carried out of band, but still be on Google documentation or on Cloudflare documentation to provide. It's just like smooth line and streamlined throughout the protocol. One thing which is important through that is anyone can set, similar to the user agent, anyone can set the user agent and like host keys on the internet. So discovery is not trust, but it helps.

8:53build it as you carry on history with it. And finally, we want to have some sort of binding, so the key can be hosted at any domain. If I fetch the keys of Google, I could re -host the keys of Google on my domain. Same with Cloudflare workers, we could have the keys of Cloudflare workers hosted at one address and this key re-hosted elsewhere. And so what's important is that similar to what we said, discovery is not trust, it's not because I see certain keys on certain domain that like the keys are actually directly associated with the domain. That's what domain binding is doing us. When we have for instance the keys which are hosted on the well -known URL such as like well -known HTTP message signature directory we know that like the party that is in control of the domain has taken some special action in order to host a key at that place and this way we're able to provide some domain binding.

9:49Okay let's talk a bit in terms of like diagrams and interaction between the different parties. So we have the client, so in our case the clients are our friendly bot. And then we have the origin, it could be like your origin, be Cloudflare, the advantage here is like it works throughout intermediaries. So you have a client, makes one request, two requests, three requests, we can see it's like a client that like actively wants to be like identified, it like sets a signature agent, maybe it has like a custom user. that it sets etc it makes a lot of requests and like everything goes through and at some point because you have some strength on your infrastructure because you start looking at your logs a lot of events you start noticing that like there's an increase of traffic on your site and so what you do you can look at the signature agent you fetch the keys which the signature indicates and now that you have the key you can like validate the request signature with the keys The signature proves possession of the key because people in order to make a signature need to have the keys in the first place and therefore can apply their local policy.

10:56The next time that they will be able to receive a request, a signature or a signature agent, they already have the key. The key is already known and therefore they can still validate the request. The same key ID means like there's some continuity and therefore if I receive more requests down the line from the same. actor, I can start to build some history based on that actor. And again, I can apply some policy. It can be like some rate limiting, can be like allowing, can be blocking, can we make whatever policy I want based on the knowledge that I have of the actor. One more thing about the well -known URL. The keys can be served from like any arbitrary URL. If they are served from a well-known, we gain a bit more information that the domain owner, which is in control of that has placed and make some special things in order to host those keys at this address.

11:47For instance, if the keys are on clarflare .com slash well -known slash HTTP message signatures, I know that clarflare and the owner of clarflare .com has made some special work in order to make that happen. So it provides some more guarantees and I can tie the keys to the domain which serves them.

12:09Great. So WebBotals has been kind of like a year and a half of it already and we had some learnings from deployments. It has been deployed by bots of various sizes. We have like small bots and we have like bigger bots. There are a lot of open source implementation, TypeScript, REST, Go, a lot of languages. We're probably not aware of like all the implementation that exists. There are some which are more like S.Trick language or like your partner. in love because of the design, feel free to contact us. In terms of what really worked, at least in observation that we had with webbordos, it's like it's signing that happened in the HTTP layers. We also considered, if you look back at the post we made a year and a half ago about webbordos, we also considered doing that at TLS because we have good mechanism.

12:56We have mutual TLS and we're allowed to do that. The thing is, it required a bit more modification and it was a bit harder for people to deploy. Should it be on the bot side? Because again, if you use a Cloudflare worker, you don't control the TLS stack, for instance, it was also harder for people to implement on the server side, should be on the Cloudflare side, but at deployment of various sizes. And therefore it seemed to have been rather good. It's extensible because of the request semantic that we have with HTTP headers, meaning that we provide a minimum set of headers that you should sign in order to be compliant with You shouldn't like customize it and extend it because you have. particular needs for your application, which should be like a client ID, whatever headers, you can do that. And we've seen all the protocol extend web bottles in order to do this.

13:43And finally, one of the main changes, if you're an implementer of web bottles in the room, one of the main things that we've seen people struggle with is we've changed the signature agent from a signature string to a signature dictionary, which allows for multiple signatures to be carried along with one request. Imagine you have multiple requests, each intermediary wants to sign them. This is one thing to know that's been like, the spec is evolving. That's like really something I want to highlight. It's evolving, it's changing. And this is because we just on our own kind of do everything right on the first time. And that's why we took and discussed with people in the open at the IETF in order to like make the protocol evolve. So it's sort of, and it's strong and works for everyone. In terms of code, I tend to like TypeScript.

14:30because they are easy to deploy on Cloudflare platform. So you have like htp-message signature and our library, which is web models. You can use it like already like directly in workers, you can use it in Chrome. And it's not too many lines of code, you probably can like reduce that. It's just in terms of like expressivity, I do feel this was like the easiest. We have the signature agent, so here's like micro.test. We have the request on which we attach the signature agent. Then we create like the fields that we want to assign, so as I said. we want to like have some timestamps, so like the time at which the signature was created, the time it expires, and once we have all the fields we use like the signatures to get like the new headers we want, which is the signature, the signature agent, and the signature input. And frankly that's it. You can just like roll it out today on your own. One of the things that like it really helped here is I think for Origin it allows them to have like more confidence in like the messages that IP addresses, user agents, they worked, but they were not an ideal signal through that.

15:33And really the ability for Regin to be able to choose to have a confident signal that they have worked and reduce the number of incidents that they had. Being able to allow crawler on your website so you have better discoveries through search engines, like AI engines, etc. is something which is important, otherwise you don't get discovered. Similarly for Bolton agents, it allows them to like prove the mechanism with being tampered with. This is something that they control themselves, it's not something which has like a governance or whatsoever, and it's compatible with existing method. I don't have to move away and change my IP addresses, change my user agent, that's something that I can add on top to provide more confidence to the origin that I'm contacting. And that's it. This has been being really open at the ATF. This is where and why the protocol is still evolving. We have some So you have like all the mailing lists, the draft and etc.

16:22at these links. We also have GitHub with like the implementation that I mentioned. And that's it.