0:02Hello everyone, I'm Will, this is Ashish, and we're from the London office, and it's great to be here today to talk about Answer Engine Optimization, AEO. So today we're going to, I should say, we're both researchers, we've spent quite a lot of time thinking about this problem lately, and so today we want to present our sort of take on this space as a minute, as it's quite nascent, we want to share. interesting research we've done here and how it's helping shape our AO product. Currently in a closed beta, but hopefully we're going to move that forward fairly soon and start getting more people using it. Okay. Cool. Right. So for years, being found online is typically a search engine problem, right?

0:53It's a ranking. and you want to be high in that ranking and this the whole concept of seo search engine optimization was how to boost your presence in that list um and the tracking of this and how well you're performing is to do with things like uh impressions and click through um so this field is still very much active and it's not going anywhere anytime soon but things are changing in the sense that now we have and syringes right so we have um we have my digital assistants. We have things like ChatGPT, et cetera, Gemini, which you're having conversations with. And increasingly, we're seeing that users are asking these assistants to compare, to explain, and to recommend. And what's really important about this in the distinction somewhat from SEO is the decision is therefore happening before any sort of ranking is taking place in search engines, right?

1:48There's no search happening prior to this decision being made. and interesting the signals are different too so it's not the same as these impressions and the click through which we were previously looking at pressure so now we care about citations and mentions um and it gets and then not only citations and mentions it gets more complicated because then we want to start looking at like semantic meaning the intent behind these conversations how you're being represented so it's a very complex problem and it gets even more complex when we then look more towards where the engine is heading and having agents go off and have these conversations with answer engines, right? And so at some point, they'll hand off to the human. But at that point, you don't really know what's occurred between those two, how your recommendation has evolved. And so I guess the take home here is AO. It's fairly emergent.

2:39It's something which lots and lots of people are working towards. And we're going to take you through basically the flow and how we're approaching it. it but i think we need to consider three sort of cool things here which is one uh earning consideration right we want to know um that in high intent scenarios you're being recommended we also want to be able to see the competitive field we want to know if you're being recommended alongside who you consider to be your competitors whether or not they're being recommended ahead of you or why that is or maybe you're being recommended in completely different sectors to one you consider yourself aligning to and why is that um and finally and one of the most important aspects for us at Cloudflare is how can we connect that to outcomes, helping our actionability here. And so that's something which Cloudflare is very well positioned to do because we have a lot of these really interesting signals which we're going to share with you today.

3:34So before we get into aging, I just wanted to spend a few minutes to talk about the network at large. So we analyzed Cloudflare radar between May of 2025 versus May of 2020. 26. And we chose OpenAI for this analysis. So as you can see, there's been a 116 % increase in the OpenAI search bot crawls. What this means is OpenAI has been increasing crawling to build out this search index. We also see a 32 % increase in ChatGPT user bots. These are essentially user initiated bots for folks like you and me to try and ask about questions of like, what's the local it needs to shop. We also see... only a 7 % increase for the GPT bot, and these are bots trying to crawl the web to train the next flagship model. So what this tells us is that the trend is shifting.

4:19The trend is shifting from training to providing answers through agents. And this definitely puts AEO front and center. And I think while this is an exciting space and a time to be in, this is a really hard problem to solve. And the reason I think it's a really hard problem is because if these four gaps. And generally, with AEO platforms, there are a few common patterns, a few common designs. You've got these synthetic prompts, you funnel these synthetic prompts through your agentic harness of choice, if it's OpenAI, Anthropic, Gemini, perplexity even. And then you kind of roll up these scores into some understanding of where your brand is being visible. The thing with these AEO platforms is that it kind of...

5:08exposes these four gaps. So the first one here is for prompt bias. So prompt bias is essentially when you're changing wording, keyword, tone, and style of the prompt, and that actually greatly affects the responses coming out of these agents. When you want to control for prompt bias, you end up actually probing the same prompt n number of times, three, five, even eight times. And that actually kind of results in the second gap of probe cost. So a lot of customers are trying to figure out what's the balance between identifying a good prompt and then probing the right number of times so they can have a semblance of a grounded AEO score. Gap number three is the action gap. A lot of the times AEO scores are after the fact. Customers don't know if their AEO score is because there's a gap in the content or because there's some sort of a bot positioning issue.

5:59I think providing intelligence understand where that gap is and providing actionable intelligence is the need of the R. The fourth most important gap for us is the missing ground truth gap. And I think this is the one that we as Cloudflow want to index heavily on. We want to leverage the edge, the patterns we see in terms of who is crawling whom, what the referrals look like, where do active human crawls happen versus agent crawls. Understanding those patterns and leaving them into our product is something we're definitely doing. deeply interested in. I'll hand it back to you to talk about the proposed solution. So what does the complete answer look like? We think, based on the research we're going to show you in the coming slides, that we should treat it as a cyclical process.

6:48We've got these three key steps, but the whole thing should be a feedback loop. And we should iterate. We should be able to actually see incremental progress So to start off with, We want to measure what assistants say. So these are different assistants, not just one. We want to make sure that we're covering what assistants people tend to use. So that's like Gemini, GPC, Avrovia, et cetera, Claude.

7:13But it's important to track the mentions. It's important to track citations, work out the share of the model, tease out other relationships we can do by taking all these in parallel. But this is always going to be a sample of synthetic answers. That's not to say that you can also augment that. with additional curated prompts, prompt sets you purchase. They're very valuable and lots of work goes into that. But at the end of the day, they're telling you what an assistant might say and not what real conversations have been had by potential customers. So with that in mind, we need to actually observe what really happens. And that's where the signals in our network come in. Essentially, we want to know what's fetching your content and for what reason. And we also want to know the actual transfer between... um an agent and a human so when someone an ai referral essentially when an agent refers the human and the human clicks through and then engages with your domain um and so that's first part of your ground truth so that's number two and number three is acting on it we want to make sure that we can actually um put things in place to improve upon these things so once we start noticing trends we want to then see can we impact those can we actually get you recommended at an earlier stage can we get you so but the click-through is actually happening, and that's something else we're going to talk to about in a bit.

8:31But I think, essentially, if we go back to what Matthew was saying yesterday in terms of the deli shop owner, them knowing your names, or the teenager building their first site, and wanting to help them with that, essentially, the internet works when it knows you and helps you to get things done, and it's not just about your rank. So we need to make sure that we're... treating this as a much more sort of holistic process. So we're going to build to test our idea. And that's what we've been researching. And yeah, we're going to share some of the findings now,

9:05We've conducted a series of experiments with the intention of identifying how to best target these problems. So over here is basically our experiment where we conducted where we tried to elicit the number of times Cloudflare was mentioned or cited with these 50 samples being sampled eight times across each of these three vendors. The intention of this is to look at, does it matter who you ask the question to? So if you look at the mentions for OpenAI versus Google versus Anthropic, that's the left textured bar. We see that for OpenAI, Classler is being mentioned 74 % of the time, but it's cited just nearly as half that with 47%. With Anthropic, it's 57 % on the mentions, and then it's being cited 11%. So And then more importantly, with Google, it's been mentioned 90% of the time, but it's been cited 19% of the time.

9:55So the point here is, even with the small samples as of 50 prompts being sampled eight times, each of these vendors have their own logic in terms of mentioning and citing. So it's really important to understand with these AEO platforms and scores, how you're reasoning about what that overall rolled up score looks like and how to actually understand the nuances for each of these vendors. along with understanding who you're asking these prompts to the next part of the puzzle is what's being asked so over here we looked at understanding different prompt styles so we tried to understand how much do the aeo citations and mentions change depending upon what's being asked so bottom up if you look at prompts that are more conversational versus the plain question we see uh the citations go from one to four and then if it's slightly more keyword focused or comparative or even ranked lists the number of citations jump up.

10:50So the intention here again is very much emphasizing the way you phrase the prompts really matter and it kind of makes it front and center for these synthetic panels we're creating. How do we actually put one together that is one grounded but also more expansive so that we understand what the user has been asking. how do we make sure that it handles all of these different prong varieties? And more importantly, handling costs, because the more varieties you have, the more you have to account for the different probe variations for each of these vendors and their costs.

11:31So yeah, the previous two slides that she's shown with is prompt bias is a thing, right? And we wanted to see what we could do in terms of mediating this. in the sense that we want to reduce costs, but also we want to reduce noise, right? We want to boost that signal coming out of these. So what we've been playing around with is using decision models and classifiers. Essentially, we've got a huge amount of data now. We're collecting a lot of synthetic AEO prompts, our own sort of VIN-house AEO prompts as well. And so given that data source, but also the data source of our knowledge graph, which we've been building as part of the AEO... product um which looks at sort of yeah not only citation and mention patterns but also sort of co -linkages within these networks sync uh co -citation looking at labeling by industry etc we have this sort of wealth of knowledge which you can start to tie together and actually start to use to to make things run better in the ao space and so as an example here what we have is we we we fine -tuned a um decision model um and then we realize that actually you don't need a decision model of that scale right so you get still fairly decent accuracy but if you just distill it down into a much smaller classical machine learning model right and you can run that on the edge so that means essentially you can filter your prompts for essentially zero cost um as you're using them um and by filtering i mean we can train classifiers for different purposes this one example here is looking how we can boost for

13:01productive prompts so various prompts which tend to give you a result which will site you or a competitor so and it lies with your industry area but we can also boost it in different ways like we have classifiers now which are running to boost certain semantic attendant the point being that actually we don't need to spend a huge amount of money on probes we don't need to have these really expansive prompt sets to get meaningful answers so this is reducing the cost massively but still getting the signal and it's removing a lot of the noise associated with bad prompting which is is quite a bit of a problem. And so the intention here is that we're going to use this on real AEO prompts and so that we can then make that sort of, if we go back to the earlier slides with the synthetic prompting side of things, we can make that more robust and run at scale.

13:55So that's all covering the first two gaps, which as CJ identified earlier, around the prompt bias and prompt. cost the probing cost um the other things we need to start tackling though is actually how we do the observability and the action ability here um and so i want to talk a bit about signals from the network um because we really want to know what people do with the information recommendations which assistants would have told them because an assistant um and it's an answer style to what the model already knows inherently right so um they've learned that from past crawls the what we tend to find from the sympathetic side of things prompting side of things is it's interrogating the model's core knowledge, but it's not really looking at the latter steps in the answer agent process. So what we need there is the signal which comes from the network. So I guess what I want to take home from this big here is that if we normalize the site size, which is a massive co -bounder in terms of the signal coming out for AAO, because the larger your site, the more likely it is to get referred to an answer agent.

14:58and that's it's the driving signal but then we have these other signals emerge when you account for that and we actually find that the mentions and the citations actually have very little impact on whether or not you'll get an ai referral from um from an answer engine and actually the things driving that are coming from the crawls they're coming from the training bots and they're coming from the user initiated fetches and so what this data is telling us is those are things we need to target in terms of action ability to it, and actually, she's going to talk a bit more about it. So I think the key here is that user-initiated fetches have a correlation with referrals. And I think our next part of the experiment was to try and analyze what happens if with user-initiated bots, if your firewall is misconfigured.

15:47We tried to analyze what are the different permutations and combinations where bot positioning affects user-initiated bots. So. the top row here is we sort of assembled this list of 27 test sites. And each of these test sites had custom paragraphs where the LLM had to go and fetch them so that they could answer the question, because all of them were made up facts. So the intention was, if there's a well-configured robots.txt, if there's a well-configured firewall, how do Anthropic and OpenAI crawl the site and actually get the responses back? If things are well-configured, in almost all scenarios, they're able to crawl effectively. In scenario two is when your firewall is misconfigured and the robots .txt is configured correctly. So over here, as expected, you would get a firewall block with the 403.

16:34I think the more interesting piece here is scenario three, when we notice that with Anthropic and OpenAI, when both of the robots .txt block the user -agent bots, we notice that Anthropic actually doesn't. crawl your site because it adheres to the robots .txt, but for OpenAI, it actually crawls. And this is part of how they're positioned. With OpenAI, their positioning is that user -initiated bots are an extension of the user, so they're sort of ignoring the robots .txt. So the intention of this slide is to showcase that with just two different agentic vendors, there's so many permutations of how robots.txt firewall network on the... edge in terms of crawls, referrals, span out.

17:21And it's really important for us to actually make these insights actionable with call-to-actions that are more easier to digest and once they can actually flow into the UI. So we're just going to quickly switch gears to a demo.

17:41So for the interest of time, I'm just going to provide a layout overview of what... our AEO page looks like. And for those folks that want to actually do a deep web, I'm happy to answer some questions. And we're also going to be in the speaker lounge today. So the general layout for our UI comprises of actionable intelligence in the top section. So like I mentioned, if you happen to have bots that are misconfigured, which in this case, we have an example of the ChatGPT bot. Specifically, we have the search caller being misconfigured, you have the option of reviewing that particular robots at TXT. You can enable or disallow or allow them. Or if the block is intentional, you can actually keep this policy.

18:27And then you can train our system to understand what your behavior looks like. Furthermore, with the 403s and the firewall blocks, we also capture what's happening on the edge. So in this particular example, we can see that 27% of visits or feedback And the intention here is to give signals back of what's happening on the edge so we can act on it. And then in terms of rough layout, along with identifying configurations for industry, category, and defining your competitors, once we have this information of who you are and what does your narrative look like, we also capture performance metrics for citation and mention. We also perform share of voice analysis so we can identify where you step up against your chosen competitors.

19:18And we're also providing signals from the edge in terms of the crawler and referral by operators. So the intention is we can show you either on a seven-day average or a 30-day average, what do your crawls for operators look like? And we're also visualizing that in a time series format to give you guys better insights of where the trends are. Is it improved? is dipping.

19:43We also have an AI Visibility tab. And I think this tab particularly addresses some of the issues or gaps with prompt bias, where we're actually capturing all of the AEO metrics by vendor. So we can perform deep dives by different prompt categories and also look at each individual prompt and their responses that come out.

20:05And we're also capturing. in citations is a way for us to understand what are those sites that actually have the most amount of clicks that are occurring through them.

20:18All right. If we can hop back to the slide, please.

20:28Thank you, guys, for attending. We'd love to ask some questions.

20:34Go for it. I'm just going to give you a mic here.

20:50It's not that I got lost.

20:58Is it on now? I noticed at the bottom of... most of the slides, it's something about using API as opposed to like the actual consumer app. Is that what is also being experienced in the AEO product itself? Is there a reason? I guess it's a multi, like why, what are the consequences of that? And what is the customer feedback? Yeah, it's a good question. And yeah, it's something, these are more the research slides, which we were using the API for. It's a very good question. The product now is moving towards using the consumer app because you're exactly right. It needs to mirror what the consumers are doing. Interestingly as well, we're also finding that a lot of the API parameters aren't being respected quite often around location and things like that.

21:45And particularly when you've got a regional business and you want to be represented within that region, things like that really matter. And so the API is a bit limiting for that. What we're actually finding, we didn't present the data on this at a minute, but the pre-pending... sort of like context into these is much better than actually using the APIs in terms of getting domain locality and all that sort of thing so yeah the app is going to be released it's going to be much more focused on the consumer driven side yeah good question there I would also be curious if you have a hypothesis untested I suppose about how the memory in the consumer apps would affect so for example if I'm over years having conversations with Claude say that implied that i like cloudflare yeah it seems like it would be more likely to recommend cloudflare at the top of those like yeah any thoughts on how that could be measured it's that's a really interesting problem when we've discussed um we're trying to do it in a minute by setting up like personas and trying to model it as like different consumer types it's a very hard problem though because that's like an infinite space right so we need to work in a way to sort of approximate the consumer pools and so i think this is going to be a case of having to do some quite extensive data science prior to rolling something out like that out um but at the minute everything's going to be yeah quite generalized

23:09question there

23:17what is the timeline to release it as that looks exciting you guys killed half of startups in team Good question. I'm going to hand it off to the innovators in our product team to answer that.

23:34We just made researches. I would have pawned that one off, guys. We really appreciate it. Yeah, so as you can see, we're kind of in the research stage. And right now, we're in closed beta. So we're working with a few partners, mostly Cloudflare existing customers, because everything builds on top of the CD. end data um we don't want to jump straight into picking random prompts and seeing if you're in those answers and so everything starts with what is the agent bot in crawler traffic then we'll use that to formulate what synthetic prompts we um we pick from there and make sure the representative um but to get the question it's coming uh we're in closed data at the moment probably expanding that to a broader pool within the next i'd say month or so um and so probably the end of this year, early next, as well.

24:21There, too.

24:28Hi. You know, in one of your slides, you mentioned that we add robots.txt. Then it starts showing up. So does that mean, like, now LNM functionality is getting overwritten? Because if there is no robot.txt, then it's not coming up. So robots. robots .txt is if it doesn't exist, then bots are implicitly allowed. So I think the nuance there is with user bots specifically. With user bots, Anthropic is honoring robots .txt even for user initiated bots. But OpenAI doesn't honor it because their intention is if it's user initiated, then it's an extension of the user. So I think there's slight nuances there between Robots are TXT positioning generally means implicitly allow if it doesn't exist.

25:20Is it similar to like, you know, putting the Google tag and then they start appearing it? Sorry, repeat the question? Is it similar to like, you know, putting the Google tag, then they are appearing in the like Google search?

25:34It's, I think the paradigm is like slightly shifted now. It's really about getting the configuration right about how you want to be visible for brand. I think it's all about understanding if you want to be implicitly allow or explicitly disallow for certain bots. For example, I know news sites tend to explicitly disallow certain sites. So I think that it's similar to the idea for tags, but I think it's a slightly more AEO view for that. Thank you.

26:08All right. Really cool research. thank you for sharing that. I have a question, a follow -up on that, Robots .txt. Is there a reason that you're benchmarking against Robots .txt rather than LLMS .txt? Is it because the adoption level is not there yet? Or do we expect, in other words, do we expect some kind of apples -to -apples for AI and crawlers when it comes to LLMS .txt investment? Good question. So I think from our So far, robots.rxd is the most widely adopted. So the intention for this analysis was to see, across the most widely adopted configuration, how do different agents behave. That was the main reason for choosing robots.rxd.

26:55I think LLM.rxd is also partially used. However, I think robots.rxd is the primary configuration OK.

27:10so there's been a lot said that really good seo is also really good aeo i'm wondering if you've done any analysis that shows that um sort of just having really good seo means you have really good aeo that's a good question i think um we actually started off as an seo shop back in the way back in the research days so i think there's definitely parallels between SEO and AEO. So a good example was the original paper that inspired SEO and AEO specifically was the parallels between PageRank in the SEO industry and the idea for like prominence in the AEO space where you're trying to understand like for example within a given LLM response how many of the tokens in the response can be attributed to a certain brand.

27:58So there's definitely parallels that you can draw between SEO and AEO, I think the intention there is, if you want to try and move towards the agendic flow, is where the gaps between SEO and AEO becomes a little more prevalent.

28:22I just have a question, and there's one behind you as well.

28:30Did you analyze? frequency of visibility based upon schema markup, whether there's a markdown text in the page, other kinds of parameters like that? That's on the roadmap. So basically, we have a product you can use now, Agent Readiness, and a lot of those things are actionable items such as markdown, et cetera. What we're aiming to tease out now is given those as a set of variables. So if we go back to one of those... as I showed before when we had the sort of the crawl statistics and the prompts and mentions that was just a subset of all the different variables we're looking to interrogate it they all appear and they all have a signal the ones we pursued and the start to present here were the crawls because it was the strongest link but yes we want to go and we want to look at those other things and then it becomes a much more sort of complex analysis in terms of the covariance between each of those um so yeah it's on the roadmap and it definitely anything which is actionable and particularly in agent readiness which you can then go and suggest you go and you change it in terms of the Cloudflare dash.

29:32That's exactly what we want because that goes into the feedback loop, which then allows you to track. And then you'll see on the charts we showed when those points in time, when you're initiating those changes, and hopefully you'll see your trends increase. And that's what we're finding so far in the initial studies, yeah. I'm going to sit in one more question up here. Thank you. In your research, are you able to or have any insights around... which search indexes are being queried. It used to be that Google was the only game in town and Bing, we love you, Bing. But now it seems like everybody's trying to build their own index, including Cloudflare. So it's a great question. I think we could say yes, and then we might publish a blog post around that shortly. But we can definitely tease that information out.

30:20And you're right, in terms of index use, it's a really interesting area. looking out what all these agents have built upon is really interesting and something we're working on. So, yes, great question. And yes is the answer, yeah.

30:35It's a good watch this space. It's not cancer. Is that great? And we don't want to speak a thing yet. We got one here as well, if you want to speak. Let's put in one more.

30:49Hey, I'm just curious, in terms of the prompt bias because the prompt can come from the user and the user prompt can go to different agents sometimes agent will rewrite those prompts and in order to actually intervene this part i don't think either cloudflare or as the source provider have any capability to intervene those two steps say So how are you going to reduce the variance in terms of the prompt bias? So are you asking in terms of how real people are using answer agents? Is that what you're asking in terms of can we?

31:35So in the four problems you laid out, so the problem number one is people might ask various different, asking various different ways. So that problem. can come from user can also come from agent rewritten prompts yeah so how does like cloud flare actually being able to tackle that part or like maybe generalize that part yeah our generalization is it is tricky because at one hand you want to generalize it and it works at scale because this will be immensely costly if we wanted to take basically tailor it to every individual and agent etc and it has a lot on their context so to begin with we're looking at keeping it generalizable but we need to come up with a way that we can like i said before these personas where we can actually interrogate it based on the user group who are particularly interested in a certain thing so it's not just about your domain representation in answer to a question but it's also linking that with the certain users and how you appear to those users so yeah it's a good question um we're primarily trying to do this based on small synthetic curated prompt sets to make this work at scale but we realized that we're going to have to then probably augment that with some sort of additional prompting and so what we're looking at at the minute is how we can sort of build that iteratively and maybe start with like a core generalizable prompting set and then maybe we can tailor that and see how that looks

33:07with these different user groups but also what we need is for user research like what what kind of users do we want to model what domains that we're focusing on here, I think we need to narrow down the problem space a little bit, which is what we'll probably end up doing as we start iterating on this first. Like we can't just unleash an AEO product which works for every single domain and every single sort of user could be targeted. Yeah, I think so. Ashish, do you have anything to add on that one? I think the intention for us is to try and look at our traffic on the edge. And I think we do have products within Cloudflare that do have access to human products. I'm just going to let David speak about the roadmap there, but I think there's potential for us to understand real human prompts, synthetic prompts, and then build this cross-sectional funnel so we can actually have a good quality prompt set that can help customers.

33:57But we can talk about this more in the speaker session. Thank you, everyone. Very much, Chas.