Expert Insights

What AI search optimisation is, and how it differs from SEO

Most of what is sold as AI search optimisation is ordinary SEO with a new invoice attached. A small part of it is real, and one setting in it can remove your business from AI answers entirely.

30 September 2026

Let us start with the part that will cost us work to admit. Google has published its own position on this, and the position is that optimising for AI search is still SEO.

Google's words, in its guidance on AI features in Search: the best practices for SEO remain relevant for AI features in Google Search, there are no additional requirements to appear in AI Overviews or AI Mode, and no other special optimisations are necessary. Google goes further and names the vocabulary directly, saying that GEO stands for generative engine optimisation and AEO for answer engine optimisation, that these are terms you may see used, and that from Google Search's perspective, optimising for generative AI search is optimising for the search experience, and thus still SEO.

Asked in its own documentation whether SEO is still relevant for generative AI search, Google's answer is three words. In short, yes.

So if you have been offered AI search optimisation as a separate discipline with its own secret methods, the largest search engine in the world disagrees with the premise. We sell AI Visibility as a distinct service and we will explain honestly why, because the honest reason is narrower and more useful than the sales version.

What is actually different, and it is not the writing

Three things separate AI visibility from classic SEO, and none of them is a different way of writing pages.

The first is the mechanism. Google's AI features do not answer from memory. Google documents retrieval-augmented generation, which it also calls grounding, as relying on its core Search ranking systems to retrieve relevant, up-to-date pages from the Search index. It also documents query fan-out, where the model generates a set of related queries at once and fetches additional results. So an AI answer about your business is usually assembled from pages retrieved at the moment somebody asks, not from something a model memorised during training.

The second is the controls, which are genuinely new and genuinely AI-specific. That is the next section, and it is the most important part of this page.

The third is measurement, which is thinner than anybody selling dashboards will admit. That is the last section.

Different surface, different controls, different measurement. That is the real separation. Not a different algorithm and not a different kind of sentence.

The setting that decides whether you appear in AI answers at all

Google Search Console now carries a Search generative AI control, under Settings. Inclusion is the default for all properties. If a property is excluded, Google states that the site's content is prevented from being visible to users in Search generative AI features, including being linked to within these features and helping to ground AI responses. Google also states that excluding does not affect classic Search ranking or indexing.

So the switch exists, it is on by default, and it is separate from everything else. If somebody has turned it off, your pages can rank normally in Search while your business simply does not appear in AI Overviews or AI Mode. Nothing in your analytics would announce that. Go and look at the setting.

Beyond that, Google's published eligibility rule for being shown as a supporting link in AI Overviews or AI Mode is that a page must be indexed and eligible to be shown in Google Search with a snippet, and that there are no additional technical requirements. The word snippet is doing work there. If a page carries a nosnippet directive, it is not snippet-eligible, and it is therefore not eligible for those features either.

The crawler mistake that removes businesses from ChatGPT

This is the most damaging error in the whole subject, and it is sitting in copied robots.txt files all over the web.

Each major AI provider runs more than one crawler, and they do different jobs. One gathers material that may be used to train models. A different one supports the live search that produces answers with citations. They are not interchangeable, and blocking the wrong one removes your business from AI answers while doing nothing about training.

OpenAI documents GPTBot as the agent used to make its foundation models more useful and safe, which is training. It documents OAI-SearchBot separately, for search, used to surface websites in ChatGPT's search features, and states plainly that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Block GPTBot and you have opted out of training. Block OAI-SearchBot and you have opted out of appearing.

Anthropic splits the same way. ClaudeBot collects web content that could contribute to training. Claude-SearchBot navigates the web to improve search result quality for users. Perplexity documents PerplexityBot as designed to surface and link websites in search results on Perplexity, and states that it is not used to crawl content for AI foundation models, so blocking it removes you from Perplexity results.

Google splits it too, and here the popular advice is backwards. Google-Extended governs the use of content for training future Gemini models and for grounding in Gemini applications. Google states that it does not impact a site's inclusion in Google Search and is not used as a ranking signal. Blocking Google-Extended does not remove you from AI Overviews or AI Mode. Googlebot access is what those depend on.

Two further notes worth having. The tokens anthropic-ai and Claude-Web appear in a great many blocking templates and are not in Anthropic's current crawler documentation, so a robots.txt built on them is not doing what its author thinks. And robots.txt is a request rather than a lock. The specification that defines it says in as many words that its rules are not a form of access authorisation, and both OpenAI and Perplexity document user-triggered fetchers that may not follow it.

If you take one action from this page, open your robots.txt and check which of these you are blocking. It is a ten-minute job and we find it wrong more often than we find it right.

What Google explicitly says you do not need to do

This section exists because these four things are being sold, and Google has addressed all of them.

You do not need special files. Google's guidance names llms.txt directly and says you do not need to create new machine-readable files, AI text files, markup or Markdown to appear in Google Search including its generative AI capabilities, because Google Search itself does not use them. It adds that keeping such a file is fine, and that doing so will neither harm nor help your visibility or rankings, because Google Search ignores them.

On llms.txt specifically, the accurate description is a proposal rather than a standard. The site that publishes it calls it a proposal to standardise on using an llms.txt file. There is no RFC and no standards-track document. Chrome's Lighthouse does include a check for it under an agentic browsing category, calling it an emerging convention and noting that providing one is optional at the moment, and that audit does not fail a site for not having one. No AI provider documents reading yours. Our own site publishes one, and that is the honest status of it: harmless, cheap, not a visibility measure, and nobody should sell it to you as one.

You do not need special structured data. Google says structured data is not required for generative AI search and that there is no special schema.org markup you need to add, while recommending you continue using it as part of ordinary SEO because it helps with eligibility for rich results. Its one instruction in the AI context is to make sure your structured data matches the visible text on the page.

You do not need to rewrite for machines. Google states that you do not need to write in a specific way just for generative AI search, because AI systems understand synonyms and general meanings, and separately that there is no requirement to break your content into tiny pieces for AI to understand it. Worth noting that Microsoft's guidance for Bing leans the other way and does recommend clear headings and question-led structure, so this is a point on which two providers differ rather than a settled fact.

You do not need to manufacture mentions. Google acknowledges that its generative features can show what is being said about products and services across the web, including blogs, videos and forum discussions, and immediately adds that seeking inauthentic mentions across the web is not as helpful as it might seem, because its ranking systems focus on high-quality content while other systems block spam. Google also names the related tactic directly: creating a page for every question people might ask, primarily to manipulate rankings or generative AI responses, violates its scaled content abuse policy.

What you can actually measure, which is less than you have been told

Search Console has a generative AI performance report, rolled out to all websites worldwide during 2026. It gives impressions. Not clicks, not queries, not position. Its dimensions are pages, countries, dates and devices, and it covers AI Overviews and AI Mode together rather than separately. Google also notes that it draws from data already inside the Web search type in the main performance report, so those impressions are already counted in your totals and should not be added on top.

Bing Webmaster Tools has an AI Performance report built on citations rather than impressions, giving total citations and cited pages. ChatGPT referrals can be identified in analytics, because OpenAI documents that ChatGPT includes a utm_source parameter naming itself.

What none of this gives you is the thing most tools promise, which is the list of prompts your business appears for. Google's report has no query dimension. Bing's gives citations, not prompts. Anything offering prompt-level AI visibility is producing its own sampled estimate, and Google is unusually blunt about this category, warning site owners to be wary of third-party tools that promise ranking success or claim to use internal Google metrics, and stating that no third-party tool has access to its internal ranking or AI systems.

You will also notice that this page contains no percentages. That is deliberate. There are a great many statistics circulating about AI search, about how often answers cite pages outside the top ten results, about how much click-through has fallen, about how much better AI traffic converts. No provider publishes any of them. They come from studies whose sampling, geography and query sets you cannot inspect, run by companies selling the service being measured. We would rather publish a page with no numbers in it than one number we cannot stand behind.

So is the term real

The work is real. The label is not settled. Google names GEO and AEO only to say they are terms you may see, and that the work is still SEO. Microsoft uses generative engine optimisation in its own product announcement. Two of the largest search providers do not agree on whether the category exists, no standards body defines it, and anybody claiming a settled methodology is describing their own opinion.

What is defensible, today, with documentation behind every piece of it: make sure you are indexed and snippet-eligible, check that the Search Console generative AI control is set to include you, check that you are not blocking the search crawlers while meaning to block the training ones, keep your business information consistent and current on the surfaces Google itself names, and write pages a person would actually want. That is a short list. It is also most of what works.

Find out what AI says about you.

The free deep digital audit includes what assistants currently say about your business, and whether anything on your side is preventing them from seeing it at all. No account access required, no charge, no obligation.

Questions

Is AI search optimisation different from SEO?

Google's own published position is that it is not. Its guidance states that SEO best practices remain relevant for AI features in Search, that there are no additional requirements to appear in AI Overviews or AI Mode, and that optimising for generative AI search is still SEO. What is genuinely different is not the technique but the surface, the controls available to you, and how little of it can currently be measured.

Do I need an llms.txt file to be found by AI?

No. Google states that you do not need to create AI text files, markup or Markdown to appear in Google Search including its generative AI capabilities, because Google Search does not use them, and that keeping one will neither harm nor help your visibility. It is a proposal rather than a standard, there is no RFC for it, and no AI provider documents reading yours. Publishing one is harmless and cheap. It is not a visibility measure.

Will blocking GPTBot stop my business appearing in ChatGPT?

No, and this is the most expensive misunderstanding in the subject. OpenAI documents GPTBot as the crawler for training its models, and OAI-SearchBot separately for search, stating that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking GPTBot opts you out of training. Blocking OAI-SearchBot opts you out of appearing. The same split applies to ClaudeBot and Claude-SearchBot at Anthropic.

Can I see which questions my business appears for in AI answers?

Not through official tooling. The Search Console generative AI report gives impressions only, with no query dimension and no clicks. Bing's AI Performance report gives citations rather than prompts. Any tool offering prompt-level visibility is producing its own sampled estimate, and Google explicitly warns that no third-party tool has access to its internal ranking or AI systems.