SEO in the AI Era: The Complete Guide

Most of what is written about AI search falls into two piles. One says everything has changed and your existing work is worthless. The other says nothing has changed and this is all hype. Both are wrong in ways that cost money.

This guide sets out what actually changed, what did not, and a framework for finding where your own visibility is breaking. It is the same framework behind the audit I run for clients, published openly, partly because the method should be inspectable, and partly because a framework you can apply yourself is more useful than one you have to buy.

Who this is for

Marketing directors, CMOs, and founders responsible for organic visibility at companies with an existing content operation. It assumes you understand search fundamentals. It does not assume you have looked at AI search in any structured way, because most teams have not.

---

Part one: what actually changed

For most of the history of search, position and traffic moved together. Holding position three for a query with volume produced a predictable share of clicks. Position became a proxy for traffic, close enough that teams stopped distinguishing between them.

AI-generated answers broke that relationship. When a search produces a synthesised answer, the user often has what they came for. The query happened. Your page may have been retrieved, assessed, and even used. The click did not occur.

This is why the most common symptom is so disorienting: rankings hold, traffic falls, and nothing in the standard reporting explains it. The full diagnostic for that pattern is worth reading before you act on anything else, because three other explanations produce similar-looking declines and call for entirely different responses.

The second change is where answers now happen. Google's AI Overviews are the visible part. Behind them, a growing share of commercial research happens inside ChatGPT, Perplexity, Claude and Gemini, where there is no results page at all, just an answer that names some companies and not others.

What did not change

The technical and structural work that earns a ranking is the same work that makes a page citable. A page that cannot be crawled, rendered, or parsed will not be retrieved by an AI system either. Site architecture, internal linking, page speed, structured data, and authority signals all still matter, but their job description changed. They stopped being the whole game and became the entry requirement.

This is the part both camps get wrong. SEO did not die. It became a prerequisite for something else.

---

Part two: The Visibility Stack

Four layers. Each one is a prerequisite for the layer above it. Work on layer four while broken at layer one and nothing happens, which is the single most common and most expensive pattern I encounter.

4. CITATION    Is there anything here that cannot be synthesised elsewhere?
3. ENTITY      Do these systems know who you are, consistently?
2. STRUCTURE   Is there an extractable answer in this page?
1. ACCESS      Can the systems reach, render and parse it at all?

Diagnose from the bottom. Fix from the bottom.

Layer 1: Access

The unglamorous layer, and the one that disqualifies more sites than anything else.

What breaks it:

• Crawler directives excluding AI user agents, often added in 2023 to prevent training use and never revisited
• Content that is not present in the initial server-rendered HTML
• Substance hidden behind an interaction that loads on click
• Bot protection and rate limiting configured against scrapers, catching legitimate crawlers alongside them
• Slow or unreliable responses causing fetches to time out

The training-versus-retrieval distinction matters enormously here and is almost always conflated. Blocking training has close to zero effect on whether you are cited today. Blocking retrieval removes you from consideration entirely, on every relevant query, starting immediately. The full audit of what you are currently allowing covers the specific agents and what each one controls.

How to check: fetch your key pages as a crawler would and confirm the content is in the raw HTML. Read your robots.txt line by line and be able to justify every directive. Check server logs for AI user agents: whether they arrive, and what response codes they get.

Layer 2: Structure

AI systems extract passages, not documents. This has a direct consequence for how content is written and templated.

A page that answers its question in a self-contained paragraph near the top is extractable. The same information delivered across twelve hundred words of narrative build-up is not. There is no passage to lift.

What works:

• Answer-first blocks. State the direct answer before the elaboration, in a self-contained form that makes sense lifted out of context.
• Descriptive headings. Headings that say what the section contains rather than teasing it.
• Self-contained paragraphs. A paragraph depending on the three before it is a poor extraction candidate.
• Correct structured data. Schema is a comprehension aid that helps a system parse what a page contains and how its parts relate. It is not a ranking lever and does not guarantee citation.
• Genuine FAQ sections where questions are actually being asked, marked up with FAQPage schema.

None of this requires writing badly. Answer-first structure serves the busy human reader too. This is one of the rare cases where the machine and the person want the same thing.

What does not work: mechanically converting every page into a question-and-answer format, keyword repetition aimed at language models, or any instruction embedded in page content telling a model what to say. The last one does not work and is the sort of thing that gets a domain treated as adversarial.

Layer 3: Entity

These systems build an internal model of who you are, what you do, and what you are authoritative about. That model is assembled from signals across the whole web, not just your site.

What strengthens it:

• Consistency. Identical name, role, and description across your site, structured data, LinkedIn, YouTube, conference bios, and guest appearances. Where these conflict, the system's confidence drops.
• Complete Person and Organization schema, with a sameAs array pointing at every profile you control.
• Corroboration. Being discussed in independent places associates you with a topic more strongly than asserting it repeatedly on your own domain.
• A clear topical focus. A domain covering five subjects deeply is understood better than one covering thirty shallowly.

Entity work is cheap relative to its effect and almost nobody audits it. If you have rebranded, repositioned, or changed focus in the last three years, there is a good chance these systems still describe you as what you used to be.

Layer 4: Citation

The top layer, and the one everyone starts with.

A model can produce a competent general answer from the hundred articles that already exist on any well-covered topic. It has read all of them. What it cannot do is invent your data, your results, or your methodology.

What earns citation:

• Original data. Research, surveys, benchmarks. The highest-leverage content investment available, and the one most companies never make.
• First-hand results with real numbers and stated sources.
• Named methodology. A documented way of doing something, which can be referenced.
• Specific experience. "When I ran marketing for a SaaS hospitality company" carries more weight than "companies often find".

What does not: more content saying what is already widely said. Volume is the most expensive mistake in this category because it feels like progress. It adds cost to the side of the ledger already underperforming.

---

Part three: measuring it

Everything above is testable, and until you test it you are guessing.

The short version: build a fixed set of prompts covering how people describe your category, your brand, and the topics you claim. Run each in a clean session with memory and personalisation disabled, several times, across the assistants your buyers use. Score each result as cited, mentioned, or absent. Record how you were described and which competitors appeared instead. Repeat quarterly with identical prompts.

Two controls determine whether the exercise is worth anything:

Clean sessions. Asking an assistant about your company from your own account with memory on returns what it knows about you as a user, not what it knows about your company as an entity. This is the most common way teams reassure themselves with a meaningless result.

Repeat runs. These models are non-deterministic. One run is an anecdote. The frequency of citation is the finding.

The complete method covers prompt construction, scoring, and the retrieval-versus-training split that determines what your results actually mean.

---

Why Bing Webmaster Tools matters more than most marketers think

Most marketing teams stopped opening Bing Webmaster Tools years ago, on the reasonable logic that Bing's share of traditional search did not justify the attention. That logic no longer holds, for a structural reason rather than a market share one.

Microsoft's index does not only serve Bing. It also feeds Copilot, and it has been used to ground answers in other assistant products. When an assistant retrieves a live source to answer a question, it is pulling from an index. If your pages are not in that index, cleanly, they cannot be retrieved, and no amount of content quality changes that.

This makes Bing Webmaster Tools an access-layer instrument. It answers a question Google Search Console cannot: whether Microsoft's side of the ecosystem can see and read your site.

What to check:

1. Index coverage. Confirm the pages you care about are indexed, not merely discovered. Discovered but not indexed is the failure state that matters.
2. Crawl errors and blocked resources. A robots.txt or firewall rule that blocks Bingbot while permitting Googlebot is a common and invisible failure, and bot management services are a frequent cause.
3. Site scan. Bing's crawler-based audit surfaces on-page issues, and it is free.
4. IndexNow. Bing supports IndexNow, which pushes new and updated URLs to the index rather than waiting for a crawl. For a site publishing several times a week, that shortens the gap between publishing and being retrievable. In practice it is a single API key file at the site root and a ping on publish from the build pipeline.
5. Referral traffic from assistant surfaces. Traffic arriving from Copilot and similar surfaces shows in your own analytics as referral traffic. Segment it. Even at low volume it tells you which pages are being surfaced, which is a different and more useful signal than which pages rank.

What it does not give you: Bing Webmaster Tools does not report which prompts caused your page to be cited, and it does not report citations inside answers where no click occurred. Nothing does, currently. That gap is why prompt-level baseline capture is a manual protocol rather than a dashboard.

One caution. Microsoft ships changes to this product often, and the specific reports it offers have been renamed and reworked more than once. Treat the names above as a starting point and confirm against what the tool actually shows before you act on any single report.

---

Common mistakes

Publishing more into the problem. The content losing clicks to AI answers is overwhelmingly content that summarises the well established. More of it produces more pages competing for searches that no longer generate visits.

Treating AEO as a separate discipline. The terms are largely interchangeable and the work is not separable from SEO. Buying a second vendor for it usually means paying twice for overlapping work.

Optimising layer four while broken at layer one. Investing in original research on a site that blocks the crawlers, or renders its content client-side.

Ignoring entity signals after a repositioning. If your description on LinkedIn does not match your website, you are actively weakening the systems' confidence in what you do.

Measuring nothing, then claiming a result. Without a baseline captured before you changed anything, you cannot say whether it worked, and any number you cite afterwards is selected rather than measured.

---

Glossary

AEO: Answer Engine Optimisation. Optimising to be the source an AI-generated answer draws on.

GEO: Generative Engine Optimisation. Used interchangeably with AEO.

AI Overviews: Google's AI-generated summaries appearing above conventional results.

Retrieval: When an AI system searches the live web and answers from what it finds. The surface your work influences.

Training: When a model answers from what it learned during training, with no live lookup. Not directly optimisable.

Entity: The internal representation an AI system holds of a person, company, or concept.

Extraction: The process of lifting a relevant passage from a page to use in an answer.

Citation: Appearing as an attributed, usually linked source in an AI-generated answer.

---

Where to start

If you have not measured your current position, start there. Every fix before that point is a guess, and you will have no way of knowing later whether it worked.

If you have measured, work the stack from the bottom. Access, then structure, then entity, then citation. It is tempting to start at the top because that layer is the interesting one. It is also the one that does nothing if the three below it are broken.