Optimizing a small business website for AI-driven search
AI Search

Optimizing for AI-Driven Search: What Small Business Sites Get Wrong

Most sites that never get cited in an AI answer are not losing on content quality. They are failing earlier, at a gate they do not know exists.

A customer no longer types "junk removal Tucson" and scans ten blue links. They ask an assistant to recommend somebody, and it names three businesses. If you are not one of the three, the search is over before your site was ever a candidate.

The reflex is to assume this needs a whole new discipline. It mostly does not. Google's own position is blunt: there are no additional requirements to appear in AI Overviews or AI Mode, and no special optimizations necessary. What changed is not the rulebook. It is that a page now has to survive four separate gates instead of one, and small business sites tend to fail at the first two — access and rendering — while everyone argues about the fourth.

This page is the site-side work: what to check, in what order, and what the platforms actually document. The local half of this — Google Business Profile, reviews, citations, being the answer to "near me" — is a different problem with different levers, and it lives on how to improve AI search and local visibility.

What actually changed, and what did not

Classic search matched a query to a ranked list of documents and handed you the list. AI-driven search decomposes the question, retrieves passages from several documents at once, synthesizes an answer, and cites a handful of sources. Three consequences follow, and only three.

Ranking first stopped being the finish line. Being cited inside the answer is the placement that gets seen. A page can rank fourth and be the source quoted, or rank first and be skipped, because retrieval happens at passage level rather than page level.

One question becomes many. Assistants expand a single prompt into multiple background queries and pull from whatever answers each fragment best. That rewards pages that answer sub-questions cleanly and completely in one place, and punishes pages where the answer is spread across three sections and a video.

The machine reads the HTML, not the design. No AI system sees your layout. It sees text, structure and markup. Every decision that trades machine-readable structure for visual polish is now a direct cost.

What did not change: Google's guidance on succeeding in AI search is the same set of fundamentals it has published for years — unique content, good page experience, technical accessibility, structured data that matches what is visible, and understanding the full value of a visit. There is no secret token. Anyone selling you one is selling you something.

The short version

You cannot be cited by a system that cannot fetch your page, cannot see your content in the rendered HTML, cannot extract a clean self-contained answer from it, or cannot work out who you are and where you operate. Fix those four things in that order. Everything else on this page is detail.

The four gates a page has to pass

Work these top-down. A page blocked at the edge never reaches its own robots.txt, and no amount of content rewriting fixes a fetch that never happened.

1. Access

Can the crawler fetch the page at all? Edge and firewall rules, robots.txt directives, HTTP status codes. This is where most silent failures live, and it is the cheapest one to fix.

2. Rendering

Is the content actually in the HTML that comes back? Copy injected by JavaScript, buried in tabs, or suppressed by snippet controls may never be available to quote.

3. Extraction

Can a self-contained answer be lifted from the page without the surrounding context? This is a writing and structure problem, not a technical one.

4. Trust

Is it clear who wrote this, what the business is, and where it operates? Entity clarity is what decides which of several adequate sources gets named.

Gate 1: crawler access

There are two completely different kinds of AI crawler, and the distinction is the single most useful thing on this page. Training crawlers collect content that may be used to train future models. Retrieval crawlers fetch pages to answer a question somebody is asking right now, and they are the ones that produce citations. Blocking a training crawler is a licensing decision. Blocking a retrieval crawler removes you from the answer.

Most small business sites that are invisible to AI assistants never made either decision. A security plugin, a host preset or a CDN rule made it for them.

User agent What it is for What blocking it costs you
Googlebot Google Search index, which is what feeds AI Overviews and AI Mode Everything. You leave Search and the AI features together
Google-Extended A control token, not a crawler. Governs training and grounding in Gemini apps Nothing in Search. Google documents that it does not affect inclusion in Search or its AI features
OAI-SearchBot Retrieval for ChatGPT search results Your pages stop appearing in ChatGPT's live web answers
GPTBot Training crawler for OpenAI foundation models No effect on ChatGPT search visibility. A licensing choice only
Claude-SearchBot Retrieval for Claude's web search You drop out of Claude's cited sources
ClaudeBot Training crawler for Anthropic models No effect on retrieval. A licensing choice only
PerplexityBot Retrieval and indexing for Perplexity answers You are not a Perplexity source
Bingbot Bing index, which grounds Copilot and several third-party assistants Bing, Copilot, and anything downstream of that index

The Google-Extended row is where people get this backwards. It is a robots.txt token with no user agent of its own — Googlebot still does the crawling — and Google states it does not affect a site's inclusion in Search or its AI features. If you want out of AI Overviews and AI Mode specifically, that control now lives in Search Console under Search generative AI, not in robots.txt, and Google says using it does not affect standard rankings. Blocking Google-Extended to hide from AI Overviews accomplishes nothing except forfeiting Gemini.

  • Open yoursite.com/robots.txt and read every line. Look for any Disallow aimed at GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot or PerplexityBot. Plugin presets and "protect my content" toggles add these silently.
  • Check the layer above robots.txt. Cloudflare and several managed hosts ship bot-fighting defaults that block AI agents at the edge, before your rules are ever consulted. A 403 at the firewall is invisible in every SEO tool you own.
  • Decide training separately from retrieval. Allowing OAI-SearchBot while disallowing GPTBot is a coherent position: appear in the answers, stay out of the training set. Blocking both because a plugin offered one checkbox is not.
  • Fetch your own pages as a bot. A curl with the crawler's user agent string tells you in one second what months of guessing will not. If it returns 403 or a challenge page, nothing downstream matters.

Gate 2: what is actually in the rendered HTML

Retrieval systems quote text they can read in the response. Several common build patterns hide text from them while leaving it perfectly visible to a human, which is why this failure is so hard to spot by looking at the site.

Client-side rendering

Service descriptions, pricing and FAQs injected by JavaScript after load. Google renders JavaScript but not reliably or promptly, and most third-party retrieval crawlers do far less of it. Use URL Inspection in Search Console and read the rendered HTML, not the source.

Accordions and tabs

Content that only exists after a click is a gamble. Native details and summary elements keep the answer in the HTML. A JavaScript widget that fetches panel content on demand does not.

Snippet controls

nosnippet, data-nosnippet and a restrictive max-snippet limit what can be shown, and by extension what can be quoted. Sites inherit these from old plugin configs and never look again.

Text baked into images

Price lists, hours and service tables set as graphics are unreadable and unquotable. If a number matters to a buying decision, it belongs in HTML text.

Noindex left on

Staging flags that survived launch. Not eligible for Search means not eligible to be an AI source. Confirm the indexing status of every commercial page, not just the homepage.

Slow or unstable pages

Retrieval fetches time out. The same Core Web Vitals work that helps human visitors also decides whether a crawler gets a complete page or an abandoned request.

The two-minute version of this audit: load the page with JavaScript disabled and see what survives. Whatever disappears is content you are hoping a machine will infer.

Gate 3: writing passages that can be extracted

This is where most of the actual work is, and where the advice online is worst. It is not about a special format or a keyword density. It is about whether a two-hundred-word chunk of your page still makes sense when it is lifted out and read alone, because that is exactly what happens to it.

Structure

  • Ask the question in the heading, in the words a person would use
  • Answer it in the first two sentences underneath, then elaborate
  • One question per section, with the section self-contained

Language

  • No "as mentioned above" or "as we discussed" — the reader may not have the above
  • Name the subject in the paragraph rather than relying on a pronoun from three paragraphs back
  • Plain declaratives beat clever setups; the summary is what gets read

Substance

  • Specific numbers, prices, timeframes and thresholds, not ranges hedged into uselessness
  • Original data, method or experience that is not on the other nine results
  • Comparison tables, which retrieval handles unusually well

The uncomfortable part of this: commodity content has stopped working entirely. If your page restates what twenty other pages already say, a synthesizer has no reason to cite yours over any of them — it can produce that summary without you. The only durable position is having something on the page that cannot be assembled from the other results. For a service business that usually means real prices, real process detail, real failure cases, and a named human who has done the work. That is most of what content creation is now for.

Gate 4: entity clarity

When several sources answer a question adequately, the tiebreak is confidence about who is speaking. Structured data does not get you into an AI answer by itself — Google is explicit that no special markup is required — but it removes ambiguity about identity, location and offering, and ambiguity is what gets you dropped.

  • One Organization or LocalBusiness definition, sitewide, in a single graph. Duplicate and contradictory blocks are worse than none. Two schema plugins both publishing an Organization block is the most common version of this, and it is invisible until you validate.
  • Markup that matches visible text word for word. An FAQPage block whose answers do not appear on the page is the fastest route to having your structured data ignored.
  • A named author with checkable credentials. Not "the team." A person, with a bio page, an occupation, an employer and a history that can be corroborated elsewhere.
  • Name, address and phone identical everywhere. Site footer, Google Business Profile, every directory. A machine matching entities treats a formatting difference as a possible different business.
  • Corroboration off your own site. Assistants weight what other sources say about you. Directory listings, reviews, press mentions and profiles are how a claim on your own site becomes a fact about your business.

The mechanics of getting the markup right — graph design, @id references, avoiding duplicate entities — are the subject of my technical SEO work, and it is the single most common thing I find broken on sites that were otherwise built competently.

How to tell whether any of this is working

For two years there was no official data here at all. That changed on June 3, 2026, when Google launched Search Generative AI performance reports in Search Console, giving site owners a dedicated view of impressions inside AI Overviews, AI Mode and the generative features in Discover. Two caveats worth knowing before you go looking: the report shows impressions rather than clicks, and it rolled out to a subset of sites rather than everyone at once, so it may not be in your property yet.

Search Console

Check for the generative AI report. If it is there, it is the only first-party measure of AI visibility that exists. If it is not, use the standard performance report and watch for impressions rising while clicks flatten, which is the signature of being summarized.

Referral traffic

In analytics, segment referrals from assistant domains. The volume will look trivially small against organic. Judge it on conversion rate instead, where it usually is not small at all, because the visitor arrived pre-qualified by a recommendation.

Manual prompt checks

Ask two or three assistants the questions your customers ask, monthly, in a fresh session, and record who gets named. Crude, but it is the only way to see the competitive picture and it costs ten minutes.

What does not work

Every platform shift produces a market in shortcuts. These are the ones currently being sold to small businesses.

llms.txt as a strategy

Google's guidance says you do not need to create machine-readable files, AI text files or Markdown to appear in Search or its AI features, and an Ahrefs study of 137,000 domains found the overwhelming majority of llms.txt files were never fetched by anything. Cheap to add if it amuses your developer. Not a plan.

Publishing volume to feed the machines

Forty thin articles produce forty thin articles. These systems are deciding who to trust, and mass-produced commodity content is evidence against you, not for you.

Claiming your content is "100% human-written"

No retrieval system checks for this, no ranking system rewards it, and it is usually not true. Assess the work on accuracy and usefulness. I use AI in drafting here and say so.

Keyword stuffing, reheated

Repeating a phrase does nothing when the retrieval is semantic. Entity clarity and answer quality are the levers; term frequency has not been the lever for a long time.

Blocking the crawlers, then wondering

Roughly half of small businesses restrict AI crawlers, mostly over content ownership. That is a legitimate position to hold deliberately. It is not compatible with also wanting to appear in AI answers.

Buying "AI visibility" scores

The tools tracking assistant mentions are useful for monitoring a channel you already have. None of them create one, and most of what they charge for is a prompt run on a schedule.

The AI search readiness checklist

Run this against your own site. Every unchecked line is a reason you are not in the answer.

  • Robots.txt audited line by line. No accidental Disallow on retrieval crawlers, and training crawlers allowed or blocked on purpose.
  • Edge and firewall checked. CDN, WAF and host bot rules verified as not challenging AI agents before robots.txt is read.
  • Pages fetched as each bot. Real 200 responses with real HTML, confirmed rather than assumed.
  • Rendered HTML verified. URL Inspection shows your service copy, pricing and FAQs in the rendered output.
  • Nothing critical behind JavaScript. Tabs, accordions and dynamic panels contain nothing that is not also in the HTML.
  • Snippet controls reviewed. No stale nosnippet, data-nosnippet or restrictive max-snippet on commercial pages.
  • Indexing status confirmed per page. No leftover noindex, no canonical pointing somewhere unexpected.
  • Question-shaped headings. Section headings phrased the way a customer would ask, answered directly beneath.
  • Answer-first paragraphs. First two sentences of each section resolve the question without needing the section above it.
  • Self-contained passages. No orphan pronouns, no back-references, subject named where it matters.
  • Specifics on the page. Real prices, timeframes, thresholds and process detail rather than hedged ranges.
  • Something not available elsewhere. Original data, method, or first-hand experience a synthesizer cannot assemble without you.
  • One clean entity graph. A single Organization or LocalBusiness definition sitewide, validated, no duplicates from a second plugin.
  • Markup matching visible text verbatim. Especially FAQ answers.
  • Named author with a real bio. Credentials that can be checked off-site.
  • NAP identical everywhere. Footer, Google Business Profile, every directory, character for character.
  • Off-site corroboration growing. Reviews, listings and mentions that confirm what your site claims.
  • Measurement in place. Search Console checked for the generative AI report, assistant referrals segmented, monthly prompt checks logged.

Questions about AI-driven search

Do I need to do anything special to appear in Google's AI Overviews or AI Mode?

No. Google's documentation states that there are no additional requirements to appear in AI Overviews or AI Mode and no special optimizations necessary. The same fundamentals apply: content worth citing, a page people can use, technical accessibility, and structured data that matches what is visible. The practical difference is that weaknesses which used to cost you a few positions now remove you from the answer entirely, so the fundamentals are less forgiving than they were.

What is the difference between an AI training crawler and an AI retrieval crawler?

Training crawlers such as GPTBot and ClaudeBot collect content that may be used to train future models. Retrieval crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages to answer a question somebody is asking right now, and they are the ones that produce citations. Blocking a training crawler is a content licensing decision with no effect on whether you get cited. Blocking a retrieval crawler removes you from the answer. Decide the two separately.

Does blocking Google-Extended remove me from AI Overviews?

No, and this is the most common misunderstanding in the whole topic. Google-Extended is a robots.txt control token with no crawler of its own. It governs whether content already crawled by Googlebot can be used for training and grounding in Gemini apps, and Google states it does not affect a site's inclusion in Search or its AI features. AI Overviews and AI Mode are fed by the ordinary Google index. If you specifically want out of them, that control is in Search Console under Search generative AI, and Google says using it does not affect your standard rankings.

Does adding an llms.txt file help?

There is no evidence that it does. Google's guidance says you do not need machine-readable files, AI text files or Markdown to appear in Search or its AI features, and an Ahrefs analysis of 137,000 domains found the overwhelming majority of llms.txt files were never fetched by any crawler. It costs almost nothing to publish one if your developer wants to, but treating it as your AI search strategy means spending your attention on a file nothing reads instead of the four gates that decide the outcome.

Why does an assistant cite my competitor when my page ranks higher?

Usually one of three reasons. Their passage is more extractable, meaning the specific answer sits in one self-contained chunk while yours is spread across a page. Their entity is clearer, so the system is more confident about who they are and where they operate. Or your page is technically less available than you think, because a crawler is being challenged at the edge or the content renders client-side. Check availability first, structure second, and content quality third, because that is the order the failures actually occur in.

How do I tell whether AI search is sending me anything?

Three checks, cheapest first. Look in Search Console for the Search Generative AI performance report, launched on June 3, 2026, which shows impressions inside AI Overviews and AI Mode, though not clicks and not yet for every site. Segment referral traffic from assistant domains in your analytics and judge it on conversion rate rather than volume. And once a month, ask two or three assistants the questions your customers ask and record who gets named. The third one is crude but it is the only view of the competitive picture you can get.

Does using AI to write content hurt my chances of being cited?

The tool is not the issue. Commodity content is. Google's guidance targets content produced primarily to attract search traffic and mass automation across topics, not the use of software in drafting. A page built on real expertise, checked for accuracy, and carrying specifics that are not on the other results will be cited whether or not a model helped write it. A page assembled from a prompt with nothing original in it will not, and volume makes that worse rather than better. I use AI in drafting on this site and say so openly.

How long does this take to show results?

Access and rendering fixes can register within weeks, because they only need a recrawl. Structure and content changes follow on the recrawl cycle after that. Becoming a regularly named source in a competitive category usually takes three to six months, because you are also waiting on reviews to accumulate and on other sites to mention you, and neither of those is under your direct control. Progress is qualitative before it is quantitative: you start appearing at all, well before you start appearing first.

Want to know which gate you are failing?

I will run the checklist above against your site and tell you where it actually breaks. If it is three things you can fix yourself in an afternoon, that is what I will tell you.

Get an SEO audit

You can also see what this has done for other businesses, read about my Tucson SEO services, or get in touch and call me directly at (520) 207-6000. David Cragg, Tucson SEO. In search since 1990.

Sources: Google Search Central documentation on AI features and your website, the Search Central blog on succeeding in AI search and on Search Generative AI performance reports, plus published crawler documentation from OpenAI, Anthropic and Perplexity. Crawler names and platform controls change frequently; check the vendor documentation directly rather than anyone's summary, including mine. Written by David Cragg with AI assistance in drafting, reviewed and verified before publication.