Optimizing for AI-Driven Search: The Four Gates Small Business Sites Fail

Most sites that never get cited in an AI answer aren't losing on content. They fail earlier, at a gate the owner doesn't know exists.

A customer used to type "junk removal Tucson" and scan ten blue links. Now many of them ask an assistant for a recommendation and get two or three names. If your business isn't one of them, that search is over before your site was ever considered.

The usual reaction is to assume this needs a whole new discipline. It mostly doesn't. Google says no special optimizations are needed to appear in its AI features. What changed is the path a page travels before it is quoted. It now has to pass four gates in order, and small business sites usually fail the first two while everyone argues about the fourth.

This page covers the site-side work: what to check, in what order, and what the platforms document. What Google published about AI search, the tactics it says to skip, and how to measure the results are covered in how to improve AI search visibility.

Written by David Cragg, Tucson SEO consultant, in search since 1990. Published May 16, 2026. Last reviewed September 17, 2026.

On this page: What changed · The four gates · Crawler access · Rendered HTML · Passages · Entity clarity · What doesn't work · Checklist · Questions

What changed, and what didn't

Classic search matched a query to a ranked list of pages and handed you the list. AI-driven search does more steps. Google's generative AI guide says its systems retrieve fresh pages from the index to ground an answer, and use query fan-out to split one question into several related searches. ChatGPT, Claude and Perplexity work in broadly similar ways with their own crawlers and indexes. Three practical consequences follow.

The citation matters more than the position. The page quoted inside the answer is the one people see. A page in fourth place can be quoted while the first result is skipped.

One question becomes several. A page that answers the related sub-questions clearly, in one place, gives the system more to use than a page that scatters the answer across three sections and a video.

The machine reads your HTML, not your design. A crawler sees text, structure and markup. A design choice that hides text from the HTML is a direct cost.

What didn't change: the fundamentals. Google's own list is crawling allowed in robots.txt and by your CDN or host, important content available as text, a good page experience, and structured data that matches the visible page.

The four gates a page has to pass

Work from the top down. A page blocked at the firewall never gets as far as its own robots.txt, and rewriting the content can't fix a fetch that never happened.

Gate 1: crawler access

AI companies run different crawlers for different jobs, and blocking each one costs something different. Training crawlers collect content that may be used to train future models. Search crawlers build the index an assistant searches when it answers. User-triggered fetchers load a page because someone asked about it right now. Blocking a training crawler is a licensing decision. Blocking a search crawler or a user fetcher takes you out of the answers.

Most small business sites that are invisible to AI assistants never made that decision. A security plugin, a hosting preset or a CDN rule made it for them.

The Google-Extended row is where people get it backwards. It has no user agent of its own; Googlebot does the crawling. According to Google's crawler documentation, it governs training and Gemini grounding, and it doesn't affect whether you appear in Search. Blocking it to hide from AI Overviews does nothing for that goal. The switch that does control AI Overviews is a Search Console setting, covered on the AI search visibility page.

For Anthropic's three bots, the Claude help center documents each one, and each needs its own robots.txt rule. Search Engine Journal's comparison covers how OpenAI and Perplexity treat their user-triggered fetchers.

• Read yoursite.com/robots.txt line by line. Look for any Disallow aimed at OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User or PerplexityBot. "Protect my content" toggles in plugins add these without asking.
• Check the layer above robots.txt. CDNs and managed hosts often ship bot-blocking defaults that stop crawlers at the edge before robots.txt is ever read. A 403 from the firewall doesn't show up in most SEO tools.
• Decide on training separately from search. Allowing OAI-SearchBot while blocking GPTBot is a coherent position: appear in answers, stay out of training. Blocking both because a plugin offered one checkbox isn't.
• Fetch your own pages as a bot. A request such as curl -I -A "OAI-SearchBot" https://yoursite.com/ shows the status code a crawler gets. A 403 or a challenge page means nothing further down this list matters yet.

This isn't theoretical. When I ran text-only fetches against this site in September 2026, one of my own blog category archives answered with a "JavaScript is required" redirect page instead of its content. A browser never shows that. A crawler that doesn't run JavaScript sees nothing else.

Gate 2: what is actually in the rendered HTML

Answer engines quote text they can read in the response. Several common build patterns hide text from them while leaving it perfectly visible to a person, which is why this failure is so hard to spot by looking at the site.

Content injected by JavaScript

Service descriptions, prices and FAQs loaded by scripts after the page arrives. Google can process JavaScript if it isn't blocked, but other crawlers may do far less. Use URL Inspection in Search Console and read the rendered HTML, not just the page source.

Tabs and accordions that load on click

Native details and summary elements keep the answer in the HTML, which is how the FAQ on this page is built. A widget that fetches panel content only when someone clicks doesn't.

Snippet controls

nosnippet, data-nosnippet and max-snippet limit what Google can show from a page, AI features included. Sites inherit these from old plugin settings and never look again.

Text set inside images

Price lists, hours and service tables saved as graphics can't be quoted. Google's own advice is to make important content available as text. If a number matters to a buying decision, it belongs in HTML.

A leftover noindex

A staging setting that survived launch. A page that isn't eligible for Search isn't eligible to be an AI source. Check the indexing status of every page that sells something, not just the homepage.

Slow or unstable pages

Fetches time out. The same page speed work that helps visitors also decides whether a crawler gets a complete page or gives up.

The two-minute version of this audit: load the page with JavaScript turned off and see what's left. Whatever disappears is content you're hoping a machine will guess.

Gate 3: writing passages that can be extracted

This is where most of the real work is. It isn't chunking; Google says it doesn't need pages broken into tiny pieces, and you shouldn't write for a machine. It's about writing each section so a person who lands in the middle of the page can still follow it. A person skimming and a system pulling a passage read the same way.

The hard truth: if your page says what twenty other pages already say, an answer engine has no reason to cite yours. It can write that summary without you. For a service business, the lasting advantage is real prices, real process detail, real problems and a named person who did the work. Google calls that non-commodity content, and helping owners get it onto the page is most of what content creation means now.

Gate 4: entity clarity

When several sources answer a question well enough, the tiebreaker is how confident the system is about who is speaking. Structured data won't get you into an AI answer by itself, and Google says no special markup is needed. What markup does is remove doubt about who you are, where you work and what you sell. Doubt is what gets a source dropped.

• One business definition, sitewide, in a single graph. Duplicate or contradictory blocks are worse than none. Two plugins that each publish an Organization block is the most common version.
• Markup that matches the visible text. FAQ markup whose answers aren't on the page is the fastest way to have your structured data ignored.
• A named author with checkable credentials. A real person with a bio page and a history that others can confirm, not "the team."
• Name, address and phone identical everywhere. Site, Google Business Profile, every directory. A system matching businesses treats a mismatch as a possible second business.
• Confirmation from other sites. Directory listings, reviews and press mentions are how a claim on your own site becomes an accepted fact about your business.

I found all of these problems on my own site. After I moved the practice from Temecula to Tucson, an old block of structured data kept printing the retired Temecula phone numbers on every page, right next to the correct Tucson graph. The theme was also printing a second set of social sharing tags. Neither problem shows up in the browser. Both show up in the page source, which is why I check changes against the live source instead of trusting a plugin's dashboard. Getting that markup right is part of my technical SEO work, and it's the most common thing I find broken on otherwise well-built sites.

How to tell whether it's working

Fixes to access and rendering show up as soon as the pages are crawled again. Whether that turns into visibility is a measurement question, and Google now provides a report for its own AI features. How to read it, and how to track the assistants Google can't see, is covered in the measurement section of my AI search visibility guide.

What doesn't work on the site side

Google keeps its own list of GEO tactics to skip, and I go through that list on the companion page. These are the site-side habits I see most often on small business sites.

Blocking the crawlers, then wondering

Keeping your content out of AI systems is a legitimate choice if you make it on purpose. It doesn't work alongside wanting to be recommended by those same systems.

A "100% human-written" badge

No answer engine checks for it, no ranking system rewards it, and it is often not true. Judge content on accuracy and usefulness. I use AI tools in drafting here and say so.

Keyword stuffing with a new name

Repeating a phrase does nothing when the system matches on meaning. Clear entities and good answers are what count.

Hiding the details for a sales call

Leaving prices and process off the page to force a phone call also leaves an answer engine nothing specific to quote. It will quote a competitor who published those details.

The AI search readiness checklist

Run this against your own site. Each unchecked line is a reason you may be missing from the answer.

• Robots.txt read line by line. No accidental blocks on search crawlers or user fetchers, and training crawlers allowed or blocked on purpose.
• Firewall and CDN checked. Bot protection verified as not challenging AI agents before robots.txt is read.
• Pages fetched as each bot. Real 200 responses with real HTML, confirmed rather than assumed.
• Search Console generative AI control set to include. Checked on the domain property, since child properties inherit it.
• Rendered HTML verified. URL Inspection shows your service copy, prices and FAQs.
• Nothing important loaded only on click. Tabs and accordions contain nothing that isn't already in the HTML.
• Snippet controls reviewed. No leftover nosnippet or restrictive max-snippet on pages that sell.
• Indexing confirmed page by page. No stray noindex, and no canonical tag pointing somewhere unexpected.
• Question-shaped headings. Phrased the way a customer asks, and answered right below.
• Sections that stand alone. No "as mentioned above," and the subject named where it matters.
• Specifics on the page. Real prices, timeframes and process detail.
• Something only you could write. Your own data, method or experience.
• One clean business graph. A single Organization or LocalBusiness definition, validated, with no duplicate from a second plugin.
• Markup matching the visible text. Especially FAQ answers.
• Named author with a real bio. Credentials someone can check elsewhere.
• Name, address and phone identical everywhere. Character for character.
• Growing confirmation from other sites. Reviews, listings and mentions that back up what your site says.

Sources

• AI features and your website, Google Search Central
• Optimizing your website for generative AI features on Google Search, Google Search Central
• Google's common crawlers, including Google-Extended, Google Crawling Infrastructure
• Anthropic crawler documentation, Claude Help Center
• How Anthropic, OpenAI and Perplexity split their bots, Search Engine Journal

Crawler names and platform controls change often. Check each vendor's documentation directly rather than anyone's summary, including mine. Written by David Cragg with AI help in drafting, then reviewed and checked against the sources above.