Can AI-Driven Search Read Your Small Business Website?

A customer used to type "junk removal Tucson" and scan ten blue links. Now plenty of them ask ChatGPT, Claude, Perplexity or Google's AI Mode for a recommendation and get two or three names back. If your business isn't one of them, the search ended before your site was ever considered.

The usual reaction is to buy a new kind of optimization. Google says no special optimizations are needed to appear in its AI features, and in my experience that's closer to the truth than most of what gets sold as GEO. What changed is the path a page travels before AI-driven search can quote it. This article covers the first stretch of that path: whether AI crawlers can reach your pages and read them. It's where small business sites fail most often, and it's the cheapest part to fix.

Two other articles cover the rest. What Google published about its AI features, the tactics it says to skip and how to measure results are in how to improve AI search visibility. Finding and fixing structured data that contradicts itself is in two schema blocks saying two different things.

What Did I Find When I Tested My Own Site?

The same failures I look for on client sites. In September 2026 I ran text-only fetches against this site, the way a crawler that doesn't run JavaScript sees it. One of my blog category archives answered with a "JavaScript is required" page instead of its content. A browser never shows that page. A crawler sees nothing else.

In October 2026 I tried again through an AI assistant's web tool. The homepage and several articles came back with their full text. Most of the service pages, the blog index and a few older articles came back as a firewall page reading "You are being redirected..." and nothing more. The host's firewall treats a request it can't verify as a bot and challenges it. People never notice, because their browsers pass the challenge in a fraction of a second.

Nothing on those service pages was wrong. The assistant simply never got to read them. That's the problem this article is about, and it's the first thing I check on any site that says it's invisible in AI answers.

What Changed When People Started Asking Assistants Instead of Searching?

Classic search matched a query to a ranked list and handed you the list. AI search does more steps. Google's generative AI guide says its systems pull fresh pages from the index to ground an answer, and split one question into several related searches. ChatGPT, Claude and Perplexity work in broadly similar ways with their own crawlers and indexes. Three practical consequences follow.

• The citation matters more than the position. The page quoted inside the answer is the one people see. A page in fourth place can be quoted while the first result is skipped.
• One question becomes several. A page that answers the related questions clearly, in one place, gives the system more to use than one that scatters the answer across three sections and a video.
• The machine reads your HTML, not your design. A crawler sees text, structure and markup. A design choice that keeps text out of the HTML costs you directly.

What didn't change is the foundation. Google's own list is crawling allowed in robots.txt and by your host or CDN, important content available as text, a good page experience, and structured data that matches the visible page.

Which Four Checks Does a Page Have to Pass Before an Assistant Can Quote It?

Work from the top down, because a failure at one step makes everything below it irrelevant.

1. Access. Can the crawler fetch the page at all? Firewall and CDN rules, robots.txt and status codes decide it. Most silent failures live here, and it's the cheapest step to fix.
2. Rendering. Is the content in the HTML that comes back? Text injected late by JavaScript, loaded only on a click or set inside an image may never be available.
3. Extraction. Can a clear, self-contained answer be lifted from the page? This is a writing and structure problem, not a technical one.
4. Identity. Is it clear who wrote the page, which business it is and where it works? When several sources are good enough, clarity decides which one gets named.

Which AI Crawlers Should You Allow, and Which Can You Block?

AI companies run different bots for different jobs, and blocking each one costs something different. Training crawlers collect content that may be used to train future models. Search crawlers build the index an assistant searches when it answers. User-triggered fetchers load a page because someone asked about it right now. Blocking a training crawler is a licensing decision. Blocking a search crawler or a user fetcher takes you out of the answers.

What does Google-Extended actually control?

This is the row people get backwards. Google-Extended has no user agent of its own, because Googlebot does the crawling. Google's crawler documentation says it governs whether your content can train Gemini models and ground answers in the Gemini apps, and that it doesn't affect whether you appear in Search. Blocking it to stay out of AI Overviews does nothing for that goal. The setting that does govern Google's AI features lives in Search Console, and I cover it in the AI search visibility article.

How fast do robots.txt changes take effect?

Not instantly. OpenAI says a robots.txt change can take about 24 hours to register for search. Other crawlers re-read the file on their own schedules. Change the file, wait a day, then test again before deciding it didn't work.

Is Your Firewall Blocking AI Crawlers Before robots.txt Is Read?

Robots.txt is a request. A firewall is a wall. Most hosting plans and CDNs sold to small businesses now ship with bot protection switched on, and it runs before robots.txt is ever read. A crawler stopped at the edge gets a 403 or a challenge page, and most SEO tools never report it, because the tools get challenged too or never try.

• Fetch your own pages as a crawler. A request such as curl -I -A "OAI-SearchBot" https://yoursite.com/ shows the status a bot with that name gets. A 403, a 503 or a page that says "You are being redirected" means the firewall stopped it.
• Treat a clean result with some suspicion. That test only checks rules based on the user-agent name. Many firewalls also score the request's IP address, so a real crawler can be blocked even when your test passes.
• Allow by verified identity, not by name. Anyone can call their scraper Googlebot. Google and OpenAI publish the IP ranges their crawlers use, and a good firewall lets you allow verified bots by those ranges. Allowing a user-agent string alone also lets in the impostors.
• Check every layer. Host firewall, CDN and security plugin can each run their own bot rules. On WordPress sites I take over, I regularly find two or three layers stacked on top of each other, set up years apart by different people.
• Recheck after any security change. A plugin update or a new firewall preset can quietly undo an allowance you set months ago.

Is the Answer Actually in the HTML the Crawler Receives?

Answer engines quote text they can read in the response. Several common build patterns hide text from them while leaving it perfectly visible to a person, which is why this failure is so hard to spot by looking at the site.

• Content injected by JavaScript. Service descriptions, prices and FAQs loaded by a script after the page arrives. Google can process JavaScript if it isn't blocked. Other crawlers may do much less.
• Tabs and accordions that load on click. Native details and summary elements keep the answer in the HTML, which is how the questions at the bottom of this page are built. A widget that fetches panel content only when clicked doesn't.
• Snippet controls. nosnippet, data-nosnippet and max-snippet limit what Google can show from a page, AI features included. Sites inherit them from old plugin settings and never look again.
• Text set inside images. Price lists, hours and service-area maps saved as graphics can't be quoted. If a number matters to a buying decision, it belongs in HTML.
• A leftover noindex. A staging setting that survived launch. A page that isn't eligible for Search isn't eligible to be an AI source either.
• Slow or unstable pages. Fetches time out. The speed work that helps visitors also decides whether a crawler gets a complete page or gives up.

The two-minute version of this check: load the page with JavaScript turned off and see what's left. Whatever disappears is content you're hoping a machine will guess. Finding which of these a site has, and fixing it once in the theme instead of page by page, is a large part of my technical SEO work.

How Do You Write a Passage an Assistant Can Lift?

Google says pages don't need to be broken into tiny chunks, and you shouldn't write for a machine. Write each section so a person who lands in the middle of the page can still follow it. A person skimming and a system pulling a passage read the same way.

If your page says what twenty other pages already say, an answer engine has no reason to cite yours. It can write that summary without you. For a service business, the lasting advantage is real prices, real process, real problems and a named person who did the work. Getting that onto the page, in the owner's words, is most of what content creation means now.

Where Does Structured Data Fit In?

It removes doubt, which is different from getting you in. Google says no special markup is needed for its AI features. What markup does is make it unambiguous who you are, where you work and what you sell, and doubt is what gets a source dropped when several are good enough. The failure I find most is two or three blocks describing the business differently, one from the theme, one from the SEO plugin and one from a snippet somebody pasted years ago. How to find and clean those up is in the duplicate schema article.

How Do You Know Whether AI Crawlers Are Reaching You?

Start with your server logs, because they record what actually arrived. Count hits from OAI-SearchBot, Claude-SearchBot and PerplexityBot separately. Steady visits mean the door is open. None at all for months means it's shut somewhere, usually at the firewall. Most managed hosts let you download raw access logs from the control panel, even when their dashboards don't show bot traffic.

Whether those visits turn into visibility is a separate measurement question. Google now reports on its own AI features in Search Console, and I explain how to read it, and how to track the assistants Google can't see, in the measurement section of the AI search visibility article.

What Do Tucson Small Business Sites Get Wrong Most Often?

These come from sites I've worked on here, not from a published study.

The host's firewall is on and nobody knows it. A lot of Tucson trades bought hosting bundled with the website, and the bot protection came with it, set to its strictest default. The owner has never seen the setting, and the web designer who chose the plan has moved on.

The service area lives in a picture. A map graphic shaded across Oro Valley, Marana and Vail tells an assistant nothing. A sentence naming the towns you serve, on a page it can fetch, tells it everything it needs.

Prices and process are left off on purpose. Owners hide them to force a phone call. An assistant asked "what does a Tucson AC tune-up cost" can't quote a page that won't say, so it quotes the competitor who did.

What's on the AI Crawler Access Checklist?

Run this against your own site. Each line you can't check off is a reason you may be missing from the answer.

• Robots.txt read line by line, with no accidental blocks on search crawlers or user fetchers.
• Training crawlers allowed or blocked on purpose, not by a plugin preset.
• Host firewall, CDN and security plugin each checked for bot challenges.
• Key pages fetched as each major crawler, with real 200 responses and real HTML.
• Verified crawler IP ranges allowed where the firewall supports it.
• Rendered HTML confirmed in Search Console's URL Inspection, including service copy, prices and FAQs.
• Nothing important loaded only on click or only by script.
• No leftover nosnippet, restrictive max-snippet or noindex on pages that sell.
• Prices, hours and service areas written as text, not images.
• Headings phrased as the customer's question and answered right below.
• One business definition in the structured data, with no duplicate from a second plugin.
• Server logs checked for search crawler visits after every change.

What Do Clients Say About Working With David?

Four of the ten reviews on my Google profiles, quoted in full. Most were left on the Temecula SEO profile during the Southern California years, before the practice moved to Tucson in 2022. The two labeled SEO Tucson name the new business in the review itself.

Where Does the Information on This Page Come From?

Crawler names and platform controls change often, so check each vendor's documentation directly rather than anyone's summary, including mine. The firewall findings and the Tucson observations come from my own work and are labeled that way.

• Google Search Central: AI features and your website. Why no special optimization is needed, and what is.
• Google Search Central: Optimizing your website for generative AI features. Grounding, query fan-out and the foundation list.
• Google crawling infrastructure: Google's common crawlers. What Google-Extended does and doesn't control.
• OpenAI: Overview of OpenAI crawlers. GPTBot, OAI-SearchBot, ChatGPT-User, published IP ranges and the 24-hour note.
• Claude Help Center: Anthropic's crawlers. ClaudeBot, Claude-SearchBot and Claude-User.
• Search Engine Journal: how Anthropic, OpenAI and Perplexity split their bots. How each treats user-triggered fetchers.
• Google Search Central: Robots meta tag, data-nosnippet and X-Robots-Tag. The snippet controls that also limit AI features.

Written by David Cragg, Tucson SEO consultant, in search since 1990. I check every change I make against the live page source, which is how I found my own firewall turning assistants away.

On how this site is written: I use AI tools to draft and to check my own work. AI does not supply the facts about your business, it does not invent sources, and nothing goes up here that I have not read first. If you find a claim on this site you can't verify, call me and I'll fix it or pull it.

First published May 16, 2026. Page last reviewed October 27, 2026 by David Cragg.