GEO Crawl Audit: How AI Crawlers Actually Treat a 370-Page Store

Published 2026-10-19 · 956231 Studio · Method: full-server access logs, sitemap and on-page inspection, 370 URLs · All counts below are measured, not estimated

Short answer: AI crawlers are not ignoring small stores — on this site GPTBot made 87 requests in 9 hours while Googlebot made 29 requests across 12 unique URLs and Bingbot made zero. The bottleneck was never crawl permission. It was that every page the crawlers reached averaged 253–307 words of sales copy, carried no data, and therefore gave a citation engine nothing to quote. Fixing access took one afternoon. Fixing citable substance is the actual work.

Why this audit exists

Most GEO advice is written by people selling GEO tools, and it tends to assume the failure mode is technical: no llms.txt, blocked AI bots, missing schema. We wanted to know what the failure actually looks like on a store that already has all of that in place, so we instrumented a live 370-page e-commerce site, read its raw nginx logs, and counted what each crawler did.

The site in question sells 27 downloadable Python toolkits, runs 100% locally with no API key, and ships with llms.txt, llms-full.txt, explicit allow rules for ten AI crawlers, and FAQPage schema on 101 pages. If the standard checklist were the answer, this site should have been cited. It was not.

What we measured

SignalWindowMeasured
GPTBot requests9 hours87
Googlebot requests9 hours29
Googlebot unique URLs9 hours12
Bingbot requests9 hours0
PerplexityBot requests9 hours0
ClaudeBot requests9 hours0
Googlebot fetches of sitemap.xml9 hours0
Recorded on-site events9.5 days945
Events attributed to search engines9.5 days3
Events attributed to AI assistants9.5 days0
CTA clicks9.5 days1

The crawl distribution is the finding

Two numbers deserve attention. First, GPTBot is 3× more active than Googlebot on this site, and it goes deeper: 43 of its 87 requests were aimed at task-specific landing pages, and dozens more walked individual audit result pages one by one. OpenAI's crawler was doing exactly what the GEO advice says it should do.

Second, Googlebot never fetched the sitemap, and spent 4 of its 29 requests on /api/event — an analytics endpoint that returns no indexable content. Nearly 15% of the crawler budget was consumed by a URL that should never have been crawlable.

What this rules out: the hypothesis that AI engines cannot see small sites. They can. Within a single working day, GPTBot-class crawlers read more of this site than Google did. Access was never the wall.

Why being crawled did not turn into being cited

Crawl access and citation are different problems, and the gap between them is content substance. We measured every page type the crawlers were reaching:

Page typePagesAvg wordsLongestTables
Task landing pages123073700
Audit reports532532760
Auto-generated result pages256~5606410

Not one of the 53 reports exceeded 800 words. Not one carried a table. A citation engine asked to answer "how do AI crawlers treat a small store" cannot quote a 253-word product summary, because a product summary contains no measurement, no method, and no conclusion. It contains sales copy.

This is the failure mode that the standard GEO checklist does not name: you can pass every technical gate and still have nothing to say.

Three concrete defects we found and fixed

1. The sitemap contradicted itself

370 URLs were submitted. 124 of them carried a noindex directive on the page itself. The sitemap was telling Google "please index this" and the page was telling Google "do not index this" about the same URL. On top of that, 257 of the 370 entries — 69% of the file — were auto-generated result pages whose titles collapsed to 211 unique strings.

Fix: the sitemap was split into four tiered files by content value, and every noindex URL was removed. Submitted URLs went from 370 to 245, with thin result pages segregated into their own low-priority file.

2. crawler budget was wasted on an API endpoint

Googlebot fetched /api/event four times. That endpoint accepts a POST and returns a JSON acknowledgement; there is nothing there to index. Each such request is crawl budget spent on nothing.

Fix: Disallow: /api/ was added to robots.txt.

3. hreflang was pointing at pages that are not translations

The homepage declared hreflang alternates to /de/, /uk/ and /ru/. But those pages are not German, British-English and Russian versions of the homepage — they are three separate long-form articles on unrelated regulatory topics (German product liability, UK IEN deadlines, Russian marketplace listing rules).

hreflang means "same content, different language." Pointing it at different content is a misuse that search engines are entitled to ignore, and in aggregate it dilutes the signal. Fix: the false equivalences were removed and the declaration collapsed to a single self-referential x-default.

What the competitors actually publish

We checked llms.txt availability on five well-funded companies in the AI-tooling and digital-product space. The result is not what the GEO vendor narrative predicts:

Companyllms.txtSizeWhat it contains
n8nYes1.1 KB4 research reports + Markdown twins
SintraNo——
LindyNo——
ZapierNo——
GumroadNo——
This site (before)Yes11.0 KBProduct catalogue + prices + buy links

Only one of five publishes the file at all, and it is the smallest one in the set — four links, not forty. Its purpose is to point AI tools at research reports and at Markdown versions of those reports, so ingestion costs the model as little as possible.

The site being audited had a llms.txt more than 9× larger, and it was the wrong type of document: a machine-readable catalogue of things for sale. A catalogue answers "what can I buy." A citation engine is answering "what is true." Those are different questions, and only the second one earns a reference.

What we changed

Results you can verify in the logs

Everything above is checkable from outside. The tiered sitemaps are public at /sitemap-core.xml, /sitemap-reports.xml, /sitemap-geo.xml and /sitemap-audit.xml. The rebuilt index is at /llms.txt, and any task landing page returns a Markdown twin by appending .md to its URL.

What we explicitly do not claim: that these fixes produced traffic. Crawl acceptance is same-day verifiable; ranking and citations are not. This audit reports the state of access and the state of content substance. Referral outcomes belong in a separate measurement taken after a full re-crawl cycle.

Frequently asked questions

Do AI crawlers actually visit small e-commerce sites?
Yes, and often more aggressively than Google. On a 370-page store we measured GPTBot making 87 requests in a 9-hour window, versus 29 from Googlebot across only 12 unique URLs. The constraint is not crawl access. It is whether the page contains something worth citing.
Does having a llms.txt file get you cited by AI search engines?
Not by itself. Of five well-funded competitors checked, only n8n published one, at 1.1 KB listing four research reports plus their Markdown twins. A llms.txt that lists product pages and prices is a catalogue; AI engines cite sources that carry data, method and conclusions.
Why would a sitemap hurt indexing?
When it contradicts itself. A sitemap that lists URLs carrying a noindex directive sends two opposite signals about the same page. In our audit 124 of 370 submitted URLs were noindex, and 69% of the sitemap was thin auto-generated pages averaging 500–640 words. We split the sitemap into four tiered files and dropped every noindex URL.
How long before a crawl fix shows up in referral traffic?
Crawl-rate changes can appear within hours because crawlers pick up sitemap and robots changes quickly. Ranking and citation changes take weeks. We treat the two separately: crawl acceptance is verifiable in server logs same-day, referral traffic is not attributable until a full re-crawl cycle has completed.

Get the GEO audit kit

27 tested kits · Python scripts · MIT licensed · runs 100%% locally · buy once, instant download · sold via Gumroad (merchant of record)

Get it now · $27.55 Compare kits →