Short answer: AI crawlers are not ignoring small stores — on this site GPTBot made 87 requests in 9 hours while Googlebot made 29 requests across 12 unique URLs and Bingbot made zero. The bottleneck was never crawl permission. It was that every page the crawlers reached averaged 253–307 words of sales copy, carried no data, and therefore gave a citation engine nothing to quote. Fixing access took one afternoon. Fixing citable substance is the actual work.
Most GEO advice is written by people selling GEO tools, and it tends to assume the failure mode is technical: no llms.txt, blocked AI bots, missing schema. We wanted to know what the failure actually looks like on a store that already has all of that in place, so we instrumented a live 370-page e-commerce site, read its raw nginx logs, and counted what each crawler did.
The site in question sells 27 downloadable Python toolkits, runs 100% locally with no API key, and ships with llms.txt, llms-full.txt, explicit allow rules for ten AI crawlers, and FAQPage schema on 101 pages. If the standard checklist were the answer, this site should have been cited. It was not.
| Signal | Window | Measured |
|---|---|---|
| GPTBot requests | 9 hours | 87 |
| Googlebot requests | 9 hours | 29 |
| Googlebot unique URLs | 9 hours | 12 |
| Bingbot requests | 9 hours | 0 |
| PerplexityBot requests | 9 hours | 0 |
| ClaudeBot requests | 9 hours | 0 |
| Googlebot fetches of sitemap.xml | 9 hours | 0 |
| Recorded on-site events | 9.5 days | 945 |
| Events attributed to search engines | 9.5 days | 3 |
| Events attributed to AI assistants | 9.5 days | 0 |
| CTA clicks | 9.5 days | 1 |
Two numbers deserve attention. First, GPTBot is 3× more active than Googlebot on this site, and it goes deeper: 43 of its 87 requests were aimed at task-specific landing pages, and dozens more walked individual audit result pages one by one. OpenAI's crawler was doing exactly what the GEO advice says it should do.
Second, Googlebot never fetched the sitemap, and spent 4 of its 29 requests on /api/event — an analytics endpoint that returns no indexable content. Nearly 15% of the crawler budget was consumed by a URL that should never have been crawlable.
What this rules out: the hypothesis that AI engines cannot see small sites. They can. Within a single working day, GPTBot-class crawlers read more of this site than Google did. Access was never the wall.
Crawl access and citation are different problems, and the gap between them is content substance. We measured every page type the crawlers were reaching:
| Page type | Pages | Avg words | Longest | Tables |
|---|---|---|---|---|
| Task landing pages | 12 | 307 | 370 | 0 |
| Audit reports | 53 | 253 | 276 | 0 |
| Auto-generated result pages | 256 | ~560 | 641 | 0 |
Not one of the 53 reports exceeded 800 words. Not one carried a table. A citation engine asked to answer "how do AI crawlers treat a small store" cannot quote a 253-word product summary, because a product summary contains no measurement, no method, and no conclusion. It contains sales copy.
This is the failure mode that the standard GEO checklist does not name: you can pass every technical gate and still have nothing to say.
370 URLs were submitted. 124 of them carried a noindex directive on the page itself. The sitemap was telling Google "please index this" and the page was telling Google "do not index this" about the same URL. On top of that, 257 of the 370 entries — 69% of the file — were auto-generated result pages whose titles collapsed to 211 unique strings.
Fix: the sitemap was split into four tiered files by content value, and every noindex URL was removed. Submitted URLs went from 370 to 245, with thin result pages segregated into their own low-priority file.
Googlebot fetched /api/event four times. That endpoint accepts a POST and returns a JSON acknowledgement; there is nothing there to index. Each such request is crawl budget spent on nothing.
Fix: Disallow: /api/ was added to robots.txt.
The homepage declared hreflang alternates to /de/, /uk/ and /ru/. But those pages are not German, British-English and Russian versions of the homepage — they are three separate long-form articles on unrelated regulatory topics (German product liability, UK IEN deadlines, Russian marketplace listing rules).
hreflang means "same content, different language." Pointing it at different content is a misuse that search engines are entitled to ignore, and in aggregate it dilutes the signal. Fix: the false equivalences were removed and the declaration collapsed to a single self-referential x-default.
We checked llms.txt availability on five well-funded companies in the AI-tooling and digital-product space. The result is not what the GEO vendor narrative predicts:
| Company | llms.txt | Size | What it contains |
|---|---|---|---|
| n8n | Yes | 1.1 KB | 4 research reports + Markdown twins |
| Sintra | No | — | — |
| Lindy | No | — | — |
| Zapier | No | — | — |
| Gumroad | No | — | — |
| This site (before) | Yes | 11.0 KB | Product catalogue + prices + buy links |
Only one of five publishes the file at all, and it is the smallest one in the set — four links, not forty. Its purpose is to point AI tools at research reports and at Markdown versions of those reports, so ingestion costs the model as little as possible.
The site being audited had a llms.txt more than 9× larger, and it was the wrong type of document: a machine-readable catalogue of things for sale. A catalogue answers "what can I buy." A citation engine is answering "what is true." Those are different questions, and only the second one earns a reference.
/api/, and a non-standard LLMs: directive that crawlers ignore was removed.x-default.text/markdown behind a path-scoped allowlist so source files stay blocked.Everything above is checkable from outside. The tiered sitemaps are public at /sitemap-core.xml, /sitemap-reports.xml, /sitemap-geo.xml and /sitemap-audit.xml. The rebuilt index is at /llms.txt, and any task landing page returns a Markdown twin by appending .md to its URL.
What we explicitly do not claim: that these fixes produced traffic. Crawl acceptance is same-day verifiable; ranking and citations are not. This audit reports the state of access and the state of content substance. Referral outcomes belong in a separate measurement taken after a full re-crawl cycle.
Get the GEO audit kit
27 tested kits · Python scripts · MIT licensed · runs 100%% locally · buy once, instant download · sold via Gumroad (merchant of record)
Get it now · $27.55 Compare kits →