Pollen · Guide · Updated 2026-09-17
The short answer: AI assistants (ChatGPT, Gemini, Claude, Perplexity) recommend products from stores they can read and parse. For a WooCommerce store that comes down to four things, in priority order: (1) let AI crawlers in - robots.txt and your CDN both; (2) deepen your structured product data - WooCommerce emits basic Product JSON-LD, but brand, GTIN, and real attributes are usually missing; (3) put your catalog on the surfaces assistants actually query - a Google Merchant Center feed covers Gemini and Google's AI answers, and an agent API (MCP) makes your catalog directly queryable; (4) measure from your server logs, because page caches hide most AI-bot traffic. llms.txt is cheap and worth serving, but there is no evidence it drives recommendations today, so it goes last, not first. The rest of this page is the how, with the honest caveats.
Two different things get called "AI traffic", and they need different fixes:
| Crawling | Retrieval | |
|---|---|---|
| What it is | Bots (GPTBot, ClaudeBot, Google-Extended) reading pages for training and indexes | An assistant fetching your page live to answer a real shopper's question (for example OAI-SearchBot, Perplexity's fetcher) |
| Why it matters | Gets you into the model's world at all | The business event: a human asked, your store was consulted |
| What gates it | robots.txt + CDN allowing the bot | Being in the assistant's index; retrievers generally cannot fetch pages they have no link to |
Two practical consequences. First, index-gated retrieval means an assistant typically won't fetch a URL it can't discover - unlinked pages are invisible, so your machine-readable surfaces need to be linked from pages crawlers already visit. Second, assistants have largely separate indexes; published citation analyses find very little domain overlap between ChatGPT's and Perplexity's citations, so check each surface separately instead of assuming one result generalizes.
Allow at minimum: GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot. On WordPress, check that no SEO or security plugin is appending disallow rules you didn't write.
The trap is above WordPress: CDN and host-level "block AI bots" switches return 403 at the edge, so your robots.txt never even gets consulted. We have seen a store with a perfectly welcoming robots.txt whose every GPTBot request died at the CDN. If you use Cloudflare, review the AI-bots setting; then verify from outside with a real crawler user-agent, not just by reading config.
WooCommerce core outputs basic Product JSON-LD: name, price, availability. That makes you parseable, not recommendable. When a shopper asks "cotton oversized tee under 1500 that runs big", the assistant can only match stores whose data carries fabric, fit, and sizing - as structured data or at least as parseable text.
What to add, in impact order:
brand, gtin/sku on every offer. Absent on most small-store feeds we audit.Industry analyses report that products with more complete schema get included in AI shopping answers substantially more often. Treat the exact percentages as directional, but the direction is consistent with everything we measure.
llms.txt is a plain-text index of your site for AI crawlers, served at /llms.txt. The honest status in 2026: no major provider has confirmed consuming it, and there is no public evidence it changes recommendations. It is also free to serve, takes minutes, and is read by some smaller crawlers. So: publish it, link it (a rel="alternate" link in your head lets index-gated crawlers discover it), and spend your real effort on steps 1, 2, and 4.
AI-bot visits are a server-side event. Two things break naive measurement on WordPress: JavaScript analytics never see bots (they don't run JS), and full-page caches serve bots from cache so PHP never runs - we found a store where ChatGPT's live fetches left zero trace because LiteSpeed served them from cache. If you measure, exclude AI user-agents from page caching (each major cache plugin supports UA exclusions) or read the web-server logs directly. Separate crawl visits from retrieval visits; retrieval is the number that means a human asked.
| Approach | What you get | What it costs you |
|---|---|---|
| Do nothing | Whatever WooCommerce emits by default: basic schema, no feeds, no measurement | Assistants answer your category's questions with other stores |
| DIY (this guide) | Full control: robots + CDN, schema plugin or code, feed plugin, log analysis | Several plugins to configure and keep coherent; attribute writing is manual per product |
| Single-purpose feed plugins | One surface each (for example an ACP or GMC feed generated from existing data) | Feeds inherit your data's gaps: a feed of empty descriptions is an empty feed |
| Managed layer (what Pollen does) | Enrichment writes the missing attributes, then feeds, schema, llms.txt, an MCP endpoint, and AI-visit analytics from one connection | A subscription; the free tier scores your catalog and shows the gaps without one |
There is no public evidence that any major assistant consumes llms.txt today, and some provider statements suggest they don't. It costs nothing to serve, it is read by some smaller crawlers, and it may matter later - so publish it, but treat structured product data and crawler access as the real levers.
For a store, blocking AI crawlers mostly protects you from being recommended. Product pages are marketing, not proprietary content: if an assistant can't read them, it answers shopper questions with stores it can read. Check your CDN too - some 'block AI bots' switches 403 crawlers at the edge even when your robots.txt allows them.
Assistants recommend from what they can parse and what their index contains. Common causes, in order: your product data is too thin to match the question (no attributes, empty descriptions), a crawler block at the robots or CDN level, and index gaps - ChatGPT and Perplexity have largely different indexes, so being visible in one says nothing about the other.
Expect index propagation on the order of a week or more, and it differs per assistant. Retrieval-style answers (where the assistant fetches your page live) can reflect changes much faster than training-data mentions. This is why weekly re-checks are signal and daily ones are noise.
No - it is more manual. Shopify now ships some machine-readable surfaces by default for eligible stores. On WooCommerce you own the whole stack, which means nothing is done for you, but nothing is locked away from you either: robots policy, structured data depth, feeds, and an agent API are all yours to control.