Being Cited by AI Assistants
The three kinds of AI crawler, robots.txt and llms.txt, and how to write for extraction.
Two things decide whether an AI assistant cites your site, and neither one is a meta tag you paste in once and forget. You have to not block the crawlers that actually send traffic, and you have to write content that survives being lifted out of context and shown on its own.
Don't block the bots that send you traffic
AI crawlers come in three kinds, and they behave in completely different ways:
| Kind | What it does | Sends you traffic? | Examples |
|---|---|---|---|
| Training | Builds model weights | No | GPTBot, ClaudeBot, CCBot |
| Search index | Builds the assistant's retrieval index | Yes, as citations | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Live fetch | Fetches on demand when a user asks | Yes, directly | ChatGPT-User, Claude-User, Perplexity-User |
Most sites that "block AI" block the training crawlers and take out the search-index and live-fetch
crawlers by accident, which removes them from AI answers entirely rather than opting out of
training the way they intended. /robots.txt, served by SeoController rather than a static file
so it can build an absolute sitemap URL, welcomes all three kinds by default. Edit
AI_USER_AGENTS there if you want to change that policy.
Google-Extended and Applebot-Extended are not crawlers; they're training opt-out tokens.
Disallowing them doesn't remove you from AI Overviews, which reads from ordinary Googlebot traffic
instead. Treat blocking them as a training-consent decision, separate from crawl access.
Check your CDN too. Cloudflare and similar providers ship a "block AI crawlers" toggle that
rejects requests at the edge, before they ever reach the Rails app. A perfect robots.txt changes
nothing if that toggle is on, and nothing in the app's own logs will tell you it happened. Verify it
once, directly.
Write content that can be quoted
An assistant lifts a passage out and shows it with no surrounding context. Structure every post and every docs page assuming that's exactly what will happen to it:
- Make the title (or the docs page's
##heading) the question someone would actually type. - Restate that question in the first sentence.
- Answer it within about 80 words, with something concrete: a number, a named list, a yes with its condition attached.
- Then go deeper, under
##headings that are near-questions themselves. Every section needs to make sense read completely alone, since that's the unit an assistant is likely to extract. - Prefer specifics over adjectives: exact numbers, named things, and worked examples beat "fast," "easy," or "powerful" every time.
- State ranges and limits honestly. Sources that qualify their own claims get cited more often, not less.
The content that wins is the content where you're the best available source for the answer. Anyone can restate a popular tutorial in their own words. Only you can say what your product actually costs, how it actually behaves, or what you learned building it.
Next
See how the share cards that accompany a citation get generated: OG Images.
