Two things decide whether an AI assistant cites your site, and neither one is a meta tag you paste
in once and forget. You have to not block the crawlers that actually send traffic, and you have to
write content that survives being lifted out of context and shown on its own.

## Don't block the bots that send you traffic

AI crawlers come in three kinds, and they behave in completely different ways:

| Kind | What it does | Sends you traffic? | Examples |
|---|---|---|---|
| Training | Builds model weights | No | GPTBot, ClaudeBot, CCBot |
| Search index | Builds the assistant's retrieval index | Yes, as citations | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| Live fetch | Fetches on demand when a user asks | Yes, directly | ChatGPT-User, Claude-User, Perplexity-User |

Most sites that "block AI" block the training crawlers and take out the search-index and live-fetch
crawlers by accident, which removes them from AI answers entirely rather than opting out of
training the way they intended. `/robots.txt`, served by `SeoController` rather than a static file
so it can build an absolute sitemap URL, welcomes all three kinds by default. Edit
`AI_USER_AGENTS` there if you want to change that policy.

`Google-Extended` and `Applebot-Extended` are not crawlers; they're training opt-out tokens.
Disallowing them doesn't remove you from AI Overviews, which reads from ordinary Googlebot traffic
instead. Treat blocking them as a training-consent decision, separate from crawl access.

**Check your CDN too.** Cloudflare and similar providers ship a "block AI crawlers" toggle that
rejects requests at the edge, before they ever reach the Rails app. A perfect `robots.txt` changes
nothing if that toggle is on, and nothing in the app's own logs will tell you it happened. Verify it
once, directly.

## Write content that can be quoted

An assistant lifts a passage out and shows it with no surrounding context. Structure every post and
every docs page assuming that's exactly what will happen to it:

1. Make the title (or the docs page's `##` heading) the question someone would actually type.
2. Restate that question in the first sentence.
3. Answer it within about 80 words, with something concrete: a number, a named list, a yes with its
   condition attached.
4. Then go deeper, under `##` headings that are near-questions themselves. Every section needs to
   make sense read completely alone, since that's the unit an assistant is likely to extract.
5. Prefer specifics over adjectives: exact numbers, named things, and worked examples beat "fast,"
   "easy," or "powerful" every time.
6. State ranges and limits honestly. Sources that qualify their own claims get cited more often, not
   less.

The content that wins is the content where you're the best available source for the answer. Anyone
can restate a popular tutorial in their own words. Only you can say what your product actually
costs, how it actually behaves, or what you learned building it.

## Next

See how the share cards that accompany a citation get generated:
[OG Images](/docs/content-seo/og-images).