How do you start a blog on your app and actually get found?
Files beat a database, and the biggest mistake is blocking the crawlers that actually send you traffic. The whole setup, including what we built for this site.
How do you start a blog on your app and actually get found?
Store posts as files rather than in a database, publish the machine-readable surfaces that search engines and AI assistants look for, and make sure you are not accidentally blocking the crawlers that send you traffic. The last one is the most common own goal, and the one nobody notices, because nothing about it produces an error.
This post is the blog you are reading. Here is how it is built.
Files, not a database
The instinct is a posts table with an admin interface. Resist it.
A database-backed blog needs authentication, permissions, an editor, image uploads, a draft workflow, and a preview mode. That is a real product, and it is not the one you are building.
Files skip all of it. A post is a markdown file with a few lines of metadata at the top. The filename is the address. You already have a text editor, and your writing gets reviewed and versioned exactly like your code.
The tradeoff is honest: publishing requires a deploy, and non-technical colleagues cannot post without one. For a product blog written by the people who build the product, that is a fine trade, and it is the right default until it stops being.
What every post needs generated for it
This is the part worth automating once, because doing it by hand per post is how it gets skipped:
- A canonical address. Build it from the path, never from the full URL including query strings. Otherwise a link with a campaign parameter becomes a second copy of the page and splits its ranking.
- Social card tags, so a shared link unfurls properly.
- Structured data, marking the page as an article with an author, a publication date, and a modification date.
- A sitemap entry with a last-modified date.
- A feed. Atom, and a JSON feed alongside it.
None of this is per-post work if you generate it from the post's own metadata.
The three surfaces most sites are missing
robots.txt, served by the app
Usually a static file with a stale sitemap line. Serve it from the application instead and it can know its own address, stay correct across environments, and survive a rebrand.
llms.txt
A convention that is quickly becoming worth having: a markdown index of your site at /llms.txt,
listing your pages and posts with a one-line description each, so an assistant can understand what
you have in one fetch rather than by crawling. Pair it with a full-text version that inlines
everything.
Markdown alternates
Serve the raw source of each post at the same address with a .md extension, and link it from the
page. A crawler or a tool that wants your text does not have to parse your HTML to get it.
The mistake that quietly costs the most
AI crawlers are not one thing. They come in three kinds, and they behave completely differently:
| Kind | What it does | Sends you traffic? |
|---|---|---|
| Training | Builds model weights | No |
| Search index | Builds the assistant's retrieval index | Yes, as citations |
| Live fetch | Fetches on demand when someone asks | Yes, directly |
Most sites that decide to "block AI" write rules aimed at the training crawlers and take out the fetchers by accident. The result is that they disappear from AI answers entirely, while every other metric looks completely normal.
If you want traffic from assistants, the search-index and live-fetch categories are the ones to welcome, explicitly, by name.
Two things people get wrong here. Google-Extended and Applebot-Extended are not crawlers;
they are training opt-out tokens, and disallowing them does not remove you from AI Overviews. And a
crawler cannot be blocked by robots.txt alone if something in front of your app is already refusing
it.
Three layers that can make you invisible, silently
This is the part worth checking today, whatever else you take from this post. Any of these can hide your site with no error, no log line, and no symptom:
- Your CDN. Providers now ship a "block AI crawlers" control that rejects requests at the edge. A perfect robots.txt changes nothing if that is on.
- Browser version checks. Frameworks that turn away outdated browsers answer with a bare refusal. Exclude your public content from that check; an article is prose, not an application.
- Rate limiting. A blanket per-address limit will catch a crawler sweeping your site, and a refusal is read as "this site is unhealthy, come back less often".
One command tests all three at once:
curl -sI -A "OAI-SearchBot" https://yourdomain.com/blog | head -1
If that is not a 200, nothing else on this page matters.
Write so you can be quoted
The structural work above gets you readable. This gets you cited.
An assistant lifts a passage from your page and shows it with no surrounding context. So:
- Make the title the question someone types. Not a clever phrase.
- Restate the question in the first sentence.
- Answer it within eighty words, with something concrete: a number, a list, a yes with its condition. Assume this paragraph will be shown alone, because it will be.
- Then go deep, under headings that are themselves nearly questions. Each section must make sense read on its own.
- Prefer specifics to adjectives.
- State your limits honestly. Sources that qualify their claims get cited more, not less.
The last piece of advice is the one that matters most: write about the thing where you are genuinely the best available source. Anyone can restate a popular tutorial, and the results already have forty of those. Nobody else can explain what your product actually does, what it costs, or what you learned building it.
That is the post worth writing first.
