One source of copy for people and for AI agents
Every page on this site also exists as markdown, for agents that would rather not parse HTML. How we serve it without a second copy to maintain, and without confusing Google.
Not everything that reads a website is a person in a browser. Assistants fetch pages to answer a question, agents read documentation before calling an API, and search engines summarise before they link. Most of them want text. What they get is HTML: navigation, scripts, class names, and somewhere in the middle, the paragraph they came for.
So every page on this site has a markdown twin. The home page, the blog index and each post are also served as plain markdown, with the same words in the same order. This post is about how that works, and about the three rules that keep it from turning into a second website.
Two ways in
An agent can ask for markdown in two ways. It can add .md to the path: /blog.md, /blog/<post>.md, and /index.md for the home page. Or it can request the normal URL with an Accept: text/markdown header and get markdown back from the same address. Ask with q=0 and you get HTML, because that is an explicit “not this”.
Both paths end at one route handler, through rewrites in next.config.js. One detail matters: the rewrites have to run in the beforeFiles phase. The pages are prerendered, and on Vercel a static page matches during the filesystem phase, before ordinary rewrites ever run. Put the rewrite in the usual place and the header is quietly ignored.
Rule one: the HTML page is canonical
A second copy of every page is exactly what search engines punish when it looks like duplicate content. The obvious fix is noindex on the twins. We didn’t do that. Instead, each twin names its HTML page as canonical in a response header, which is Google’s advice for alternates that aren’t HTML:
Content-Type: text/markdown; charset=utf-8
Link: <https://hypermetron.com/blog>; rel="canonical"Search Console shows it working. /index.md is listed as “Crawled, currently not indexed”, and the home page it points to is indexed. Google read the twin, understood what it was, and folded it into the real page. That is the result we wanted.
Rule two: generated, never written
A twin written by hand drifts from its page within a week. So none of them are written. The page copy lives in small data modules: the home page headlines, the list of services, the projects, and the blog posts themselves. The React components render those modules as HTML, and a markdown controller renders the same modules as markdown. Change a headline and both versions change in the same commit.
That also decided where copy is allowed to live. A sentence typed straight into a component would appear in the HTML and be missing from the markdown. Keeping copy out of components is now a project convention, not just a preference.
Rule three: say what the page says
The twin is not a summary, and it is not a version written for machines. It has the same headings, the same list items and the same links. The only additions are a front-matter block at the top (title, description, canonical URL, language, last update) and, on posts, a contact line at the end. If an agent quotes the twin, it quotes what a visitor would have read.
Telling agents where to look
A twin nobody can find is no use, so three files point to them:
/llms.txt, following the llmstxt.org proposal: who we are in a few lines, then every page next to its markdown twin./llms-full.txt: every page’s markdown in one file, for a model that wants the whole site in a single fetch./sitemap-ai.xml: a sitemap of the twins, listed inrobots.txtnext to the normal sitemap. We submit only the normal one to Google, since the twins are not meant to rank.
Saying what the content may be used for
robots.txt can say who may fetch a page. It can’t say what they may do with it afterwards. The Content Signals proposal adds a line for that, covering search, AI answers and AI training:
User-agent: ClaudeBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /
Disallow: /api/Two details matter. Next.js generates robots.txt from a fixed set of fields, and Content-Signal isn’t one of them, so we write the file by hand. And a crawler obeys only the most specific group that names it. A signal in the * group is invisible to any bot that has its own group, so the line is repeated in every group.
The policy itself is a business decision. This is our own marketing copy, and we want models to find it, quote it and learn it, because a model that knows what we do can recommend us. The crawlers that search and cite, from OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft and Meta, are allowed. Two crawlers that collect only for training, with no route back to us, are blocked: CCBot and Meta-ExternalAgent. A site with paid content would choose differently, and the file makes that a one-line change.
Where it came from
None of this started here. We built it first for HyperCRM, our clinic software, and ported it to this site when we moved the pages from a CMS into code. Most of the work was not the markdown. It was the rules: one canonical page, one source of copy, and no separate version written only for machines.