← Tech
▚Tech

Making a Blog Machine-Legible: llms.txt, robots.txt, and JSON-LD

A working spec for the three files that make a site easier for AI to parse and cite — an llms.txt content index, an AI-permissive robots.txt, and Schema.org JSON-LD — plus the caveats that matter in 2026.

S Synortex ·

TL;DR — Three low-cost files reduce friction for AI readers: /llms.txt, a curated content index; a robots.txt that explicitly admits AI crawlers and references the sitemap; and JSON-LD structured data (BlogPosting, Organization, BreadcrumbList) on every page. None is a guaranteed win — llms.txt is not a ranking signal and FAQ rich results are being retired — but together they remove parsing overhead. This blog ships all three.

Search engines are no longer the only systems reading your pages. ChatGPT, Claude, Perplexity, and Google's AI answers fetch, parse, and sometimes quote web content. Content buried in cluttered markup with no machine-readable signals raises their cost to extract — and lowers your citation rate. The stack below is the practical minimum, stated with the caveats intact.

robots.txt: admit the crawlers deliberately

AI bots fall into three classes, and the distinction is operationally important:

Class Function Examples
Training Collects content to train models GPTBot, ClaudeBot, CCBot, Bytespider
Search index Builds the index behind AI answers (returns citations) OAI-SearchBot, PerplexityBot, Claude-SearchBot
Live fetch Reads a page on immediate user request ChatGPT-User, Perplexity-User

If you want AI visibility — most blogs do, since the search-index bots drive citations and referral traffic — name them explicitly:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://terapep.com/sitemap.xml

Two operational notes. Google-Extended and Applebot-Extended are not crawlers — they are control tokens that govern whether already-crawled content can feed AI training, with no effect on Search ranking. And robots.txt is advisory: reputable bots comply, but a hard block for a bot that ignores rules (Bytespider has that reputation) belongs at the edge or WAF. To opt out of training while keeping AI-search citations, Disallow: / the training bots and Allow: / the search-index ones.

llms.txt: a curated index for language models

llms.txt is a convention from Jeremy Howard of Answer.AI — a single Markdown file at the site root (/llms.txt) that hands an LLM a clean index of your key content, so it need not crawl and de-clutter the full site to model it.

The format is strict and ordered:

  1. An # H1 with the site name (the only required line)
  2. A > blockquote summary
  3. Optional prose
  4. ## H2 sections, each a link list: - [name](url): note

Minimal example:

# Terapep

> A blog about technology and good food — practical notes, honest takes, and recipes worth keeping.

## Posts

- [How Korean Ramyeon Took Over the World](https://terapep.com/blog/korean-ramyeon-global-boom/): The export data behind K-ramen's rise.
- [Does GEO Actually Work?](https://terapep.com/blog/does-geo-actually-work/): What the research really says about AI-search optimization.

A companion llms-full.txt inlines all content into one Markdown file for single-fetch ingestion — watch the size, since it can exceed a context window on a large site.

The caveat is load-bearing: llms.txt is "mainly useful for inference" — when an LLM is actively assisting a user — not training, and no major AI vendor has committed to it as a ranking input. Google has explicitly said you don't need it for its AI features. Treat it as a low-cost convenience some tools read, not a guaranteed result.

JSON-LD: structured data that still produces results

Structured data declares what a page is. Embed it as <script type="application/ld+json">. For a blog in 2026, the high-value types are BlogPosting, Organization/WebSite, and BreadcrumbList.

A trimmed BlogPosting:

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Make Your Blog AI-Readable",
  "description": "A practical guide to llms.txt, robots.txt, and JSON-LD.",
  "image": "https://terapep.com/og.jpg",
  "datePublished": "2026-06-15T09:00:00+09:00",
  "dateModified": "2026-06-15T09:00:00+09:00",
  "author": { "@type": "Organization", "name": "Terapep" },
  "publisher": {
    "@type": "Organization",
    "name": "Terapep",
    "logo": { "@type": "ImageObject", "url": "https://terapep.com/favicon.svg" }
  },
  "mainEntityOfPage": "https://terapep.com/blog/make-your-blog-ai-readable/"
}

Two 2026 corrections. Always include a timezone offset (+09:00) — without it Google assumes Googlebot's timezone. And do not chase FAQPage rich results: Google is sunsetting FAQ rich results for most sites. The markup stays valid and may help AI, but the snippet is gone. (A real FAQ section with question-shaped headings still helps — see the GEO post.) Validate with Google's Rich Results Test before shipping.

Implementation in Astro

Astro suits this because it serves static HTML by default — content sits in the markup with no hydration, which is exactly what AI fetchers want.

  • robots.txt: a static file in public/, or a src/pages/robots.txt.ts endpoint to inject the site URL.
  • llms.txt: a src/pages/llms.txt.ts endpoint that iterates posts and returns text/plain.
  • JSON-LD: a component fed by frontmatter, rendered once per page — site-wide Organization/WebSite in the base head, per-post BlogPosting + BreadcrumbList in the post layout.

FAQ

What is llms.txt and do I need it?

llms.txt is a Markdown index of a site's key content, served at /llms.txt, that helps language models understand the site at inference time. It is a low-cost convenience some AI tools read, but it is not a ranking signal, and Google says you don't need it.

How do I let AI crawlers read my site?

Add explicit User-agent / Allow: / blocks in robots.txt for the bots you want (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, etc.) and include a Sitemap: line. robots.txt is advisory — enforce hard blocks at the edge.

Does FAQ schema still work in 2026?

FAQ rich results are being retired by Google for most sites, so the snippet is gone. The JSON-LD remains valid markup, and a real FAQ section (question-shaped headings plus self-contained answers) still aids AI comprehension.

Sources: llmstxt.org, Google AI optimization guide, Google: web publisher controls, Schema.org Article docs, Search Engine Land.

Image: Markus Spiske, CC0 / public domain, via Wikimedia Commons.

#llms-txt#seo#schema#ai#astro
← Back to all posts