How to Make Your Documentation AI-Friendly: llms.txt, Content Negotiation, and Markdown Serving

By Facundo F · Updated on 27th February 2026

AI coding assistants are becoming the primary way developers interact with documentation. When Claude Code, Cursor, or GitHub Copilot fetches your docs, it gets bloated HTML full of navigation, footers, and styling markup — burning tokens on noise instead of content.

The fix is straightforward: serve Markdown when AI agents request it. Companies like Cloudflare, Stripe, Anthropic, and Mintlify already do this. The approaches range from a simple /llms.txt file to full HTTP content negotiation at every URL.

This guide covers all three approaches — llms.txt, content negotiation, and edge conversion — with implementation examples for Laravel, Express, Django, and static sites.

Why HTML Is a Problem for AI Agents

When an AI agent fetches a documentation page, most servers return the full HTML rendering — navigation bars, sidebars, footers, analytics scripts, and CSS class names. All of this consumes tokens without adding information.

Here's a real example. This HTML snippet from a typical docs page:

<div class="documentation-wrapper">
  <header class="docs-header">
    <nav class="breadcrumb">...</nav>
  </header>
  <aside class="sidebar">...</aside>
  <main class="content">
    <h1 class="heading-primary">Deployment Commands</h1>
    <p class="text-body">To deploy your application:</p>
    <pre class="code-block"><code>deployhq deploy production</code></pre>
  </main>
  <footer class="docs-footer">...</footer>
</div>

The same content in Markdown:

## Deployment Commands

To deploy your application:

    deployhq deploy production

The HTML version uses roughly 4x more tokens for identical content. A simple ## About Us heading costs about 3 tokens in Markdown; its HTML equivalent — <h2 class="section-title" id="about">About Us</h2> — burns 12-15 tokens before accounting for wrapper divs.

Scale this across an entire documentation site, and you're looking at significant context window waste. An AI agent that could reference 20 documentation pages in Markdown can only fit 5 of the same pages in HTML.

flowchart LR
    A[AI Agent] -->|Accept: text/markdown| B[Your Server]
    A -->|Accept: text/html| B
    B -->|Markdown| C[3 tokens per heading]
    B -->|HTML| D[15 tokens per heading]
    C --> E[20 pages in context]
    D --> F[5 pages in context]

Three Approaches to AI-Friendly Documentation

There are three main strategies for making your docs accessible to AI agents. You can combine them.

1. llms.txt — The Directory Approach

The llms.txt specification is a proposed standard that works like a robots.txt for AI models. Instead of telling crawlers what not to access, it tells AI agents where your most authoritative content lives.

Place a Markdown file at /llms.txt on your domain:

# DeployHQ Documentation

> DeployHQ automates code deployment from Git repositories to any server.

## Getting Started

- [Quick Start Guide](https://www.deployhq.com/support/getting-started): Deploy your first project in 5 minutes
- [Server Setup](https://www.deployhq.com/support/servers): Connect SSH, FTP, or cloud servers
- [Repository Connection](https://www.deployhq.com/support/repositories): Link GitHub, GitLab, or Bitbucket

## Deployment

- [Build Pipelines](https://www.deployhq.com/blog/build-pipelines-in-deployhq-streamline-your-deployment-workflow): Configure build commands and deployment steps
- [Atomic Deployments](https://www.deployhq.com/blog/what-is-atomic-deployment): Zero-downtime releases with symlink switching
- [API Reference](https://www.deployhq.com/blog/using-deployhq-s-api-automating-your-deployment-workflows-with-scripts-and-webhooks): Automate deployments via REST API

## Optional

- [Changelog](https://www.deployhq.com/changelog): Recent platform updates
- [Blog](https://www.deployhq.com/blog/archive): Technical articles and tutorials

The format is simple:

An H1 with your project name (required)
A blockquote with a brief summary (optional)
H2 sections with categorised link lists
An Optional section for content that can be skipped if context is limited

Major companies already publish llms.txt files: Anthropic, Cloudflare, Stripe, and Mintlify among others. As of 2025, adoption has grown rapidly — hundreds of thousands of websites now implement it.

Limitations: llms.txt is a curated directory, not a content delivery mechanism. The linked pages still serve HTML unless you also implement content negotiation. Think of it as the table of contents; you still need the content itself to be AI-friendly.

2. Content Negotiation — The HTTP Standard Approach

Content negotiation is a 27-year-old HTTP standard where clients specify their preferred format via the Accept header. The same URL serves different formats to different consumers — HTML for browsers, JSON for mobile apps, RSS for feed readers, and now Markdown for AI agents.

When an AI agent sends Accept: text/markdown, your server returns clean Markdown. When a browser sends Accept: text/html, it gets the rendered page. Same URL, same content, different format.

This is the highest-ROI approach for most sites because it works with every AI tool that supports HTTP fetching, requires no special client-side integration, and follows web standards that search engines already understand.

Important: Content negotiation at the same URL is not cloaking. Cloaking would be serving different content to different user agents — like showing search engines one page and users another. Serving the same content in a different format (Markdown vs HTML) at the same URL using standard HTTP headers is perfectly fine.

3. Edge Conversion — The Zero-Code Approach

If you use Cloudflare, you can enable Markdown for Agents on your zone. When AI agents request pages with Accept: text/markdown, Cloudflare's edge network automatically converts the HTML response to Markdown on the fly — no server-side changes required.

This is the fastest path to AI-friendly docs if you're already on Cloudflare. For other CDN providers, check if they offer similar edge transformation features.

Implementation: Content Negotiation

Here's how to implement content negotiation in four popular frameworks. The pattern is the same everywhere: check the Accept header, return Markdown if requested, HTML otherwise.

Laravel

namespace App\Http\Controllers;

use Illuminate\Http\Request;
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\File;
use Illuminate\Support\Str;

class DocsController extends Controller
{
    public function show(Request $request, $path = 'index')
    {
        $markdown = $this->loadMarkdown($path);

        if (!$markdown) {
            abort(404, 'Documentation not found');
        }

        $accept = $request->header('Accept', '');

        if (str_contains($accept, 'text/markdown')) {
            return response($markdown)
                ->header('Content-Type', 'text/markdown; charset=utf-8')
                ->header('Vary', 'Accept');
        }

        if (str_contains($accept, 'text/plain')) {
            return response($markdown)
                ->header('Content-Type', 'text/plain; charset=utf-8')
                ->header('Vary', 'Accept');
        }

        return response()
            ->view('docs.show', ['content' => Str::markdown($markdown)])
            ->header('Vary', 'Accept');
    }

    private function loadMarkdown(string $path): ?string
    {
        // Prevent directory traversal
        $clean = str_replace(['../', './'], '', $path);
        $file = resource_path("docs/{$clean}.md");

        return File::exists($file) ? File::get($file) : null;
    }
}

Route registration:

Route::get('/docs/{path?}', [DocsController::class, 'show'])
    ->where('path', '.*')
    ->name('docs.show');

Express / Node.js

import { readFile } from 'fs/promises';
import { join } from 'path';
import { marked } from 'marked';

const DOCS_DIR = './docs';

app.get('/docs/:path(*)', async (req, res) => {
  const safePath = req.params.path.replace(/\.\./g, '');
  const filePath = join(DOCS_DIR, `${safePath || 'index'}.md`);

  let markdown;
  try {
    markdown = await readFile(filePath, 'utf-8');
  } catch {
    return res.status(404).send('Documentation not found');
  }

  res.set('Vary', 'Accept');

  if (req.accepts('text/markdown')) {
    return res.type('text/markdown; charset=utf-8').send(markdown);
  }

  if (req.accepts('text/plain')) {
    return res.type('text/plain; charset=utf-8').send(markdown);
  }

  const html = await marked(markdown);
  res.type('html').send(renderLayout(html));
});

Django

from pathlib import Path
from django.http import HttpResponse
from django.shortcuts import render
import markdown

DOCS_DIR = Path(__file__).resolve().parent / 'docs'

def docs_view(request, path='index'):
    safe_path = path.replace('..', '')
    file_path = DOCS_DIR / f'{safe_path}.md'

    if not file_path.exists() or not file_path.is_relative_to(DOCS_DIR):
        return HttpResponse('Not found', status=404)

    md_content = file_path.read_text(encoding='utf-8')
    accept = request.META.get('HTTP_ACCEPT', '')

    if 'text/markdown' in accept:
        response = HttpResponse(md_content, content_type='text/markdown; charset=utf-8')
        response['Vary'] = 'Accept'
        return response

    if 'text/plain' in accept:
        response = HttpResponse(md_content, content_type='text/plain; charset=utf-8')
        response['Vary'] = 'Accept'
        return response

    html_content = markdown.markdown(md_content)
    response = render(request, 'docs/show.html', {'content': html_content})
    response['Vary'] = 'Accept'
    return response

Static Sites (Hugo, Jekyll, Docusaurus)

Static site generators don't support server-side content negotiation directly. You have three options:

Option A: Publish Markdown alongside HTML. Configure your build to output both rendered HTML and raw .md files at known paths (e.g., /docs/getting-started.md). AI tools can fetch the .md URL directly.

Option B: Use an edge function. Deploy a Cloudflare Worker, Vercel Edge Function, or Netlify Edge Function that intercepts requests with Accept: text/markdown and returns the source Markdown from your repository.

Option C: Publish llms.txt. Create an llms.txt file that links directly to your raw Markdown files on GitHub or your CDN.

Example Cloudflare Worker:

export default {
  async fetch(request, env) {
    const accept = request.headers.get('Accept') || '';
    const url = new URL(request.url);

    if (accept.includes('text/markdown') && url.pathname.startsWith('/docs/')) {
      const mdPath = url.pathname.replace(/\/$/, '') + '.md';
      const mdContent = await env.DOCS_BUCKET.get(mdPath);

      if (mdContent) {
        return new Response(mdContent.body, {
          headers: {
            'Content-Type': 'text/markdown; charset=utf-8',
            'Vary': 'Accept',
          },
        });
      }
    }

    return fetch(request);
  },
};

The Vary Header: Don't Forget It

Every response that varies by Accept header must include Vary: Accept. This tells CDNs and proxies to cache different versions for different content types. Without it, a CDN might cache the Markdown version and serve it to browsers, or vice versa.

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
Vary: Accept

Testing Your Implementation

Use curl to verify all three response types:

# Request Markdown (what AI agents send)
curl -H 'Accept: text/markdown' https://yourdomain.com/docs/getting-started

# Request plain text
curl -H 'Accept: text/plain' https://yourdomain.com/docs/getting-started

# Request HTML (default browser behavior)
curl -H 'Accept: text/html' https://yourdomain.com/docs/getting-started

For the Markdown response, verify:

Content-Type: text/markdown; charset=utf-8
Vary: Accept header present
Body contains raw Markdown without HTML tags

Structuring Documentation for AI Comprehension

Beyond the delivery format, how you write documentation affects how well AI agents understand it. These principles improve comprehension for both humans and machines:

Keep information self-contained. Each documentation page should make sense on its own without requiring the reader to cross-reference other pages. When AI agents retrieve a single page, scattered context across multiple files leads to incomplete answers.

Use semantic headings. Headings should describe the content that follows, not just label sections. Configuring SSH Deploy Keys is better than Step 3 — an AI agent processing a single chunk needs that context.

Put the answer first. Lead with what something does, then explain how. AI agents (and human readers) scan for the key information upfront.

Include complete code examples. Partial snippets that rely on context from earlier in the page are harder for AI agents to use. Self-contained, copy-paste-ready examples work best.

Use consistent terminology. If you call it a deployment in one place and a release in another, AI agents may treat them as different concepts. Pick one term and use it throughout.

Why This Matters for DeployHQ Users

Developers increasingly use AI coding assistants to help with deployment workflows. When those assistants can access your documentation as clean Markdown:

More of your docs fit in the context window, giving the AI agent the full picture instead of fragments
Build pipelines and deployment configurations are described precisely, leading to more accurate automation
Developers get better AI-assisted answers about your product, reducing support load
Your documentation works with tools like Context7 and MCP servers that feed real-time docs into AI assistants

This approach works for any documentation — API references, deployment guides, configuration manuals, or onboarding tutorials. The investment is small (a few hours of implementation) and the benefit compounds as AI tool usage grows.