llms.txt Format

llms.txt is a documentation format standard optimized for AI and Large Language Models (LLMs). This chapter explores how to create and use llms.txt files to improve your documentation's AI discoverability and readability.

What is llms.txt?

Overview

llms.txt is a structured text format designed to make documentation more understandable for AI models. It provides a standardized way to organize and present documentation content, enabling LLMs to index and retrieve information more effectively.

# Project Name

A brief description of the project.

## Quick Start

Basic instructions for installation and usage.

## API Reference

Details about core APIs and functionality.

Why Use llms.txt?

• AI Optimization: Makes content more digestible for AI models
• Improved Search: Enhances AI-powered search capabilities
• Standardization: Provides consistent documentation structure
• Discoverability: Helps LLMs find relevant information
• Context: Better context for AI assistance

Basic Structure

File Format

llms.txt files use a simple Markdown-style format:

# Main Title

## Section 1
Content describing the first section.

### Subsection 1.1
Detailed information and examples.

## Section 2
Content for another major section.

## Tags
keywords, tags, metadata

Required Sections

1. Project Title: Clear project name
2. Description: Brief project overview
3. Quick Start: Basic usage instructions
4. Key Features: List of main features
5. Documentation Links: Links to full documentation

# Rspress

Rspress is a fast, React-based static site generator.

## Description
Rspress provides powerful features for documentation sites with MDX support,
theme customization, and excellent performance.

## Quick Start
npm install rspress
npx rspress init
npx rspress dev

## Key Features
- MDX support
- Fast builds
- Theme customization
- SSG/SSR support

## Documentation
Full docs: https://rspress.dev

Creating llms.txt

Manual Creation

Create llms.txt in your project root:

# My Documentation Project

A comprehensive guide for technical documentation.

## Overview
This documentation covers our APIs, guides, and best practices.

## Installation
npm install my-package

## Configuration
// my-config.js
export default {
  theme: 'default',
  title: 'My Docs'
};

## API Endpoints

### GET /api/users
Retrieve all users
Returns: Array of user objects

### POST /api/users
Create a new user
Parameters: name, email
Returns: Created user object

## Examples

### Basic Usage
import { createUser } from 'my-package';

const user = await createUser({
  name: 'John Doe',
  email: 'john@example.com'
});

## Best Practices
- Always validate input
- Handle errors properly
- Use TypeScript for type safety

## Troubleshooting

### Common Issues
Q: Why is my import failing?
A: Make sure the package is installed correctly.

### Error Messages
"Module not found" - Check your package.json dependencies

## Resources
- GitHub: https://github.com/user/project
- Docs: https://docs.example.com
- Support: support@example.com

## Tags
javascript, typescript, api, documentation, web development

Automated Generation

Generate from existing documentation using a script:

// scripts/generate-llms-txt.js
import fs from 'fs';
import path from 'path';
import matter from 'gray-matter';

function generateLlmsTxt(docsDir, outputPath) {
  let content = '# Documentation Index\n\n';

  // Read all markdown files
  const files = fs.readdirSync(docsDir)
    .filter(f => f.endsWith('.md'));

  files.forEach(file => {
    const filePath = path.join(docsDir, file);
    const fileContent = fs.readFileSync(filePath, 'utf-8');
    const { data, content: markdown } = matter(fileContent);

    // Extract title and description
    const title = data.title || file.replace('.md', '');
    const description = markdown.split('\n')[0];

    content += `## ${title}\n`;
    content += `${description}\n\n`;

    // Extract code examples
    const codeBlocks = markdown.match(/```[\s\S]*?```/g);
    if (codeBlocks) {
      content += '### Examples\n';
      content += codeBlocks[0] + '\n\n';
    }
  });

  // Write llms.txt
  fs.writeFileSync(outputPath, content);
  console.log(`Generated ${outputPath}`);
}

generateLlmsTxt('./docs', './llms.txt');

Run the generator:

node scripts/generate-llms-txt.js

Integration with Rspress

Configure in rspress.config.ts:

import { defineConfig } from 'rspress/config';

export default defineConfig({
  // ... other config
  builderConfig: {
    output: {
      copy: [
        {
          from: './llms.txt',
          to: './llms.txt'
        }
      ]
    }
  }
});

Best Practices

Content Organization

1. Hierarchical Structure: Use clear heading hierarchy

# Main Title (H1)
## Major Section (H2)
### Subsection (H3)
#### Detailed Topic (H4)

2. Conciseness: Keep descriptions clear and concise

# Good Example
## Authentication
Use JWT tokens for API authentication.

# Bad Example
## This is How to Do Authentication in Our System
Our authentication system uses JSON Web Tokens,
commonly known as JWT, which is a popular standard...

3. Code Examples: Include practical, runnable examples

## Fetching Users

// Good - Complete example
import { getUser } from './api';

const user = await getUser('123');
console.log(user.name);

// Bad - Incomplete
getUser('123')

Keywords and Tags

Include relevant keywords at the end of the file:

## Tags
react, static-site-generator, documentation, mdx, typescript,
rspress, web-development, markdown, javascript, ssg

Metadata

Add useful metadata:

## Metadata
Version: 1.0.0
Last Updated: 2024-01-26
Author: Documentation Team
License: MIT
Repository: https://github.com/user/project

Advanced Features

Multi-language Support

Create separate files for different languages:

llms.txt        # English version
llms.zh.txt     # Chinese version
llms.ja.txt     # Japanese version
llms.es.txt     # Spanish version

Specify language in each file:

# Project Name
Language: English (en-US)

Project description...

Versioning

Maintain llms.txt for different versions:

llms.txt        # Latest version
llms.v1.txt     # Version 1.x
llms.v2.txt     # Version 2.x

Include version in content:

# Rspress v1.0

Version 1.0 features and documentation

## New in v1.0
- MDX support
- Improved routing

Linking Documentation

Link to detailed documentation pages:

## API Reference

### Configuration
Brief description...
Full details: https://docs.example.com/config

### Plugins
Overview of plugin system...
Full details: https://docs.example.com/plugins

Indexing Strategies

Content Priority

Organize content by importance:

# Project Name

## Quick Start (Priority: High)
Critical information to get started immediately...

## Core Concepts (Priority: High)
Fundamental concepts and terminology...

## API Reference (Priority: Medium)
Detailed API documentation...

## Advanced Topics (Priority: Low)
Complex use cases...

Search Optimization

Optimize content for search:

## Installation and Setup

Keywords: install, setup, initialize, getting-started

### npm Installation
npm install rspress

Aliases: package-manager, npm, installation

### yarn Installation
yarn add rspress

Aliases: package-manager, yarn, installation

Code Documentation

Document code examples thoroughly:

## Creating Configuration

Purpose: Set up project configuration
Language: TypeScript
File: rspress.config.ts

// rspress.config.ts
import { defineConfig } from 'rspress/config';

export default defineConfig({
  title: 'My Site',
  description: 'Site description',
  themeConfig: {
    navbar: [/* ... */],
    sidebar: {/* ... */}
  }
});

Explanation:
- title: Site title (required)
- description: SEO meta description
- themeConfig: Theme customization options

Automation

GitHub Actions

Automate generation with GitHub Actions:

# .github/workflows/generate-llms-txt.yml
name: Generate llms.txt

on:
  push:
    branches: [main]
    paths:
      - 'docs/**'

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'

      - name: Install dependencies
        run: npm install

      - name: Generate llms.txt
        run: node scripts/generate-llms-txt.js

      - name: Commit changes
        run: |
          git config user.name "GitHub Actions"
          git config user.email "actions@github.com"
          git add llms.txt
          git diff --quiet && git diff --staged --quiet || git commit -m "Update llms.txt"
          git push

Pre-build Hooks

Generate before building:

// scripts/prebuild.js
import { generateLlmsTxt } from './generate-llms-txt.js';

async function prebuild() {
  console.log('Generating llms.txt...');
  await generateLlmsTxt('./docs', './public/llms.txt');
  console.log('Done!');
}

prebuild();

In package.json:

{
  "scripts": {
    "prebuild": "node scripts/prebuild.js",
    "build": "rspress build"
  }
}

Validation and Testing

Validation Script

Validate llms.txt format:

// scripts/validate-llms-txt.js
import fs from 'fs';

function validateLlmsTxt(filePath) {
  const content = fs.readFileSync(filePath, 'utf-8');
  const errors = [];

  // Check for main title
  if (!content.match(/^# /m)) {
    errors.push('Missing main title (# )');
  }

  // Check for required sections
  const requiredSections = ['Quick Start', 'Description', 'Documentation'];
  requiredSections.forEach(section => {
    if (!content.includes(`## ${section}`)) {
      errors.push(`Missing required section: ${section}`);
    }
  });

  // Check code block formatting
  const codeBlocks = content.match(/```/g);
  if (codeBlocks && codeBlocks.length % 2 !== 0) {
    errors.push('Unclosed code block');
  }

  if (errors.length > 0) {
    console.error('Validation errors:');
    errors.forEach(err => console.error(`- ${err}`));
    process.exit(1);
  }

  console.log('Validation passed!');
}

validateLlmsTxt('./llms.txt');

Quality Checks

Check content quality:

function checkQuality(content) {
  const issues = [];

  // Check section lengths
  const sections = content.split(/^## /m);
  sections.forEach((section, i) => {
    if (i > 0 && section.length < 50) {
      issues.push(`Section ${i} is too short (${section.length} chars)`);
    }
  });

  // Check for code examples
  const codeBlocks = content.match(/```[\s\S]*?```/g) || [];
  if (codeBlocks.length < 3) {
    issues.push('Too few code examples');
  }

  return issues;
}

Troubleshooting

Common Issues

1. Formatting Errors

Issue: Headers not rendering correctly
Solution: Ensure space after #
Wrong: #Title
Correct: # Title

2. Missing Sections

Issue: AI can't find certain information
Solution: Add all required sections:
- Project description
- Quick start
- Key features
- Documentation links

3. Code Block Issues

Issue: Code not formatted correctly
Solution: Use proper code fences
```javascript
// Code here

### Debug Tips

Enable verbose logging:

```javascript
function generateWithLogging(docsDir) {
  console.log('Starting generation...');
  console.log(`Docs directory: ${docsDir}`);

  const files = fs.readdirSync(docsDir);
  console.log(`Found ${files.length} files`);

  files.forEach(file => {
    console.log(`Processing: ${file}`);
    // Process file...
  });

  console.log('Generation complete!');
}

Real-world Example

Complete llms.txt Template

# Rspress - Modern Documentation Framework

Rspress is a powerful, React-based static site generator
specifically designed for documentation websites.

## Description

Rspress combines the speed of Vite with the power of React to
provide an excellent documentation experience. Features include MDX support,
theme customization, internationalization, and more.

## Quick Start

### Installation
npm install -g rspress

### Create Project
rspress init my-docs
cd my-docs

### Development
rspress dev

### Build
rspress build

## Core Features

- **MDX Support**: Write React components in Markdown
- **Fast Builds**: Lightning-fast builds with Vite
- **Theme Customization**: Fully customizable theme system
- **SSG/SSR**: Static site generation and server-side rendering
- **Internationalization**: Built-in multi-language support
- **Auto-routing**: File-based automatic routing
- **Search**: Built-in full-text search
- **Responsive**: Mobile-first design

## Configuration

### Basic Configuration
// rspress.config.ts
import { defineConfig } from 'rspress/config';

export default defineConfig({
  title: 'My Docs',
  description: 'Site description',
  themeConfig: {
    navbar: [],
    sidebar: {}
  }
});

### Theme Configuration
themeConfig: {
  navbar: [
    { text: 'Home', link: '/' },
    { text: 'Guide', link: '/guide/' }
  ],
  sidebar: {
    '/guide/': [
      {
        text: 'Getting Started',
        items: [
          { text: 'Introduction', link: '/guide/introduction' },
          { text: 'Quick Start', link: '/guide/quick-start' }
        ]
      }
    ]
  }
}

## API Reference

### Configuration Options
- title: Site title
- description: Site description
- themeConfig: Theme configuration
- locales: Language configuration
- markdown: Markdown configuration

### Theme Configuration
- navbar: Navigation bar configuration
- sidebar: Sidebar configuration
- footer: Footer configuration
- socialLinks: Social media links

## Usage Examples

### MDX Components
import { Button } from './components/Button';

# My Page

<Button onClick={() => alert('Hello')}>
  Click Me
</Button>

### Custom Layout
export default function Layout({ children }) {
  return (
    <div className="custom-layout">
      {children}
    </div>
  );
}

## Best Practices

1. **Organize Content**: Use clear folder structure
2. **Use MDX**: Leverage interactive components
3. **Optimize Builds**: Enable code splitting
4. **SEO**: Add metadata and descriptions
5. **Performance**: Optimize images and assets

## Troubleshooting

### Build Errors
Q: Why is my build failing?
A: Check config file syntax and import paths

### Routing Issues
Q: Routes not working as expected?
A: Verify file structure and naming conventions

### Styling Issues
Q: Custom styles not applying?
A: Ensure CSS is imported correctly and using correct selectors

## Deployment

### Vercel
vercel deploy

### Netlify
netlify deploy --prod

### GitHub Pages
npm run build
npm run deploy:github

## Resources

- Documentation: https://rspress.dev
- GitHub: https://github.com/web-infra-dev/rspress
- Discord: https://discord.gg/rspress
- Twitter: @rspress_dev

## Version

Current: 1.0.0
Node.js: >= 16.0.0
npm: >= 8.0.0

## License

MIT License - see LICENSE file

## Tags

rspress, documentation, static-site-generator, react, vite, mdx,
typescript, ssg, ssr, markdown, docs, website, developer-tools

---

Next Steps

• Learn about Code Blocks and Containers
• Explore Custom Pages
• Read about Custom Theme

---

Best Practices

• Keep llms.txt in sync with main documentation
• Include practical, runnable code examples
• Use clear hierarchical structure
• Add snippets for all major features

Considerations

• Validate format regularly
• Avoid overly long sections
• Don't include sensitive information
• Keep content concise and relevant

Resources

• llms.txt Specification
• AI Documentation Best Practices
• Rspress Documentation Guide