Metadata-Version: 2.4
Name: wagtail-machine-readable
Version: 0.5.0
Summary: Make your Wagtail site AI-ready. llms.txt, Markdown page variants and structured data in one install.
Project-URL: Homepage, https://github.com/brett-allard-amp/wagtail-machine-readable
Project-URL: Documentation, https://github.com/brett-allard-amp/wagtail-machine-readable#readme
Project-URL: Changelog, https://github.com/brett-allard-amp/wagtail-machine-readable/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/brett-allard-amp/wagtail-machine-readable/issues
Author-email: Brett Allard <brett.allard@getamplified.co.uk>
License-Expression: MIT
License-File: LICENSE
Keywords: ai,django,generative-engine-optimisation,llms,llms.txt,seo,wagtail
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Web Environment
Classifier: Framework :: Django
Classifier: Framework :: Django :: 4.2
Classifier: Framework :: Django :: 5.2
Classifier: Framework :: Wagtail
Classifier: Framework :: Wagtail :: 6
Classifier: Framework :: Wagtail :: 7
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: wagtail>=6.3
Description-Content-Type: text/markdown

# wagtail-machine-readable

**Make your Wagtail site AI-ready: llms.txt, Markdown page variants and structured data in a single install.**

[![PyPI](https://img.shields.io/pypi/v/wagtail-machine-readable.svg)](https://pypi.org/project/wagtail-machine-readable/)
[![CI](https://github.com/brett-allard-amp/wagtail-machine-readable/actions/workflows/test.yml/badge.svg)](https://github.com/brett-allard-amp/wagtail-machine-readable/actions/workflows/test.yml)
[![Python](https://img.shields.io/pypi/pyversions/wagtail-machine-readable.svg)](https://pypi.org/project/wagtail-machine-readable/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)

LLMs, answer engines and AI crawlers are becoming the front door to your
content, and they read [llms.txt](https://llmstxt.org/), not your carefully
tuned templates. `wagtail-machine-readable` turns your existing Wagtail page
tree into spec-compliant machine-readable outputs automatically. It serves live
pages only, respects privacy restrictions, and gives every Site in a multi-site
install its own hostname. Install it, add two lines, and `/llms.txt` works.

## What you get

- **`/llms.txt`**: a spec-compliant index of your site, with an H1 site name, a
  blockquote description, H2 sections derived from your top-level pages, and
  `- [name](url): description` link lists.
- **`/llms-full.txt`**: the same page set with page content rendered to
  Markdown.
- **`.md` page variants**: every visible page served as Markdown on its own URL
  (`/about/team.md`, `/index.md` for the site root), with content negotiation
  and `rel="alternate"` links so agents can find them.
- **StreamField to Markdown extraction**: headings, links, images (alt text and
  URL), embeds and tables survive extraction instead of being stripped.
- **JSON-LD structured data**: schema.org `WebPage`/`Article`, `Organization`
  and `BreadcrumbList` per page via a template tag, with a per-page-type builder
  mapping.
- **AI crawler robots.txt**: opt-in allow and deny controls for known AI user
  agents (GPTBot, ClaudeBot, PerplexityBot and others) managed from settings.
- **AI crawler analytics**: see which AI bots read your content in an admin
  report, recorded by an opt-in middleware.
- **Editor tools**: a "Machine readable content" admin report and a page-editor
  preview mode that shows pages as machines see them.
- **Static export**: a management command that writes every output to disk for
  static hosting and CDN workflows.
- **Wagtail-native visibility rules**: only `live` pages, view-restricted
  (private) subtrees excluded, drafts excluded, multi-site aware, plus per-model
  and per-page opt-outs.

## Quickstart

```bash
pip install wagtail-machine-readable
```

```python
# settings.py
INSTALLED_APPS = [
    # ...
    "wagtail_machine_readable",
]
```

```python
# urls.py, before Wagtail's catch-all page serving include
urlpatterns = [
    # ...
    path("", include("wagtail_machine_readable.urls")),
    path("", include(wagtail_urls)),
]
```

Zero configuration produces a valid llms.txt:

```
# Acme

> Acme makes modular widgets.

## Pages

- [Contact](https://example.com/contact/): Get in touch.

## About

- [About](https://example.com/about/): Who we are.
- [Team](https://example.com/about/team/): The people.

## Blog

- [Blog](https://example.com/blog/): News and articles.
- [First post](https://example.com/blog/first-post/): Our first post.
- [Second post](https://example.com/blog/second-post/)
```

Sections come from the children of each Site's root page, and descriptions come
from `search_description` by default. Every Wagtail `Site` gets its own
document on its own hostname.

## Markdown page variants

Every visible page is also served as Markdown by appending `.md` to its slug
path, so `/about/team/` becomes `/about/team.md`, and the site root is
`/index.md`. Responses use `text/markdown` and the same visibility rules as
llms.txt, so drafts and private pages 404. Set `MARKDOWN_ENABLED` to `False` to
turn the variants off.

Bodies are produced by `MarkdownContentExtractor`, which maps rich text and
StreamField blocks to Markdown: headings (demoted below the page title),
`[text](url)` links with rich-text page references expanded, `![alt](url)`
images, embeds and URL blocks as autolinks, and `TableBlock` values as Markdown
tables.

### Making the variants discoverable

Crawlers will not guess your URL convention, so advertise the variants. Add the
middleware:

```python
MIDDLEWARE = [
    # ...
    "wagtail_machine_readable.middleware.MachineReadableMiddleware",
]
```

This does two things for canonical page URLs:

- **Content negotiation**: a request with `Accept: text/markdown` ranked above
  HTML gets the Markdown variant directly (with `Vary: Accept` and a
  `Content-Location` header). Browsers never send this, so regular visitors are
  unaffected.
- **`Link` headers**: HTML page responses gain
  `Link: <.../team.md>; rel="alternate"; type="text/markdown"`.

And in your page templates, advertise the variant in the head:

```html+django
{% load machine_readable %}
{% markdown_alternate page %}
```

which renders `<link rel="alternate" type="text/markdown" href="...">`. The
middleware resolves the page for each negotiated or HTML page response, costing
one extra query. Skip it (and keep the template tag) if that matters at your
scale.

## Structured data (JSON-LD)

Add the template tag to your page template:

```html+django
{% load machine_readable %}
{% structured_data page %}
```

This renders a `<script type="application/ld+json">` element containing a
schema.org `@graph`: the page node (`WebPage` by default), a `BreadcrumbList`
from the page's ancestors and an `Organization` derived from the Wagtail Site.
Map page types to other builders (`Article` ships ready to use) or to your own
`StructuredDataBuilder` subclasses:

```python
WAGTAIL_MACHINE_READABLE = {
    "STRUCTURED_DATA_BUILDERS": {
        "blog.BlogPage": "wagtail_machine_readable.structured_data.ArticleBuilder",
    },
}
```

Mappings match base classes too, so `"wagtailcore.Page"` changes the default
for every page type.

## AI crawler robots.txt

Opt in to serving `/robots.txt` with explicit rules for known AI user agents:

```python
WAGTAIL_MACHINE_READABLE = {
    "ROBOTS_TXT_ENABLED": True,
    "ROBOTS_AI_DEFAULT": "allow",  # baseline for known AI agents
    "ROBOTS_AI_DENY": ["Bytespider"],  # per-agent overrides
    "ROBOTS_EXTRA": "Sitemap: https://example.com/sitemap.xml",
}
```

The built-in list covers GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot,
Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot,
Meta-ExternalAgent and others, and the allow and deny lists also accept agents
not on the list. Other crawlers get `User-agent: *` / `Allow: /`. If your
project already serves robots.txt, keep its URL pattern above the package
include.

## Editor tools

With `wagtail.admin` installed the package adds two reports (under **Reports**
in the admin menu):

- **Machine readable content**: every live page with its machine-readable
  status, covering whether a description is present, whether the extracted body
  is non-empty, and whether the page is excluded from outputs. Editors work
  through the "missing" and "empty" rows.
- **AI crawler activity**: recent requests from known AI crawlers (see below).

For a "view as machine" preview inside the page editor, adopt the preview mixin
(a plain class, no migration):

```python
from wagtail_machine_readable.models import MachineReadablePreviewMixin


class ArticlePage(MachineReadablePreviewMixin, Page): ...
```

Editors get a "Machine readable" entry in the preview-mode dropdown showing the
page exactly as its `.md` variant renders, so garbled extraction becomes
visible while editing, not after publishing.

## AI crawler analytics

Prove the machine-readable outputs are being read. Add the tracking middleware:

```python
MIDDLEWARE = [
    # ...
    "wagtail_machine_readable.middleware.AICrawlerTrackingMiddleware",
]
```

Requests whose User-Agent matches a known AI crawler (the same list the
robots.txt controls use, plus any agents in your allow and deny settings) are
recorded with agent, host, path and status code, and surfaced in the **AI
crawler activity** report with CSV and XLSX export. The hit model ships with its
own migration, so run `python manage.py migrate` after installing or upgrading.
Keep the table tidy from a cron job:

```bash
python manage.py prune_ai_crawler_hits --days 90
```

## Static generation

```bash
python manage.py generate_machine_readable --output ./static-export
python manage.py generate_machine_readable --site example.com
```

Single-site projects write `llms.txt`, `llms-full.txt`, the `.md` page tree
(`index.md`, `about/team.md` and so on) and, when enabled, `robots.txt` into the
output directory. Multi-site projects get one subdirectory per hostname.

## Caching

Responses carry `Cache-Control: public, max-age=3600` by default
(`CACHE_MAX_AGE`). For sites where generation itself is expensive, opt in to
caching the generated documents server-side:

```python
WAGTAIL_MACHINE_READABLE = {
    "GENERATION_CACHE_TIMEOUT": 86400,
}
```

Rendered documents are stored in Django's default cache and invalidated when a
page is published or unpublished, so a long timeout is safe.

## Settings

All configuration lives in a single dict. Every key has a working default, so
configure only what you want to change:

```python
WAGTAIL_MACHINE_READABLE = {
    "SITE_DESCRIPTION": "Acme makes modular widgets.",
    "MAX_PAGES_PER_SECTION": 25,
    "EXCLUDE_PAGE_MODELS": ["blog.BlogTagIndexPage"],
}
```

| Key | Default | Purpose |
| --- | --- | --- |
| `SITE_DESCRIPTION` | `None` | Blockquote description. A string, or a `{hostname: description}` dict for multi-site. Falls back to the root page's description chain. |
| `SECTION_STRATEGY` | `TopLevelSectionStrategy` path | Dotted path to the class deriving H2 sections from the page tree. |
| `MAX_PAGES_PER_SECTION` | `50` | Cap entries per section (`None` = unlimited). |
| `RESPECT_SHOW_IN_MENUS` | `False` | Only include pages with `show_in_menus=True`. |
| `EXCLUDE_PAGE_MODELS` | `[]` | Page models to exclude, as `"app_label.ModelName"` strings (exact type). |
| `EXCLUDE_PAGE_IDS` | `[]` | Specific page ids to exclude. |
| `DESCRIPTION_FIELDS` | `["machine_readable_description", "search_description"]` | Per-page description fallback chain; first non-empty field wins. |
| `FULL_TEXT_ENABLED` | `True` | Serve/write `llms-full.txt`. |
| `FULL_TEXT_MAX_PAGES` | `None` | Cap the number of pages in `llms-full.txt` (`None` = unlimited). |
| `MARKDOWN_ENABLED` | `True` | Serve/write `.md` page variants. |
| `EXTRACTOR` | `MarkdownContentExtractor` path | Dotted path to the `ContentExtractor` used for page bodies. |
| `CACHE_MAX_AGE` | `3600` | `Cache-Control: public, max-age=N` on responses (`0` = no header). |
| `GENERATION_CACHE_TIMEOUT` | `None` | Opt-in server-side caching of generated documents, in seconds. |
| `STRUCTURED_DATA_BUILDERS` | `{}` | Map `"app_label.ModelName"` to `StructuredDataBuilder` dotted paths. |
| `ROBOTS_TXT_ENABLED` | `False` | Serve/write `robots.txt` from this package. |
| `ROBOTS_AI_DEFAULT` | `"allow"` | Baseline policy (`"allow"`/`"deny"`) for known AI user agents. |
| `ROBOTS_AI_ALLOW` | `[]` | Agents to explicitly allow, overriding the baseline. |
| `ROBOTS_AI_DENY` | `[]` | Agents to explicitly deny, overriding the baseline. |
| `ROBOTS_EXTRA` | `""` | Raw text appended to robots.txt (for example a `Sitemap:` line). |

Misconfigured keys are caught by Django system checks at startup.

## Customisation

**Exclude a page type** with no mixin or migration needed:

```python
class InternalToolPage(Page):
    exclude_from_machine_readable = True
```

**Per-page editor controls**: adopt the optional mixin for a dedicated
AI-consumer description and a per-page exclusion flag, surfaced in the page
editor by `MachineReadablePanel`:

```python
from wagtail_machine_readable.models import MachineReadableMixin
from wagtail_machine_readable.panels import MachineReadablePanel


class ArticlePage(MachineReadableMixin, Page):
    promote_panels = Page.promote_panels + [MachineReadablePanel()]
```

The mixin adds fields to your page model, so run
`python manage.py makemigrations && python manage.py migrate` after adopting it
(and after upgrading from a version with fewer mixin fields). Until the
migration is applied, any query against the page model, including the pages
themselves, fails with a missing-column error.

**Custom sections**: subclass `SectionStrategy` and point `SECTION_STRATEGY` at
it:

```python
from wagtail_machine_readable.generators import SectionStrategy


class NavigationSections(SectionStrategy):
    def build_sections(self, site): ...
```

**Custom content extraction**: subclass `ContentExtractor` (or
`MarkdownContentExtractor`, whose block handling and `convert_html` hook are
overridable) and point `EXTRACTOR` at it. Set `EXTRACTOR` to
`"wagtail_machine_readable.extractors.DefaultContentExtractor"` for the
plain-text behaviour of v0.1.

**Custom structured data**: subclass `StructuredDataBuilder` (or
`WebPageBuilder`/`ArticleBuilder`) and map page types to it via
`STRUCTURED_DATA_BUILDERS`.

## How it behaves

- **Visibility**: a page appears only if it is live, has no view restriction on
  itself or an ancestor, and is not excluded by settings, the class attribute or
  the per-page flag. The private and draft rules match what anonymous visitors
  can already see, so nothing non-public leaks.
- **Multi-site**: `Site.find_for_request()` scopes every request, so each
  hostname serves its own tree with absolute URLs.
- **Caching**: responses carry `Cache-Control: public, max-age=3600` by
  default. The views are `cache_page`-compatible (cache keys include the host),
  so wrapping them or enabling Django's cache middleware works per site out of
  the box. Server-side generation caching is opt-in via
  `GENERATION_CACHE_TIMEOUT`, invalidated on publish and unpublish.
- **Output safety**: titles and descriptions are collapsed to single lines and
  escaped, so page content cannot inject sections or links into the document
  structure. JSON-LD payloads escape `<`, `>` and `&`, so page content cannot
  break out of the script element.

## How is this different from django-llms-txt?

django-llms-txt is Django-generic: you describe your content to it.
`wagtail-machine-readable` is Wagtail-native. It already understands the page
tree (sections for free), `live` and privacy rules, multi-site scoping,
`search_description`, and StreamField content. Point it at a Wagtail project and
it produces the right documents with zero configuration.

## Compatibility

Python 3.11 to 3.13. Django 4.2 and 5.2. Wagtail 6.3 to 7.x. The full matrix is
tested in CI.

## License

[MIT](LICENSE)
