[Feature Request]: page-level AI manifest endpoint for crawled pages

[aagedik]

What needs to be done?

Add optional support for discovering and parsing a page-level AI manifest when available.

For example, if a page exists at:

https://example.com/blog/my-post

the crawler could optionally check for:

https://example.com/blog/my-post.ai.json

If present, the manifest could be parsed alongside the HTML and exposed as an optional structured artifact.

The specification is experimental and open for feedback:

https://aigeoradar.com/specifications/aipm

What problem does this solve?

Current AI crawlers primarily rely on HTML parsing, Schema.org, metadata extraction and heuristic processing.

A dedicated page-level manifest could provide:

• structured page metadata
• entities
• canonical information
• knowledge graph references
• AI-specific metadata

This may reduce repeated extraction work, simplify retrieval pipelines and provide a deterministic metadata source when publishers choose to expose one.

The feature would remain completely optional and would not affect existing crawling behaviour.

Target users/beneficiaries

• AI crawler developers
• Search infrastructure teams
• RAG pipelines
• LLM retrieval systems
• CMS plugin developers
• Publishers exposing structured AI metadata

Current alternatives/workarounds

Today most systems combine information from:

• HTML
• robots.txt
• sitemap.xml
• Schema.org
• llms.txt

These are valuable, but none provide a dedicated machine-readable page manifest designed specifically for AI systems.

The proposed manifest is intended to complement—not replace—existing standards.

Proposed approach

The crawler could expose an optional flag such as:

discover_ai_manifest=True

When enabled:

1. Crawl the HTML page.
2. Check whether <page>.ai.json exists.
3. If found (HTTP 200), parse it.
4. Return the parsed manifest as an optional result alongside the HTML extraction.

This keeps the feature lightweight, backwards compatible and entirely optional.

Example implementation:

page.html
page.markdown
page.metadata
page.ai_manifest