This TYPO3 extension provides an alternative Markdown representation of your pages for AI bots and crawlers.
This makes content accessible not just for humans and screen readers, but also for AI systems that consume and process web content.
- Converts any TYPO3 page to Markdown on-the-fly via content negotiation
- Works automatically with your existing page templates - no TypoScript configuration needed
- Extracts content from
<main>element (or<body>as fallback) - Automatic
<link rel="alternate">tag for Markdown discovery - Strips navigation, header, footer, and other non-content elements
- Includes page metadata (title, dates, description, categories) as YAML front matter
Use composer req b13/ai-bots-love-markdown or install it via TYPO3's Extension Manager.
After installation, add the site set to your site configuration:
dependencies:
- b13/ai-bots-love-markdownThe extension provides the following settings that can be configured per site:
| Setting | Default | Description |
|---|---|---|
ai_bots_love_markdown.enableContentNegotiation |
true |
Enable content negotiation via Accept: text/markdown header |
ai_bots_love_markdown.enableDiscoveryTag |
true |
Add <link rel="alternate"> tag to HTML pages for Markdown discovery |
ai_bots_love_markdown.pageTypeSuffix |
ai-bots-love.md |
PageType Suffix for Markdown link |
ai_bots_love_markdown.pageTypeTypeNum |
2026 |
PageType TypeNum for Markdown link |
ai_bots_love_markdown.removeElements |
script style nav footer aside form iframe noscript |
Space-separated HTML tags stripped from the markdown output. <header> is intentionally not included — article-level <header> regions often hold the H1 |
ai_bots_love_markdown.excludedDoktypes |
3,4,6,7,199,254,255 |
Comma-separated page doktypes that never produce a Markdown alternate response. Default covers TYPO3 system doktypes (external link, shortcut, mountpoint, sysfolder, recycler, etc.) |
ai_bots_love_markdown.cacheable |
true |
Allow CDN / reverse-proxy caching of Markdown responses. Disable to force every hit through to the origin — required when you want to count AI-bot deliveries accurately (e.g. via b13/ai-bot-tracker). When false, Cache-Control: private, no-store is set on Markdown responses |
To override these settings, add them to your site's settings.yaml:
ai_bots_love_markdown.enableContentNegotiation: true
ai_bots_love_markdown.enableDiscoveryTag: false
ai_bots_love_markdown.pageTypeTypeNum: 1778074315
ai_bots_love_markdown.pageTypeSuffix: 'foo.md'
ai_bots_love_markdown.removeElements: 'script style nav footer aside form iframe noscript header'
ai_bots_love_markdown.excludedDoktypes: '3,4,6,7,199,254,255,37,38,41'
ai_bots_love_markdown.cacheable: falseEditors can disable the Markdown alternate for individual pages via the page property
Disable Markdown version (TCA field pages.markdown_version, default on, rendered
inverted in the BE so the toggle reads as "disable"). When the toggle is turned on, the
<link rel="alternate"> discovery tag is stripped from the HTML response and any direct
request to .md / Accept: text/markdown returns the regular HTML.
Markdown responses share the URL of the HTML page when Accept: text/markdown is
used. To prevent reverse proxies from serving a cached HTML response to a Markdown
requester (or vice versa), the middleware adds a Vary: Accept header to every
response that goes through a site with content negotiation enabled — regardless
of whether the current request asked for Markdown.
By default (cacheable: true), TYPO3's page-level Cache-Control is preserved
on the Markdown response, so the CDN may cache. If you need every Markdown
delivery to reach the origin — typically to track bot consumption — set
cacheable: false and the response is sent with Cache-Control: private, no-store.
-
Accept header: Request any page with
Accept: text/markdownheadercurl -H "Accept: text/markdown" https://example.com/my-page/ -
URL suffix: Append
/ai-bots-love.mdto any page URLhttps://example.com/ai-bots-love.md
The extension outputs Markdown with YAML front matter, extracting metadata from HTML meta tags with fallback to TYPO3 page record data:
---
title: "From og:title > <title> > page record"
url: "From canonical URL > request URL"
description: "From og:description > meta description > page record"
image: "From og:image (if present)"
author: "From meta author > page record"
date: 2024-01-15
modified: 2024-01-20
keywords:
- From meta keywords
categories:
- From TYPO3 categories
type: "From og:type (if present)"
site: "From og:site_name (if present)"
locale: "From og:locale (if present)"
---
# Page Title
Content converted to Markdown...Metadata Priority:
title: og:title →<title>tag → page recordurl: canonical link → request URLdescription: og:description → meta description → page recordimage: og:image (if present)author: meta author → page recorddate/modified: page record (crdate/tstamp)keywords: meta keywords (if present)categories: TYPO3 sys_category (from database)
The extension automatically adds a <link> tag to all HTML pages for Markdown discovery:
<link rel="alternate" type="text/markdown" href="https://example.com/my-page/ai-bots-love.md" title="Markdown version" />Pages are only converted to Markdown if they contain the <link rel="alternate" type="text/markdown"> tag.
This means:
- Pages without the site set enabled won't be converted
- You can disable conversion for specific pages by disabling the discovery tag
- A PSR-15 middleware intercepts requests with
Accept: text/markdownheader or/ai-bots-love.mdsuffix - The normal page rendering proceeds (your existing templates, TypoScript, etc.)
- The middleware checks if the response contains the markdown alternate link tag
- If present, it extracts
<main>content (or<body>as fallback) - The HTML is converted to Markdown using league/html-to-markdown
- Navigation, header, footer, and other non-content elements are stripped
- Page metadata is added as YAML front matter
The Markdown response includes:
Content-Type: text/markdown; charset=utf-8X-Robots-Tag: noindex(to prevent indexing of Markdown version)
For best results, wrap your main content in a <main> element:
<body>
<header>...</header>
<nav>...</nav>
<main>
<!-- This content will be converted to Markdown -->
<f:render section="Main" />
</main>
<footer>...</footer>
</body>Beyond the <main> selection and the configurable removeElements list, you
can mark template regions explicitly. This gives editors and integrators
fine-grained control over what ends up in the Markdown representation, without
affecting the HTML page that humans see.
Two bundled Fluid partials do the work. They are auto-registered through the
site set, so no partialRootPaths setup is required — just <f:render> them.
Wraps the region that should become the Markdown body. Use this when
<main>-based detection is not good enough — e.g. there is no <main>
element, or <main> contains too much (sidebars, related teasers, in-page
navigation).
<f:render partial="MarkdownInclude" contentAs="content">
<article>… the real article content …</article>
</f:render>When include markers are present, everything outside them is ignored for
the Markdown output, including content that would otherwise be picked up from
<main> or <body>.
Wraps a region that should not appear in the Markdown output even though it lives inside the content area — teasers, related-article boxes, breadcrumbs, CTAs, cookie notices, share buttons.
<f:render partial="MarkdownExclude" contentAs="content">
<div class="teaser">Read also: …</div>
</f:render>Exclude regions can be nested, and they can sit inside an include region.
Each MarkdownExclude block is removed as a whole; the surrounding content is
kept.
When a Markdown response is built, the content region is resolved in this
priority order (see MarkdownResponseMiddleware):
- Include markers — the content between
<!-- markdown-start -->and<!-- markdown-end -->, if both are present. <main>element — the contents of the first<main>…</main>.<body>element — the contents of<body>…</body>as a fallback.- Whole document — last resort if none of the above match.
After the region is selected, every MarkdownExclude block inside it is
removed (depth-aware, so nested excludes are handled correctly), and any
residual markers are stripped before the HTML is converted to Markdown.
The partials emit plain HTML comments around their content:
| Partial | Opening marker | Closing marker |
|---|---|---|
MarkdownInclude |
<!-- markdown-start --> |
<!-- markdown-end --> |
MarkdownExclude |
<!-- markdown-exclude-start --> |
<!-- markdown-exclude-end --> |
Both partials only emit markers when their content is non-empty, so an empty
region produces no markers at all.
Because the markers are ordinary HTML comments, they survive TYPO3's page cache — they are baked into the cached page body together with your content. That is intentional: the Markdown conversion needs them on every request, including fully-cached ones.
Markers must not be visible to regular visitors. The
StripMarkdownMarkersMiddleware removes all four markers from every
text/html response before it leaves the server. (Markdown responses already
have the markers stripped during conversion, so they are skipped by the
content-type check.)
Note on robustness: marker stripping shortens the body, so the middleware also
drops the stale Content-Length header — leaving it in place would otherwise
cause an HTTP/2 RST_STREAM / INTERNAL_ERROR in browsers because the
declared length no longer matches the bytes on the wire.
To see the markers in the rendered HTML (e.g. to verify your partials are
wired up correctly), append ?markdown-markers=1 to any page URL. The
stripping middleware then leaves the markers in the HTML response.
This bypass is gated for safety: it only works for authenticated backend users or in non-production application contexts, so the markers are never disclosed to anonymous crawlers on a production site.
The extension dispatches three PSR-14 events. Listen via the
#[AsEventListener] attribute or by registering a listener in your
extension's Configuration/Services.yaml. All events live under the
B13\AiBotsLoveMarkdown\Event\ namespace.
| Event | Fired | Use case |
|---|---|---|
BuildHtmlMarkdownConverterEvent |
Before HTML → Markdown conversion runs | Add custom node converters, override HtmlConverter options |
AfterFrontMatterForPageIsCreatedEvent |
After the YAML front-matter array is assembled from meta tags and the page record, before serialisation | Add, remove, or replace front-matter entries (e.g. enrich with domain-specific keys) |
AfterMarkdownConversionEvent |
After Markdown content has been built, before the response is returned | Side effects on every Markdown delivery (e.g. b13/ai-bot-tracker writes a tracking row from this event) |
See Extending the YAML front matter below
for a full AfterFrontMatterForPageIsCreatedEvent listener example.
The default front matter is built from HTML meta tags and the page record. To
add domain-specific keys (e.g. seminar data, product attributes, event dates),
listen to the AfterFrontMatterForPageIsCreatedEvent. Listeners receive the assembled
data array before it is rendered to YAML and may add, remove, or replace
entries.
<?php
declare(strict_types=1);
namespace MyVendor\MySite\EventListener;
use B13\AiBotsLoveMarkdown\Event\AfterFrontMatterForPageIsCreatedEvent;
use MyVendor\MySite\Domain\Repository\SeminarRepository;
use TYPO3\CMS\Core\Attribute\AsEventListener;
final readonly class AddSeminarFrontMatter
{
public function __construct(
private SeminarRepository $seminarRepository,
) {}
#[AsEventListener]
public function __invoke(AfterFrontMatterForPageIsCreatedEvent $event): void
{
$pageId = (int)($event->pageRecord['uid'] ?? 0);
$seminar = $this->seminarRepository->findByDetailPageId($pageId);
if ($seminar === null) {
return;
}
$event->frontMatter['seminar_title'] = $seminar->getTitle();
$event->frontMatter['seminar_instructor'] = $seminar->getInstructor()?->getDisplayName();
$event->frontMatter['seminar_price_eur'] = number_format($seminar->getPriceCents() / 100, 2, '.', '');
$event->frontMatter['seminar_dates'] = array_map(
static fn (\DateTimeImmutable $d) => $d->format('Y-m-d'),
$seminar->getDates(),
);
}
}The event also exposes $html, $pageInformation, and $request (each
read-only) so listeners can resolve site, language, or routing information
when needed. Array order is preserved in the rendered YAML, so you can
prepend custom keys by recreating the array if order matters.
If you need to bypass the YAML rendering entirely, the underlying methods
MetadataService::buildFrontMatterData() and MetadataService::renderYaml()
are public and can be used directly.
The extension is licensed under GPL v2+, same as the TYPO3 Core.
This extension was created by b13 GmbH in 2026 to enable AI systems to better consume website content. As AI crawlers become increasingly important for content discovery and processing, providing a clean Markdown representation helps ensure your content is accurately understood and indexed by AI systems.
Huge credits to Dries Buytaert's inspiration on this topic:
Find more TYPO3 extensions we have developed that help us deliver value in client projects. As part of the way we work, we focus on testing and best practices to ensure long-term performance, reliability, and results in all our code.