Skip to content

AGENTS.md

Latest commit

 

History

History
270 lines (202 loc) · 9.95 KB

AGENTS.md

File metadata and controls

270 lines (202 loc) · 9.95 KB
Raw

wp-static-clone — Implementation recipes

Detailed commands and recipes for each phase. Read this when you're implementing a specific step. Parent SKILL.md has the workflow, gotchas, and output structure.

Phase 1 — Sitemaps and Yoast XSL

Fetch the index, then any child sitemaps it references, then concatenate page URLs into urls.txt. Adjust grep on the <loc> extraction if the source uses non-Yoast naming.

mkdir -p output
ROOT=https://example.com   # source site origin

curl -s -A "Mozilla/5.0" "$ROOT/sitemap_index.xml" \
  | tee output/sitemap_index.xml \
  | grep -oE '<loc>[^<]+</loc>' \
  | sed -E 's|</?loc>||g' \
  | grep -v 'image-sitemap' \
  > child-sitemaps.txt

while read -r child; do
    name=$(basename "$child")
    curl -s -A "Mozilla/5.0" "$child" -o "output/$name"
    grep -oE '<loc>[^<]+</loc>' "output/$name" \
        | sed -E 's|</?loc>||g'
done < child-sitemaps.txt | sort -u > urls.txt

# Yoast XSL (purely so sitemaps render in browser; search engines parse XML directly)
mkdir -p output/wp-content/plugins/wordpress-seo/css
curl -s -A "Mozilla/5.0" \
    "$ROOT/wp-content/plugins/wordpress-seo/css/main-sitemap.xsl" \
    -o output/wp-content/plugins/wordpress-seo/css/main-sitemap.xsl

Phase 2 — One-shot wget

Single invocation over urls.txt. The browser UA + Accept / Accept-Language headers are mandatory — Cloudflare's bot protection 403s the default Wget/1.x UA.

wget -p -k -E -nH \
    --restrict-file-names=unix \
    -e robots=off \
    -U "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36" \
    --header="Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8" \
    --header="Accept-Language: en,*;q=0.5" \
    --wait=0.3 --random-wait \
    -i urls.txt -P output/

Flags:

  • -p page requisites (CSS / JS / images / fonts referenced from HTML)
  • -k convert links to local
  • -E add .html extension; /foo/ → foo/index.html
  • -nH skip the host directory wrapper
  • --restrict-file-names=unix keep filenames clean on macOS / Linux
  • -e robots=off ignore robots.txt
  • --wait=0.3 --random-wait paces requests so Cloudflare doesn't escalate to 403 partway through

If a non-English source has localised content negotiation, swap Accept-Language accordingly. Language doesn't matter to anything else in the pipeline (the runtime-strip and reply-link regexes match by class, not text).

Phase 3 — Pull missing absolutely-referenced assets

wget -p follows <script src>, <link rel="stylesheet">, <img src> / srcset, <source>, etc. It does not follow og:image, apple-touch-icon, JSON-LD image / logo, msapplication-TileImage, or <link rel="modulepreload">. Audit and fetch the long tail across all three asset roots:

SRC=example.com
grep -rhoE "https?://$SRC/wp-content/(uploads|themes|plugins)/[^\"<'> ]*" output/ \
    | sort -u > missing.txt

wget -x -nH -e robots=off \
    -U "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ..." \
    -i missing.txt -P output/

-x (force directories) is important so files land at their original paths inside wp-content/....

Phase 5 — Yoast XSL attribution edit

import pathlib

p = pathlib.Path('output/wp-content/plugins/wordpress-seo/css/main-sitemap.xsl')
old = ('Generated by <a href="https://yoa.st/1y5" target="_blank" '
       'rel="noopener">Yoast SEO</a>, this is an XML Sitemap, meant '
       'for consumption by search engines.')
new = ('Generated statically and deployed via a static host, this is an XML '
       'Sitemap, meant for consumption by search engines.')
p.write_text(p.read_text(encoding='utf-8').replace(old, new), encoding='utf-8')

Leave the <a href="https://sitemaps.org"> link further down — it's still factually correct.

Phase 6 — Plausible analytics

WordPress's Plausible plugin proxies the script through /wp-content/uploads/<hash>/pa-XXX.js and posts events back to /wp-json/.../v1/.... Replace the entire two-script block with the standard tracker:

import pathlib, re

PATTERN = re.compile(
    r'<script[^>]*id="plausible-analytics-js"></script>\s*'
    r'<script id="plausible-analytics-js-after">.*?</script>',
    re.DOTALL,
)
REPLACEMENT = (
    '<script defer data-domain="example.com" '
    'src="https://plausible.io/js/script.js"></script>'
)
for path in pathlib.Path('output').rglob('*.html'):
    text = path.read_text(encoding='utf-8')
    new = PATTERN.sub(REPLACEMENT, text)
    if new != text:
        path.write_text(new, encoding='utf-8')

Then delete the now-unused proxy file at output/wp-content/uploads/<hash>/pa-*.js.

Phase 6 — Audit remaining absolute URLs

After scripts/strip-wp-runtime.py and scripts/selfhost-gravatars.py, audit for any source-domain URLs left:

SRC=example.com
grep -hE "https?://$SRC/" --include="*.html" -r output/ \
    | grep -vE 'rel="canonical"|og:url|"@id"|"url"|"item"|wp-json/oembed' \
    | sort -u

Anything left after the canonical / JSON-LD allow-list is something to triage. Common offenders:

  • Author archive links (<a class="wp-block-post-author-name__link" href=".../author/<slug>/">). On single-author sites WordPress often 301s these to the homepage. Probe with curl -sI -o /dev/null -w "%{redirect_url}\n" <author-url>. If they redirect to /, strip the <a> wrapper and keep the inner text — don't waste a directory on a duplicate of the home page.
  • modulepreload absolute URLs (<link rel="modulepreload" href="https://<domain>/wp-includes/...">). Convert to root-relative and fetch the underlying file (wget -p doesn't follow modulepreload).
  • Server-rendered iframes pointing at custom PHP endpoints (e.g. <iframe src="/<custom>/<script>.php">). These won't render statically. Drop the wrapping <p>.
  • Gravity Forms script blocks — wrappers have inline <script> tags (no id, just bare <script> tags referencing gform.*) above and below the form, plus an in-<head> polyfill. Strip them all — match <script>...</script> whose body mentions gform. Otherwise the page errors at load when gform is undefined.
  • gform_not_found paragraphs — pre-existing brokenness on the live site (failed shortcode render). Strip the paragraph and any orphaned lead-in heading.
  • Page archive pagination anchors (<a href=".../page/N/">). The /page/N/ URLs aren't in the sitemap, so the targets don't exist. Strip the link, keep the anchor text only.

Phase 7 — robots.txt

ROOT=https://example.com
curl -s -A "Mozilla/5.0" "$ROOT/robots.txt" -o output/robots.txt

# If you flatten output/* into the repo root, the sitemap reference is fine.
# If you keep things under output/ and configure the host accordingly, leave alone.
# If the deployed sitemap path differs, edit:
#   sed -i.bak -E 's|^Sitemap: .*$|Sitemap: https://new.example.com/sitemap_index.xml|' output/robots.txt
# (Delete the .bak after — `-i.bak` is the portable form that works on both BSD and GNU sed.)

Phase 8 — Verify

cd output && python3 -m http.server 8765 &
PORT=8765

# 1. Every URL in urls.txt resolves
while read -r url; do
    path=$(echo "$url" | sed -E 's|^https?://[^/]+||; s|/?$|/|')
    code=$(curl -sI -o /dev/null -w "%{http_code}" "http://localhost:$PORT$path")
    [ "$code" = "200" ] || echo "MISS $code  $path"
done < ../urls.txt

# 2. No remaining absolute source-domain URLs outside the allow-list
grep -hE "https?://example\.com/" --include="*.html" -r . \
    | grep -vE 'rel="canonical"|og:url|"@id"|"url"|"item"|wp-json/oembed' \
    | head

# 3. Spider-crawl for broken internal links
wget --spider -r -nd --no-parent -l 5 "http://localhost:$PORT/" 2>&1 \
    | grep -E '404|broken'

Then open the homepage and one deep page in a browser. Watch srcset images, sidebar widgets, the header banner — those break silently if missed.

Phase 9 — Deploy

Pick the recipe matching the target host from Phase 0.

Cloudflare Pages

Defaults: no build command, no build output directory, no root directory. Pages serves the repo root.

If the site has any URL changes (slug changes, dropped /page/N/ archive pagination, redirect from /category/<term>/ to a flat /blog/, etc.), add _redirects at the repo root:

/old-path/        /new-path/        301
/category/*       /blog/            301

Long-cache the hashed assets via _headers:

/wp-content/*
  Cache-Control: public, max-age=31536000, immutable

/wp-includes/*
  Cache-Control: public, max-age=31536000, immutable

Push to GitHub and Pages auto-deploys. gh repo create <user>/<repo> --private --source=. --push works.

Netlify

Same _redirects and _headers syntax as Cloudflare Pages. Optional netlify.toml:

[build]
publish = "."

Connect the repo via the Netlify dashboard.

Vercel

vercel.json at the repo root:

{
    "redirects": [
        { "source": "/old-path/", "destination": "/new-path/", "permanent": true }
    ],
    "headers": [
        {
            "source": "/wp-content/(.*)",
            "headers": [{ "key": "Cache-Control", "value": "public, max-age=31536000, immutable" }]
        }
    ],
    "cleanUrls": false,
    "trailingSlash": true
}

trailingSlash: true matches WordPress's default URL shape (and what wget -E produces). If the source uses no trailing slashes, set to false.

Generic (nginx)

server {
    listen 80;
    server_name example.com;
    root /var/www/wp-static-clone;
    index index.html;

    # Trailing-slash URLs map to <slug>/index.html
    location / {
        try_files $uri $uri/ $uri/index.html =404;
    }

    location ~ ^/wp-content/ {
        expires 1y;
        add_header Cache-Control "public, immutable";
    }
}

Generic (Apache)

.htaccess at the document root:

Options +MultiViews
DirectoryIndex index.html

<FilesMatch "^.+\.(jpg|jpeg|png|webp|svg|css|js|woff2?)$">
    Header set Cache-Control "public, max-age=31536000, immutable"
</FilesMatch>

ErrorDocument 404 /404.html