fix(engine-bootstrap): route doc-site URLs to kind: web-doc (#4)

PR #3 introduced web-doc as the fourth source kind across the artifact
contract, REFRESH/DISCOVER probes, verify.sh enum, recipes, and the
Step 3.6 cache-seed flow — but never updated bootstrap's Step 1 intake
classifier. URL inputs continued to fall through to `kind: external-doc`,
producing structurally invalid entries (external-doc is path-addressed
local markdown, not URL-addressed) and making Step 3.6 unreachable.

Re-route the intake table, repo/doc disambiguator, bare-org guardrail,
Step 2 slug table, and Step 3 stamping to produce kind: web-doc with
default crawl_mode: "sitemap". Document that bootstrap never produces
external-doc.

Harden verify.sh with per-kind url/path required-field checks so a
malformed entry is rejected at validation time (defense-in-depth for
the silent corruption hand-edits could still produce). Add fixtures
pinning the rejected shapes and the expected post-fix bootstrap output.

Author: nick-railsback

plugin/skill-engine/docs/02-artifact-contract.md

@@ -84,7 +84,7 @@ The `<area-domain>-` prefix in `references/<area-domain>-*.md` is a routing sign
 
 The bijection invariant (see "The bijection invariant" below) is unaffected by prefix choice - that contract holds regardless of how `<area-domain>-*` is filled in. The guidance here is about routing UX, not contract compliance.
 
-### External-doc sources on source-paths.json
+### `kind: "external-doc"`
 
 A `research/source-paths.json` entry can carry an optional `kind` discriminator that names the harvest treatment the engine applies to a source root. When `kind` is absent — the existing-corpus state for every contextualizer that has shipped before this addition — the entry receives the **git-managed source-root** treatment documented in [`03-engine.md`](03-engine.md): per-source SHA via `git rev-parse HEAD`, then sparse-clone or shallow-clone crawl when the SHA has changed. Back-compat is total — every existing source-paths.json entry continues to behave exactly as it did before the discriminator was introduced; `kind` is purely additive. Entries that explicitly set `kind: "external-doc"` receive the external-doc treatment described below.
 
@@ -96,7 +96,9 @@ A `research/source-paths.json` entry can carry an optional `kind` discriminator
 }
 ```
 
-**What `kind: "external-doc"` means.** The `path` field points at pre-curated markdown content that lives outside any code repository — for example, a generic accessibility reference, a SharePoint-style compliance snapshot, an authored markdown sourced outside the navigated code repos. The engine treats this content as a first-class source for [DISCOVER](08-discover-pipeline.md) without applying the git-managed SHA-then-clone flow. Harvest semantics:
+**What `kind: "external-doc"` means.** The `path` field points at pre-curated markdown content that lives outside any code repository — for example, a generic accessibility reference, a SharePoint-style compliance snapshot, an authored markdown sourced outside the navigated code repos. The engine treats this content as a first-class source for [DISCOVER](08-discover-pipeline.md) without applying the git-managed SHA-then-clone flow. external-doc is **not a bootstrap-intake kind** — it carries a contextualizer-internal `path`, not a URL. Entries of this kind arrive in `research/source-paths.json` via DISCOVER, hand-edit, or a future workflow; the engine-bootstrap scaffolder produces `kind: "web-doc"` for doc-site URLs (see [`kind: "web-doc"`](#kind-web-doc)).
+
+Harvest semantics:
 
 * **Directory `path`.** The `path` resolves to a directory containing one-or-more `.md` files, scanned **recursively** so nested subdirectories are included (the exact walk recipe — `find -L`, the `-type f -o -type l` filter, and symlink handling — is canonicalized in the Symlink containment paragraph below and the `external-doc-frontmatter` named check in the contextualizer-side `verify.sh` the plugin stamps at bootstrap). Recursion is deliberate — external-doc directories typically mirror upstream wiki or SharePoint hierarchies the maintainer has not flattened; a shallow scan would silently skip the bulk of the content.
 * **Single-file `path`.** A contextualizer that ships exactly one external-doc `.md` file (the canonical example: one accessibility best-practices markdown injected at a known path) supports `kind: "external-doc"` with `path` resolving directly to the `.md` file rather than wrapping it in a directory. The verify check handles both shapes uniformly.

plugin/skill-engine/docs/09-discover-config.md

@@ -55,7 +55,7 @@ values are accepted; the canonical schema for each lives in
 | `kind` | What it harvests | Required schema fields beyond the base | Canonical doctrine |
 |---|---|---|---|
 | `git-managed` | Git-hosted source code. `url` required. | optional `branch` | [`02-artifact-contract.md` §"source-paths.json entry shape"](02-artifact-contract.md#source-pathsjson-entry-shape) |
-| `external-doc` | Pre-curated `.md` content outside any code repo. `path` required (directory or single file). | `.md` files carry provenance frontmatter (`source_url`, `crawl_date`, `decay`) | [`02-artifact-contract.md` §"`kind: "external-doc"`"](02-artifact-contract.md#external-doc-sources-on-source-pathsjson) |
+| `external-doc` | Pre-curated `.md` content outside any code repo. `path` required (directory or single file). | `.md` files carry provenance frontmatter (`source_url`, `crawl_date`, `decay`) | [`02-artifact-contract.md` §"`kind: "external-doc"`"](02-artifact-contract.md#kind-external-doc) |
 | `web-doc` | Documentation-site content acquired via WebFetch or MCP fetch. `url` and `crawl_mode` required. | `sitemap_url` (sitemap mode, optional) or `page_list` (list mode, required); optional `crawl_filters`, `crawl_budget`; `branch` rejected | [`02-artifact-contract.md` §"`kind: "web-doc"`"](02-artifact-contract.md#kind-web-doc) |
 | `local-path` | Non-git local-filesystem source. `path` required. | none | [`02-artifact-contract.md` §"source-paths.json entry shape"](02-artifact-contract.md#source-pathsjson-entry-shape) |
 

plugin/skill-engine/engine-bootstrap-templates/verify.sh

@@ -213,10 +213,44 @@ else
             *) fail "sources[$idx] ($id): status '$status' not in {intake, proposed, confirmed, rejected}"; entries_ok=0 ;;
           esac
         fi
-        if [ -z "$url" ] && [ -z "$path" ]; then
-          fail "sources[$idx] ($id): neither url nor path is set — at least one is required"
-          entries_ok=0
-        fi
+        case "$kind" in
+          git-managed)
+            if [ -z "$url" ]; then
+              fail "sources[$idx] ($id): url is required when kind is git-managed"
+              entries_ok=0
+            fi
+            ;;
+          web-doc)
+            if [ -z "$url" ]; then
+              fail "sources[$idx] ($id): url is required when kind is web-doc"
+              entries_ok=0
+            fi
+            if [ -n "$path" ]; then
+              fail "sources[$idx] ($id): path '$path' set on kind 'web-doc' — web-doc sources are URL-addressed, not path-addressed"
+              entries_ok=0
+            fi
+            ;;
+          external-doc)
+            if [ -z "$path" ]; then
+              fail "sources[$idx] ($id): path is required when kind is external-doc"
+              entries_ok=0
+            fi
+            if [ -n "$url" ]; then
+              fail "sources[$idx] ($id): url '$url' set on kind 'external-doc' — external-doc sources are path-addressed (pre-curated local markdown), not URL-addressed"
+              entries_ok=0
+            fi
+            ;;
+          local-path)
+            if [ -z "$path" ]; then
+              fail "sources[$idx] ($id): path is required when kind is local-path"
+              entries_ok=0
+            fi
+            if [ -n "$url" ]; then
+              fail "sources[$idx] ($id): url '$url' set on kind 'local-path' — local-path sources are filesystem-addressed, not URL-addressed"
+              entries_ok=0
+            fi
+            ;;
+        esac
         if [ -n "$branch" ]; then
           if [ "$kind" != "git-managed" ]; then
             fail "sources[$idx] ($id): branch '$branch' set on kind '$kind' — branch is git-managed only"

plugin/skill-engine/skills/engine-bootstrap/SKILL.md

@@ -88,8 +88,7 @@ Accept all of these source-input shapes:
 | `git@github.com:<org>/<repo>.git` | git-managed source on GitHub (SSH form) |
 | `git+ssh://...` | git-managed source (generic SSH) |
 | `https://gitlab.com/<group>/<repo>`, `https://bitbucket.org/<user>/<repo>` | git-managed source (other hosts) |
-| `https://<host>/<path>/<page>.html` | external-doc source (single doc) |
-| `https://<host>/<path>/docs/...` (no git-host signal) | external-doc source (doc site) |
+| `https://<host>/<path...>` (any HTTP/HTTPS URL with no git-host signal) | web-doc source (default `crawl_mode: sitemap`) |
 | Absolute local path (`/Users/...`, `~/...`, `/home/...`) | local-path source |
 | Relative local path (`./foo`, `../bar`, bare `foo` referencing an existing dir) | local-path source (resolved to absolute at intake) |
 
@@ -104,11 +103,13 @@ either a git source or a doc site), ask one targeted question in
 user-language — never the engine's `kind` value directly. The canonical
 disambiguator:
 
-> Is `<input>` a source-code repo or a standalone document?
+> Is `<input>` a source-code repo or a documentation site?
 
 Accept `repo` / `doc` (or full words) and map internally: `repo` → `kind:
-git-managed`, `doc` → `kind: external-doc`. **On any other response**
-(blank `<Enter>`, `local`, `quit`, typo) re-prompt with:
+git-managed`, `doc` → `kind: web-doc` (the engine never produces
+`kind: external-doc` from URL intake — see *What this skill does NOT do*
+below). **On any other response** (blank `<Enter>`, `local`, `quit`,
+typo) re-prompt with:
 
 > Please answer `repo` or `doc` — or enter `q` to skip just this entry and
 > continue with the rest of the intake.
@@ -122,9 +123,9 @@ land in `source-paths.json`.
 
 A URL of the form `https://github.com/<org>` (no `<repo>` segment) is
 neither a recognizable git source nor a docs page — it points at an org
-landing page. **Don't fall through to the external-doc catch-all**; that
-would silently stamp `kind: external-doc` against a URL the engine
-cannot meaningfully crawl. Instead, re-prompt:
+landing page. **Don't fall through to the web-doc catch-all**; that
+would silently stamp `kind: web-doc` against a URL whose sitemap and
+page list the engine cannot meaningfully resolve. Instead, re-prompt:
 
 > `<url>` looks like a GitHub org landing page, not a specific repo or
 > doc. Paste the URL of a specific repo (e.g., `https://github.com/<org>/<repo>`)
@@ -145,7 +146,7 @@ For each accepted source, compute the following without prompting the user:
 |---|---|
 | `https://github.com/<org>/<repo>` | `<org>-<repo>` (lowercase; non-alphanumerics → hyphen; collapse runs) |
 | `git@github.com:<org>/<repo>.git` | `<org>-<repo>` (same rule, drop `.git`) |
-| `https://<host>/<path...>` (external-doc) | last meaningful path segment, lowercased; if it's a file, drop the extension |
+| `https://<host>/<path...>` (web-doc) | last meaningful path segment, lowercased; if it's a file, drop the extension. If the URL has no path segments (host-root like `https://docs.example.com/`), fall back to the host with non-alphanumerics → hyphen (e.g., `docs-example-com`). |
 | Local absolute or relative path | basename of the resolved absolute path, lowercased |
 
 On collision (two sources slug to the same id), append `-2`, `-3`, ... to the
@@ -263,31 +264,60 @@ locates the root itself from the project working directory.
 ### Stamping `research/source-paths.json`
 
 Replace the empty `"sources": []` from the template with one entry per
-intaken source, in the order supplied. Each entry:
+intaken source, in the order supplied. The per-entry shape depends on
+`kind`:
+
+**`kind: "git-managed"`** — set `url`; add `"branch": "<name>"` only if
+Step 2.4 recorded a non-default branch:
+
+```json
+{
+  "id": "<computed-slug>",
+  "kind": "git-managed",
+  "url": "<original-url>",
+  "status": "intake",
+  "archived": false,
+  "lifecycle": { "state": "unknown", "last_checked": null, "last_checked_sha": null, "proposed_url": null },
+  "discovered_via": null
+}
+```
+
+**`kind: "web-doc"`** — set `url`; default `crawl_mode` to `"sitemap"`.
+Bootstrap does not resolve the sitemap or page list here; Step 3.6
+populates the cache and the optional `sitemap_url` / `page_list` fields
+remain absent until the user edits them (or DISCOVER proposes them):
 
 ```json
 {
   "id": "<computed-slug>",
-  "kind": "<git-managed | external-doc | local-path | web-doc>",
-  "url": "<original-url-if-url>",
-  "path": "<resolved-absolute-path-if-local>",
+  "kind": "web-doc",
+  "url": "<original-url>",
+  "crawl_mode": "sitemap",
   "status": "intake",
   "archived": false,
   "lifecycle": { "state": "unknown", "last_checked": null, "last_checked_sha": null, "proposed_url": null },
   "discovered_via": null
 }
 ```
 
-Set `url` only on URL-shaped sources; set `path` only on local-path sources.
-The fields are not exclusive at the schema level (a git-managed source the
-user has cloned locally could carry both), but at intake only one is set.
+**`kind: "local-path"`** — set `path` to the resolved absolute path:
+
+```json
+{
+  "id": "<computed-slug>",
+  "kind": "local-path",
+  "path": "<resolved-absolute-path>",
+  "status": "intake",
+  "archived": false,
+  "lifecycle": { "state": "unknown", "last_checked": null, "last_checked_sha": null, "proposed_url": null },
+  "discovered_via": null
+}
+```
 
-If the user supplied a non-default branch in Step 2.4 for this source,
-add a `"branch": "<name>"` key alongside `url`. Omit the key entirely
-when the user accepted the default — downstream code defaults to HEAD
-when the field is absent. Never record an explicit `"branch": "main"`
-or `"branch": "master"` from Step 2.4; the absent-field convention is
-load-bearing for the future-proofing reason documented in Step 2.4.
+Bootstrap does **not** produce `kind: "external-doc"` entries: that kind
+is for pre-curated local `.md` content addressed by a contextualizer-
+internal `path`, not for a URL the user pastes at intake. External-doc
+entries land in `source-paths.json` via DISCOVER or hand-edit.
 
 `schema_version: 1` from the template stays as-is. The schema is additive;
 existing v1 files continue to parse cleanly.
@@ -641,3 +671,7 @@ the cache explicitly via `/skill-engine:clean-cache`.
   intake would force a multi-source intake to abort halfway; failing on
   DISCOVER lets the user paste the whole list and address broken entries
   in batch.
+- It does not produce `kind: "external-doc"` entries. external-doc
+  sources are pre-curated local markdown addressed by a contextualizer-
+  internal `path` (see [`02-artifact-contract.md`](https://github.com/nick-railsback/skill-engine/blob/main/plugin/skill-engine/docs/02-artifact-contract.md#kind-external-doc)); they arrive in `source-paths.json`
+  via DISCOVER or hand-edit, not via URL intake.

plugin/skill-engine/tests/web-doc/fixtures/schema/invalid/invalid--external-doc-missing-path.expect

@@ -0,0 +1 @@
+path is required when kind is external-doc

plugin/skill-engine/tests/web-doc/fixtures/schema/invalid/invalid--external-doc-missing-path.json

@@ -0,0 +1,13 @@
+{
+  "schema_version": 1,
+  "sources": [
+    {
+      "id": "no-path-doc",
+      "kind": "external-doc",
+      "status": "intake",
+      "archived": false,
+      "lifecycle": { "state": "unknown", "last_checked": null, "last_checked_sha": null, "proposed_url": null },
+      "discovered_via": null
+    }
+  ]
+}

plugin/skill-engine/tests/web-doc/fixtures/schema/invalid/invalid--external-doc-with-url.expect

@@ -0,0 +1 @@
+url 'https://docs.example.com/' set on kind 'external-doc'

plugin/skill-engine/tests/web-doc/fixtures/schema/invalid/invalid--external-doc-with-url.json

@@ -0,0 +1,15 @@
+{
+  "schema_version": 1,
+  "sources": [
+    {
+      "id": "url-on-external-doc",
+      "kind": "external-doc",
+      "url": "https://docs.example.com/",
+      "path": "external-fixtures",
+      "status": "intake",
+      "archived": false,
+      "lifecycle": { "state": "unknown", "last_checked": null, "last_checked_sha": null, "proposed_url": null },
+      "discovered_via": null
+    }
+  ]
+}

plugin/skill-engine/tests/web-doc/fixtures/schema/valid/bootstrap-stamp-web-doc.json

@@ -0,0 +1,15 @@
+{
+  "schema_version": 1,
+  "sources": [
+    {
+      "id": "docs-example-com",
+      "kind": "web-doc",
+      "url": "https://docs.example.com/",
+      "crawl_mode": "sitemap",
+      "status": "intake",
+      "archived": false,
+      "lifecycle": { "state": "unknown", "last_checked": null, "last_checked_sha": null, "proposed_url": null },
+      "discovered_via": null
+    }
+  ]
+}