sd-006: The Algolia crawler for developers.stellar.org strips code blocks, so the

[kalepail]

Finding

The Algolia crawler for developers.stellar.org strips code blocks, so the
index carries the prose around commands but never the commands. The flagship
case: /docs/tools/cli/install-cli indexes as a single 720-char prose record
whose install methods are bare tab labels — "Install with Homebrew (macOS,
Linux, WSL):" with nothing after the colon — because each tab's content is
a code block the crawler dropped. It is not just this page's record being
thin: no record anywhere in the index contains any actual install command
(brew/cargo/winget/install-script). The consequence is categorical — any
consumer of the docs search index cannot answer "how do I install the Stellar
CLI", the canonical first question of every new developer, with real
commands. In the 2026-07-06 QA round the answering agent quoted the indexed
prose honestly and was graded partial; the commands were unretrievable from
the surface it searched.

Evidence

Live probes on production, 2026-07-06, reproducible by querying the index for
the install-cli page and regex-checking the records:

• The page's one indexed content record (720 chars): "Install with script
  (macOS, Linux, WSL):\r\nInstall with Homebrew (macOS, Linux, WSL):\r\n
  Install with winget (Windows):\r\nInstall with cargo from source..." —
  regex checks on it: hasBrewCmd:false, hasCargoCmd:false, hasApt:false,
  hasAUR:false, hasDocker:false, hasSoroban:false.
• Index-wide: zero docs records contain brew install stellar-cli or
  cargo install --locked stellar-cli.
• Counterpoint proving the commands exist and are gateway-reachable
  elsewhere: raven search("install stellar CLI setup") ranks
  skills.openzeppelin-stellar.setup-stellar-contracts init #1 (score 271), whose
  soroban-stellar-development-setup section contains the verbatim
  curl -fsSL https://github.com/stellar/stellar-cli/raw/main/install.sh | sh.

wf_01b3347d-1b8.

Additional recorded evidence:

• eval/qa/results/2026-07-06T18-48-22-variantA.json (q-tool-cli-install, partial — verdict stands, mixed root cause)
• live verification 2026-07-06: index-record probe of /docs/tools/cli/install-cli plus regex sweep of the full index for install command strings
• upstream issue filed 2026-07-07: sd-006: The Algolia crawler for developers.stellar.org strips code blocks, so the #2569

Recommendation

Cheapest fix: configure the Docusaurus crawler to index code-block text —
add code/pre to the crawler's recordProps/content selectors — at least
on setup/install pages, where the code blocks ARE the content. Alternative if
crawler config is off-limits: duplicate the commands inline in prose on those
pages (one sentence per tab: "run brew install stellar-cli"), which also
helps human skimmers. Consumer-side workaround: skill-reading consumers can
recover — the bundled openzeppelin-stellar setup skill carries the verbatim
install-script command and this gateway already ranks it #1 for the install
query — but docs-index-only consumers have no workaround at all.

Source Record

This was found by the downstream Raven eval/improvements loop and recorded as sd-006 (stellar-docs, discovered 2026-07-06). Public record: https://github.com/kalepail/stellar-raven/blob/main/improvements/stellar-docs/sd-006-crawler-strips-code-blocks.md

[Labi-Joy]

Hi @kalepail, I am planning on taking a swing at this if nobody's on it yet.

I cloned the repo and the only Algolia config in there is the client-side search widget in docusaurus.config.ts (appId, apiKey, indexName). No crawler recordProps or content selectors live in the repo, so the crawler config looks dashboard-only from the outside. Given that, I'll go with Option 2 and inline the actual commands in the prose lines on /docs/tools/cli/install-cli using single backticks (brew install stellar-cli, the curl install-script, winget install --id Stellar.StellarCLI, cargo install --locked stellar-cli), so the crawler picks them up from inline code even while it drops the fenced blocks. The fenced blocks stay for humans and copyability.

Let me know if the crawler config is actually reachable via PR somewhere I missed, or if this is already being handled internally, thanks.

[kalepail]

I'll likely try and tackle internally. Thanks though!

[Labi-Joy]

No worries, thanks for your quick reply. Although if there're other docs work on the Raven backlog (sd-00X or otherwise) that would suit an outside contributor, I'd be happy to take one on.

[kalepail]

Verified fixed live on 2026-07-09 after crawler config v15. Raw Algolia results now include article pre code content. Exact queries for the Homebrew, winget, cargo, and install-script commands return the canonical install page at rank 1 with the command text. Broad install intent still depends on the existing promotion rule, but the reported code-block extraction defect is resolved. This proof is recorded in sd-006, so I’m closing the report as fixed.

[kalepail]

Final resolution verified 2026-07-14: public DocSearch on the primary crawler_Stellar Docs - Docusaurus index (rules enabled, docs-default-current facet) shows stellar cli install command, brew install stellar-cli, winget install Stellar.StellarCLI, and cargo install stellar-cli all rank https://developers.stellar.org/docs/tools/cli/install-cli #1 with real command text. Command text is also present in other indexed records (including soroban-settings and hello-world-frontend), confirming index-wide code-block ingestion. Source record: https://github.com/kalepail/stellar-raven/blob/a4954b6a38f66de5ce4b28fc33535464df0818fb/improvements/stellar-docs/sd-006-crawler-strips-code-blocks.md

The separate broad-intent promotion rule raven-promote-stellar-cli-install remains under Raven’s daily rule-health canary; that monitoring is not being retired with sd-006. sd-006 is retired to Raven’s resolved ledger.