Skip to content

Prose/general-purpose aspect extractor for docs__* collections (GH #981 fix #2) #1111

Description

@hhildebrand-conductor

Context

GH #981 shipped --collection on nx index md in 5.9.3, letting Markdown land in knowledge__* so nx enrich aspects can process it. That issue is closed. But the help text for --collection explicitly notes the gap:

the aspect extractor for knowledge__ targets the scholarly-paper schema; it works well for paper-shaped content but will fabricate fields on general prose / design notes. A prose extractor is tracked as a follow-on (fix #2 from GH #981).

There is no upstream issue tracking this follow-on.

Problem

When a design doc, PRD, or general prose Markdown is indexed to knowledge__* and then nx enrich aspects runs, the scholarly-paper-v1 extractor fills in experimental_datasets, baselines, venue, etc. with hallucinated content. Real structure lands in extras instead. Confidence scores drop significantly (0.62 for a PRD vs. 0.92 for an actual paper).

The workaround today is to accept the misfitted aspects and rely on extras for the real structure, or to skip aspect extraction entirely for prose docs.

Proposed

Register a docs__* (or broader general-prose) aspect extractor config in the extractor-selection step, distinct from scholarly-paper-v1. Schema should capture what prose docs actually have: title, summary, key decisions/recommendations, stakeholders, status, links/references. This would let nx enrich aspects route docs__* and prose-shaped knowledge__* to an appropriate extractor instead of forcing the paper schema.

Reproducer

nx index md /path/to/design-doc.md --collection knowledge__mydocs --force
nx enrich aspects knowledge__mydocs --dry-run
# extractor: scholarly-paper-v1 -- misfits the content

Ref: canon-conductor-compose bead 4m0 notes section (2026-05-28), which documents two instances of the workaround.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions