Structsense skills

.gitignore

@@ -205,3 +205,4 @@ cython_debug/
 marimo/_static/
 marimo/_lsp/
 __marimo__/
+.DS_Store

README.md

@@ -10,15 +10,16 @@ Skills are kept **flat**: a skill never contains another skill's `SKILL.md` nest
 
 The list below is kept in alphabetical order by skill name.
 
-| Skill | Description |
-| --- | --- |
-| [`duct`](./skills/duct/SKILL.md) | Wrap any command with [con/duct](https://github.com/con/duct) to capture wall-clock time, CPU, and memory usage as structured logs, so agents and reviewers can inspect what a run actually consumed. |
-| [`kya`](./skills/kya/SKILL.md) | Govern and review agents with [veldt-kya](https://github.com/veldtlabs/veldt-kya) (Know Your Agents) — risk-score, consensus-judge, and drift-check an agent, emit compliance evidence, and write a governance verdict that the `labnb` loop can break on. |
-| [`labnb`](./skills/labnb/SKILL.md) | Create and maintain a concurrency-safe global lab notebook outside project roots, with startup summaries of related prior work, first-class idea capture and promotion, isolated experiment workspaces, focused companion skills, and append-only indexing across projects, investigations, and tasks. |
-| [`labnb-idea`](./skills/labnb-idea/SKILL.md) | Record a promising but not-yet-implemented experiment idea in the shared lab notebook index. |
-| [`labnb-promote`](./skills/labnb-promote/SKILL.md) | Promote a lab notebook idea into a concrete experiment with explicit budgets, source links, and provenance. |
-| [`labnb-resume`](./skills/labnb-resume/SKILL.md) | Summarize prior ideas and experiments for a project slug, then choose whether to resume, promote, branch, or start new work. |
-| [`labnb-run`](./skills/labnb-run/SKILL.md) | Create and run a concrete lab notebook experiment with isolated workspace, explicit budgets, and iterative logging. |
+| Skill                                                        | Description |
+|--------------------------------------------------------------| --- |
+| [`duct`](./skills/duct/SKILL.md)                             | Wrap any command with [con/duct](https://github.com/con/duct) to capture wall-clock time, CPU, and memory usage as structured logs, so agents and reviewers can inspect what a run actually consumed. |
+| [`kya`](./skills/kya/SKILL.md)                               | Govern and review agents with [veldt-kya](https://github.com/veldtlabs/veldt-kya) (Know Your Agents) — risk-score, consensus-judge, and drift-check an agent, emit compliance evidence, and write a governance verdict that the `labnb` loop can break on. |
+| [`labnb`](./skills/labnb/SKILL.md)                           | Create and maintain a concurrency-safe global lab notebook outside project roots, with startup summaries of related prior work, first-class idea capture and promotion, isolated experiment workspaces, focused companion skills, and append-only indexing across projects, investigations, and tasks. |
+| [`labnb-idea`](./skills/labnb-idea/SKILL.md)                 | Record a promising but not-yet-implemented experiment idea in the shared lab notebook index. |
+| [`labnb-promote`](./skills/labnb-promote/SKILL.md)           | Promote a lab notebook idea into a concrete experiment with explicit budgets, source links, and provenance. |
+| [`labnb-resume`](./skills/labnb-resume/SKILL.md)             | Summarize prior ideas and experiments for a project slug, then choose whether to resume, promote, branch, or start new work. |
+| [`labnb-run`](./skills/labnb-run/SKILL.md)                   | Create and run a concrete lab notebook experiment with isolated workspace, explicit budgets, and iterative logging. |
+| [`structsense-skills`](./skills/structsense-skills/SKILL.md) | StructSense Skills transforms unstructured text and PDFs into validated, ontology-grounded structured JSON using a model-agnostic extraction pipeline. |
 
 ## Installing Into An Agent
 

skills/structsense-skills/CHANGELOG.md

@@ -0,0 +1,163 @@
+# Changelog
+
+All notable changes to `structsense-skills`. Versions follow semantic versioning.
+The `version:` field in `SKILL.md` frontmatter is the source of truth; this file
+records what changed between versions so you can tell which features your local
+copy has.
+
+## [0.4.0] — 2026-06-05 (zero-hallucination policy)
+
+### Concept mapping policy (breaking)
+- **Mapping is mandatory and tool-only.** The cascade no longer silently
+  skips alignment. Default cascade: local hybrid (`http://localhost:8000`,
+  verified at `/docs`) → BioPortal → ask user for an alternate URL →
+  **hard-stop** with an actionable error. The skill no longer falls back
+  to OLS by default (OLS lacks gene coverage); pass `--allow-ols-fallback`
+  to opt in.
+- **`concept_mapping_provenance: "llm_knowledge"` is forbidden in canonical
+  output.** Any item carrying it is automatically demoted to `unmapped`,
+  with `alignment_method: "validation_failed"`. The item itself is
+  preserved (exhaustive extraction is not compromised); only the
+  fabricated mapping is dropped.
+
+### New: `scripts/iri_validation.py`
+- Per-ontology regex patterns for UBERON, CL, NCBITaxon, MONDO, DOID, HP,
+  CHEBI, DRON, BTO, OBI, MP, HGNC, NCBIGene, MGI, UniProt, PR, GO, EFO,
+  NIFSTD, CIDO, plus a permissive structural fallback for unknown
+  ontologies (`<NS>_<NUM>` OWL IRIs, identifiers.org, semanticweb.org).
+- Accepts **legitimate cross-ontology mappings** (e.g. CIDO results that
+  reuse HP IRIs as synonyms) — the declared `ontology` field does NOT
+  have to match the IRI prefix. We only reject `llm_knowledge` provenance
+  and structurally-malformed IRIs.
+- Adds `result["validation"]` with `passed` / `demoted` counts and
+  `demoted_by_reason` breakdown (`llm_knowledge_rejected`,
+  `malformed_iri`, `wrong_ontology_pattern`, `missing_ontology_id`).
+
+### Stats clarity
+- **`stats.totals` block at the top** of every result. Top-level
+  `total_items`, `total_entity_mentions`, `total_key_term_mentions`,
+  `total_resources`, `unique_entities`, `unique_key_terms`. The human
+  summary shows `TOTAL ITEMS: …` as the second line after `task_type`.
+- **`stats.task_type` auto-inferred** when missing — entities present →
+  `ner`; resources → `resource`; activity/items → `structured_extraction`.
+- **`alignment.by_method`** no longer shows `{missing: N}` for items that
+  have `concept_mapping_provenance` set. The normalizer fills
+  `alignment_method` from the provenance (`tool` → `direct_tool_call`,
+  `skipped` → `skipped`).
+
+### Connecting / runtime
+- **`connecting/claude-desktop.md`** added. Explains Claude Desktop's
+  **split execution model** (chat UI on your machine, code interpreter in
+  Anthropic's cloud sandbox), why your `localhost:8000` is unreachable
+  from a sandboxed code-interpreter call (`/home/claude/work/...` paths
+  are the giveaway), and how to bridge via an MCP server configured in
+  `claude_desktop_config.json`.
+
+### Stats fix verified against real numbers
+Smoke-tested against the user's actual run (2331 entity mentions across
+15 labels, 210 key terms, 922 tool-mapped + 1619 skipped). The new
+output:
+- `totals.total_items: 2541` (prominent at top)
+- `task_type: "ner"` (was `null`)
+- `alignment.by_method: {direct_tool_call: 922, skipped: 1409, missing: 210}` (was `{missing: 2541}`)
+
+## [0.3.1] — 2026-06-05 (later same day)
+
+### Concept-mapping (fixed)
+- **API schema bug fixed.** `scripts/local_hybrid_map.py` was POSTing
+  `{terms: [...]}` but the real `/map/batch` endpoint takes
+  `{text: [{text, context?}], max_results}`. Verified against a live
+  deployment. The client now supports either a flat list of strings or
+  shaped dicts with optional per-term `context`. Also handles the real
+  response shape (`results` is a dict keyed by input text, score is
+  `final_score`).
+- **`prompts/alignment-via-http.md`** added: a turnkey curl + jq pipeline
+  for hitting `/map/batch` directly. Use this when in Claude Code (Bash
+  available) but the Python client isn't loaded.
+- **`SKILL.md` rule #10 strengthened**: "you MUST probe `curl
+  ${MAPPER_URL}/docs` before declaring the mapper unavailable." Plus a
+  runtime-reachability table (Claude Code → yes; claude.ai web →
+  no, needs MCP bridge; ChatGPT cloud → no, needs Action).
+
+### New scripts ported from structsense
+- **`scripts/input_loader.py`** — PDF / CSV / TXT / MD ingestion. PDF chain:
+  GROBID → PyMuPDF (`fitz`) → pdfminer.six. CLI writes a sibling `<stem>.txt`
+  so subsequent NER stages have stable character offsets.
+- **`scripts/task_detection.py`** — auto-detect task type (ner / resource /
+  structured_extraction / relation_extraction / keyphrase_extraction / …)
+  from a free-text description. Heuristic regex first; LLM fallback via
+  `llm_client`.
+- **`scripts/model_context.py`** — model context-window registry (~50 model
+  families, longest-match wins) + `estimate_payload_tokens` +
+  `compute_downstream_chunk_size` for sizing alignment/judge chunks
+  programmatically.
+- **`group_by_entity.unify_ontology_across_entities`** — when the same
+  surface form gets different ontology IDs from different chunks/models
+  (parallel extraction is the usual cause), unify on the best one
+  (tool-mapped > llm_knowledge > unmapped). Runs automatically inside
+  `attach_grouped_views`. Mentions are preserved; only the ontology
+  fields are normalized.
+
+## [0.3.0] — 2026-06-05
+
+### Schema (breaking)
+- **`paper_title` / `doi` removed from per-entity items.** They now live ONCE at
+  the top level under `source_metadata: { paper_title, doi, source_path }`.
+  `paper_location` (section / page / paragraph) stays per-entity because it
+  varies per mention.
+- **`source_model` provenance** added to every entity item (HF model id, or
+  `llm_ner:<llm_model>` for the LLM extractor).
+- **`entities_grouped[]`** (and `key_terms_grouped[]`) added: per-entity index
+  collapsing raw mentions by canonical `(entity.lower(), label)`. Each group has
+  `source_models[]`, `source_model_counts{}`, `consensus_count`,
+  `judge_score_max/avg/min`, deduplicated `sentences[]` with their
+  `paper_locations[]`, and slim per-occurrence `mentions[]`.
+- **`stats` block** always embedded: totals, label histogram,
+  **by_source_model** histogram, alignment provenance, judge score buckets,
+  per-stage timings.
+
+### Pipeline
+- **Post-processing normalizer** (`scripts/normalize_result.py`) added. Runs
+  automatically before every save, and is also exposed as a CLI so legacy
+  result files can be normalized in place. It lifts per-entity
+  `paper_title`/`doi` into `source_metadata`, tags missing `source_model`,
+  attaches `entities_grouped`, and computes `stats`. **Idempotent.**
+- **Mapper cascade**: local hybrid (`http://localhost:8000`, verify via
+  `/docs`) → BioPortal → ask the user for an alternative URL → skip alignment.
+- **HuggingFace NER ensemble** (`scripts/ner_models.py`): d4data,
+  BC5CDR-chem, NCBI-disease, BioBERT-genetic, BENT-PubMedBERT family
+  (Gene/Chemical/Disease/Anatomical/Cell-Type/Cell-Line/Organism/Bioprocess),
+  Clinical-AI-Apollo, blaze999. Selectable via `--ner-profile`:
+  `biomedical_broad` / `cns_cells` / `pharmacology` / `genetic` / `clinical` /
+  `minimal` / `all`.
+- **Final filename** is `<input_stem>_final.json` by default.
+
+### NER prompts
+- Three variants: **general**, **neuroscience**, **CNS-cells**.
+- Each ships with explicit **EXHAUSTIVENESS** rules ("extract every occurrence,
+  do not deduplicate by surface form") and explicit **WRONG / RIGHT** example
+  blocks for the `source_metadata` shape.
+- **Mask-recall pass** (`prompts/mask-recall-pass.md`) and **mask-verify pass**
+  (`prompts/mask-verify-pass.md`) for boosting recall and detecting label
+  errors.
+
+### Connecting (new)
+- `connecting/claude-code.md` — install as a Claude Code skill (`~/.claude/skills/`
+  or `.claude/skills/`).
+- `connecting/claude-skills.md` — publish as a hosted Anthropic Skill.
+- `connecting/custom-gpt.md` — wire as an OpenAI Custom GPT (Instructions +
+  Knowledge files + Actions).
+- `connecting/mcp-server.md` — expose as an MCP server.
+
+## [0.2.0] — 2026-06-04
+
+### Added
+- Multi-stage pipeline pattern (extractor → alignment → judge → human feedback).
+- Three NER variants and resource / structured-extraction prompts.
+- Ontology mapping clients: BioPortal, OLS, local hybrid.
+- JSON repair, span validator, chunking helpers.
+- Reference pipeline driver (`scripts/pipeline.py`).
+
+## [0.1.0] — 2026-06-03
+
+- Initial skill scaffolding from the structsense pipeline patterns.