|
| 1 | +--- |
| 2 | +name: lokf-librarian |
| 3 | +description: 'Scrape the host repository this skill sits inside and build/maintain the `.lokf/` knowledge bundle as a sidecar, compliant with the Linked Open Knowledge Format (LOKF) schema (a semantic profile of OKF). Use when: creating or updating concept files under .lokf/knowledge/; adding typed relationships (isPartOf/dependsOn/derivedFrom/about/references/...); choosing a LOKF class (Service/Metric/Dataset/Table/Policy/Playbook/GlossaryTerm/...); setting base_iri/context/id so frontmatter expands to JSON-LD/RDF; validating the bundle with JSON Schema and SHACL via the lokf toolkit; converting/serving the bundle as a graph; auditing .lokf/ for correctness, gaps, or bugs; preparing a LOKF change for human maintainer review; or running the scheduled LLM-librarian task that keeps .lokf/ accurate (Karpathy rule).' |
| 4 | +--- |
| 5 | + |
| 6 | +# LOKF Librarian |
| 7 | + |
| 8 | +Maintain `.lokf/` - the host repository's knowledge captured as a [**Linked Open Knowledge Format (LOKF)**](https://lokf.nolan-nichols.com/specification/) bundle. LOKF is a **semantic profile of OKF**: the same directory of Markdown + YAML frontmatter, but every field, type, and relationship is bound to a public vocabulary (schema.org / DCAT / PROV-O), so the bundle expands losslessly to JSON/JSON-LD and RDF and is queryable with SPARQL. This skill covers the full lifecycle: **scrape -> build/maintain -> audit -> hand off for review -> keep fresh on a schedule.** |
| 9 | + |
| 10 | +**Why this matters:** plain OKF gives knowledge *prose + structure*; typical hand-rolled ("vibe-coded") OKF setups add *tools* that only work in the repo that grew them; LOKF completes the stack - *prose + structure + **meaning** + tools* - because binding every field and relation to public vocabularies is precisely what lets **standard, schema-generated** tooling (JSON Schema, SHACL, SPARQL) validate and query the bundle instead of bespoke scripts. |
| 11 | + |
| 12 | +> Scope: this skill owns **only** `.lokf/`. A plain, tooling-free sibling `okf/` |
| 13 | +> would be owned by the separate **okf-librarian** skill. Every LOKF bundle is also a |
| 14 | +> valid OKF bundle - keep the two consistent, but edit each through its own skill. |
| 15 | +> If `.lokf/` doesn't exist yet, or the layout below is incomplete, run the |
| 16 | +> **lokf-scaffolding** skill first - it creates `knowledge/index.md` with the |
| 17 | +> semantic header (Rule 2), `knowledge/log.md`, the domain directories, plus |
| 18 | +> `pyproject.toml` and the `justfile` that `just lokf-validate` needs. This |
| 19 | +> skill assumes all of that is already in place. |
| 20 | +
|
| 21 | +## Layout |
| 22 | + |
| 23 | +``` |
| 24 | +.lokf/ |
| 25 | +|-- knowledge/ # the bundle - one Markdown file per concept |
| 26 | +| |-- index.md # bundle metadata (base_iri, context, versions) + TOC |
| 27 | +| |-- log.md # change history (reserved name) |
| 28 | +| |-- services/ datasets/ references/ |
| 29 | +| |-- playbooks/ glossary/ org/ |
| 30 | +|-- pyproject.toml # declares the `lokf` toolkit dependency |
| 31 | +|-- justfile # lokf-install / lokf-validate / lokf-convert / lokf-serve |
| 32 | +|-- scripts/ # (optional) knowledge-librarian.sh, the scheduled-agent wrapper (step 4) |
| 33 | +``` |
| 34 | + |
| 35 | +The LOKF **format** is defined once in LinkML (`lokf.yaml`); the JSON Schema, JSON-LD context, SHACL shapes, and OWL ontology are **generated** from it and MUST NOT be hand-edited. This repo's `.lokf/` is a *consumer* of that published schema - you author concepts, the toolkit validates and projects them. |
| 36 | + |
| 37 | +## Golden Rules (LOKF v0.1) |
| 38 | + |
| 39 | +1. **It's [OKF first](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md).** One concept per file, path = concept ID, `type` is the only strictly required field, permissive consumption. Everything plain OKF requires (enforced by the okf-librarian skill, when the repo maintains an `okf/` sibling) still holds here. |
| 40 | +2. **The bundle-root `index.md` carries the semantic header.** It declares the keys that lift the whole bundle into RDF (values shown are illustrative - the real ones are minted at scaffolding time): |
| 41 | + ```yaml |
| 42 | + lokf_version: "0.1" |
| 43 | + okf_version: "0.1" |
| 44 | + base_iri: https://acme.example/knowledge/ |
| 45 | + context: https://w3id.org/lokf/context.jsonld |
| 46 | + title: Acme Platform Knowledge Bundle |
| 47 | + description: ... |
| 48 | + license: https://creativecommons.org/licenses/by/4.0/ |
| 49 | + publisher: { type: Organization, id: https://acme.example/knowledge/org/platform-team, name: Acme Platform Team } |
| 50 | + ``` |
| 51 | + `base_iri` + concept ID mints each concept's IRI (`@id`); `context` maps frontmatter keys to IRIs. Do not remove these or the bundle degrades to plain OKF. |
| 52 | +3. **Use a class from the LOKF type vocabulary** (consumers tolerate unknowns as `lokf:Concept`): `Dataset`, `Table`, `Metric`, `Service`, `Playbook`, `Policy`, `GlossaryTerm`, `Reference`, `Document`, `Person`, `Organization`. Type-specific fields: `Table/Dataset` -> `fields`, `distribution`; `Metric` -> `unit`, `formula`, `measures`; `Service` -> `endpoint`, `http_method`, `documentation`; `GlossaryTerm` -> `definition`, `abbreviation`. |
| 53 | +4. **Prefer typed relationships over bare links** - this is LOKF's core upgrade. Each maps to a fixed RDF predicate; values are Concept IRIs (or IDs resolved against `base_iri`), all optional and multivalued: |
| 54 | + |
| 55 | + | field | predicate | meaning | |
| 56 | + |-------|-----------|---------| |
| 57 | + | `isPartOf` | `dcterms:isPartOf` | this is part of the target | |
| 58 | + | `hasPart` | `schema:hasPart` | the target is part of this | |
| 59 | + | `references` | `dcterms:references` | this refers to the target | |
| 60 | + | `dependsOn` | `dcterms:requires` | this depends on the target | |
| 61 | + | `derivedFrom` | `prov:wasDerivedFrom` | provenance | |
| 62 | + | `about` | `schema:about` | subject matter | |
| 63 | + | `sameAs` | `schema:sameAs` | same entity | |
| 64 | + | `relatedTo` | `dcterms:relation` | generic association | |
| 65 | + | `definedBy` | `rdfs:isDefinedBy` | formally defined by | |
| 66 | + | `source` | `dcterms:source` | sourced from the target | |
| 67 | + |
| 68 | + For predicates outside this set, use the generic `relations` list of reified objects (`predicate` from the `RelationType` vocab, e.g. `joinsWith`, plus `target`). Human-facing Markdown links in the body remain valid and encouraged alongside the typed fields. |
| 69 | +5. **Core fields map to ontology terms:** `title`->`schema:name`, `description`->`schema:description`, `resource`->`schema:url`, `tags`->`schema:keywords`, `timestamp`->`schema:dateModified`, plus optional `id`, `created`, `version`, `license`, `author`, `citations`. Two JSON-LD aliases let plain OKF frontmatter behave as Linked Data: `type` -> `@type` (rdf:type, the concept's class) and `id` -> `@id` (the subject IRI). |
| 70 | +6. **Stay permissive.** Missing optional fields, unknown `type`, unknown keys, and broken cross-links MUST NOT cause rejection. |
| 71 | + |
| 72 | +## 1. Scrape & build |
| 73 | + |
| 74 | +Derive concepts from the host repository (or, as an edge case, any directory tree) - never invent facts. **The bundle itself is the scrape map**: every concept records where it came from (`resource`, `derivedFrom`, `source`), and the map of knowledge sources is itself a reviewed concept. Steady-state runs are deterministic re-verification against that recorded provenance, not fresh discovery. |
| 75 | + |
| 76 | +### Bootstrap discovery - first run, or whenever the bundle has no real concepts |
| 77 | + |
| 78 | +Sweep the repository with generic heuristics and map what you find to LOKF classes: |
| 79 | + |
| 80 | +| Look at | Typical finds | Class | |
| 81 | +|---------|---------------|-------| |
| 82 | +| manifests (`package.json`, `pyproject.toml`, `go.mod`, ...), entry points, `Dockerfile`/compose files, CI config | APIs, CLIs, UIs, workers, databases | `Service` | |
| 83 | +| data and schema files (CSV/YAML/JSON/SQL), fixtures, migrations | datasets, tables | `Dataset` / `Table` (use `fields`, `distribution`) | |
| 84 | +| external standards, specs, and ontologies the code or data encodes | upstream authorities | `Reference` (wire `derivedFrom` from the encoding `Dataset`) | |
| 85 | +| README and docs install/run/operate guides | how-tos | `Playbook` | |
| 86 | +| domain terms recurring across code, data, and docs | vocabulary | `GlossaryTerm` | |
| 87 | +| ownership files (`CODEOWNERS`, manifest authors), publishers named inside data files | owners, publishers | `Organization` / `Person` - add only when another concept links to them (e.g. `Reference` -> `Organization` via `source`) | |
| 88 | + |
| 89 | +Record the resulting map as a concept: **`playbooks/knowledge-sources.md`** (`type: Playbook`), listing each knowledge source (repo path or external URL), the class(es) it yields, and how to re-check it. Discovery output thereby lives in the bundle, versioned and human-reviewed like every other concept - not in this skill. |
| 90 | + |
| 91 | +### Steady-state refresh - every later run |
| 92 | + |
| 93 | +1. **Re-verify provenance.** For each existing concept, follow its `resource`/`derivedFrom`/`source` back to the origin: does it still exist, are the facts still true, do relations still point the right way? Fix drift - including **deleting** concepts whose source no longer exists (remove their index bullets and log the removal). |
| 94 | +2. **Re-walk `playbooks/knowledge-sources.md`.** Sources listed there may have grown new assets since the last run. |
| 95 | +3. **Sweep for orphans.** Repository files or directories that no concept and no source-map entry accounts for are gap candidates: add a concept, extend the source map, or consciously leave them out. |
| 96 | +4. **Update the source map** whenever the repository's knowledge geography changes - it must stay as accurate as the concepts it feeds. |
| 97 | + |
| 98 | +Then add the semantic layer: pick the right class, set `id`, and wire typed relationships instead of guessing. Give every concept derived from the repository a `resource` (and `derivedFrom`/`source` where provenance is external) so the next refresh can re-verify it. The example below is **fictional** - an imaginary "Acme Platform" repo, not a concept of any real project; never copy its values, mint IRIs from the bundle's real `base_iri`: |
| 99 | + |
| 100 | +```markdown |
| 101 | +--- |
| 102 | +type: Service |
| 103 | +id: https://acme.example/knowledge/services/orders-api |
| 104 | +title: Orders API |
| 105 | +description: REST API serving order data to the CLI and web UI. |
| 106 | +endpoint: https://api.acme.example/orders |
| 107 | +resource: https://github.com/acme/platform/tree/main/services/orders |
| 108 | +dependsOn: |
| 109 | + - https://acme.example/knowledge/datasets/orders-db |
| 110 | +--- |
| 111 | +
|
| 112 | +# Overview |
| 113 | +
|
| 114 | +The **Orders API** generates its endpoints from `services/orders/openapi.yaml` and serves the order data consumed by the CLI and web UI... |
| 115 | +``` |
| 116 | + |
| 117 | +On every concept you create or materially change, set/refresh its `timestamp` (`schema:dateModified`) - but never bump timestamps on untouched concepts, or the diff fills with churn. Update the nearest `index.md` (bullet + `description`) and prepend a dated entry to `log.md` (newest first, ISO `YYYY-MM-DD`). `log.md` records **knowledge changes only** - concepts added/changed/removed, or the source map updated. If a run changes nothing in the bundle, write no log entry; never log administrative events ("librarian ran, no changes detected") - they do not represent a knowledge change. |
| 118 | + |
| 119 | +## 2. Audit (correctness, gaps, bugs) |
| 120 | + |
| 121 | +Use the toolkit - it gives you two independent, generated validators. From |
| 122 | +`.lokf/`: |
| 123 | + |
| 124 | +```bash |
| 125 | +just lokf-install # uv sync (first time) |
| 126 | +just lokf-validate # JSON Schema on frontmatter + assembled bundle |
| 127 | +just lokf-convert # project to Turtle/RDF; eyeball the triples |
| 128 | +just lokf-serve # SPARQL endpoint + live graph explorer (optional) |
| 129 | +``` |
| 130 | + |
| 131 | +`lokf validate` catches frontmatter/bundle-shape errors; the generated SHACL shapes catch cardinality/datatype/range violations on the projected graph. Beyond mechanical validity, audit for: |
| 132 | + |
| 133 | +- **Correctness** - class matches the asset; typed relations point the right way (`isPartOf` vs `hasPart`, `dependsOn` vs `derivedFrom`); `id`/`base_iri` mint the expected IRIs; `endpoint`/`resource` still resolve. |
| 134 | +- **Gaps** - new code/data files with no concept; untyped body links that should be typed relations; missing `id` on concepts other bundles link to; classes left as generic `lokf:Concept` that have a proper vocabulary type. |
| 135 | +- **Bugs** - malformed YAML, invalid enum/datatype (fails JSON Schema or SHACL), relation targets that resolve to nothing *intended*, missing `base_iri`/`context` in the root `index.md`, drift from the corresponding `okf/` concept (when the repo maintains a plain `okf/` sibling). |
| 136 | + |
| 137 | +Report findings as a checklist; fix mechanical issues directly and re-run `just lokf-validate`. |
| 138 | + |
| 139 | +## 3. Hand off for human maintainer review |
| 140 | + |
| 141 | +Open a PR scoped to `.lokf/` with a summary, the `lokf validate` (and, when relevant, SHACL/convert) output, and citations for every claim whose authority lives outside the repository - the standards, ontologies, and upstream systems the bundle's `Reference` concepts point at. A human maintainer verifies against the canonical source and approves before merge. Once `.github/workflows/knowledge-validate.yaml` exists (see section 4), it runs `uv run lokf validate knowledge` on every `.lokf/**` PR as the automated gate; until then, paste the local `just lokf-validate` output into the PR. |
| 142 | + |
| 143 | +## 4. Scheduled librarian task (Karpathy rule) |
| 144 | + |
| 145 | +Keep the graph continuously accurate rather than rewriting it in bursts. Three |
| 146 | +pieces automate this rebuild loop - the **lokf-scaffolding** skill's Step 5 |
| 147 | +scaffolds them; this section is their operating manual: |
| 148 | + |
| 149 | +**`.github/workflows/knowledge-librarian.yaml`** - the rebuild workflow. Runs weekly (Mondays 05:00 UTC) and on demand (`workflow_dispatch`). Each run: checks out full history -> sets up `uv` -> installs the `lokf` sidecar -> runs the agent -> validates -> diffs `.lokf/knowledge/` (only the bundle, so tool artifacts never trigger a PR) -> if anything changed, commits to a fresh `knowledge-librarian/<date>-<run_id>` branch and opens a review PR via `github-script`. Guardrails: it holds only `contents: write` + `pull-requests: write`, never pushes to the default branch, never auto-merges, and opens no PR when nothing changed. A no-change run must leave the bundle byte-for-byte untouched, including `log.md` (section 1's log policy). |
| 150 | + |
| 151 | +**`.lokf/scripts/knowledge-librarian.sh`** - the agent wrapper the workflow invokes. Point the `KNOWLEDGE_LIBRARIAN_CMD` repository variable at `bash .lokf/scripts/knowledge-librarian.sh` and set `AGENT_CLI` to your non-interactive agent command. The script selects the lokf-librarian skill, builds a prompt telling the agent to follow it and re-scrape the repo, then calls `AGENT_CLI`. Its contract: it edits **only** files under `.lokf/knowledge/` and performs no git/PR operations - the workflow owns branch/commit/PR. If `KNOWLEDGE_LIBRARIAN_CMD` is unset, the workflow's agent step is a safe no-op, so the workflow is harmless until you wire the agent up. |
| 152 | + |
| 153 | +**`.github/workflows/knowledge-validate.yaml`** - the gate. Its `validate` job runs `uv run lokf validate knowledge` on the librarian PR (and on every `.lokf/**` PR and its own weekly schedule); keep it green. The bar: the projected RDF graph should always describe the repository as it is *today*. |
0 commit comments