Skip to content

Commit 8d5958c

Browse files
SulsticeSulstice
authored
Merge pull request #15 from sensein/claude/schema-registry-meta-model-2upt56
docs: condense README — tighter prose, collapsible response shapes
2 parents 4fe6d01 + 037a968 commit 8d5958c

1 file changed

Lines changed: 58 additions & 237 deletions

File tree

‎README.md‎

Lines changed: 58 additions & 237 deletions
Original file line numberDiff line numberDiff line change
@@ -6,299 +6,120 @@
66

77
---
88

9-
## What is this?
9+
**NeuroGhost** is a public catalog of neuroscience vocabularies. Labs publish their [LinkML](https://linkml.io/) schema; the registry compares it to every other schema and surfaces which terms mean the same thing across projects.
1010

11-
Every neuroscience lab describes its data a little differently. One group calls
12-
someone an *Investigator*, another calls them a *Person*, another calls them a
13-
*Researcher*. Same idea, three names. Multiply that across every concept —
14-
brains, experiments, devices, subjects, publications — and it becomes very hard
15-
for different groups to share or combine data.
16-
17-
**NeuroGhost is a public catalog of these vocabularies.** Any lab or project
18-
can publish their schema (their definitions), and the registry automatically
19-
compares it to every other schema already in the catalog, so you can see:
20-
21-
- What terms already exist for a concept you care about
22-
- Which terms across different schemas actually mean the same thing
23-
- Which are unique to a particular project
24-
- How the definitions have evolved over time
25-
26-
Think of it as a **Rosetta Stone for neuroscience data models.**
27-
28-
---
29-
30-
## Key concepts (short version)
31-
32-
**Schema** — one group's vocabulary. A list of the things they care about
33-
(*classes*), the things they measure (*properties*), and any constraints
34-
(*rules*). Schemas are written in a simple text format called
35-
[LinkML](https://linkml.io/), one file per project.
36-
37-
**Class** — a type of thing. `Person`, `BrainRegion`, `Recording`, `Publication`.
38-
39-
**Property** — something a class has or does. A `Person` has a `name` and an `orcid`.
40-
41-
**Rule** — a constraint on a property. `age` must be a positive integer.
42-
`sampling_rate` must be measured in Hz.
43-
44-
**Alignment** — the registry's automatic comparison between two classes from
45-
different schemas, expressed as a number between 0 and 1:
46-
47-
- **0.0** = "these are definitely the same thing"
48-
- **0.5** = "these are related but not identical"
49-
- **1.0** = "these have nothing to do with each other"
50-
51-
**Registry version** — every time a schema is added or updated, the whole
52-
registry gets a new version number (like `1.2.0`). Old versions never disappear
53-
— you can always go back and see what the registry looked like at any point in
54-
time.
55-
56-
---
57-
58-
## How the distance function works (in plain English)
59-
60-
When the registry compares two classes, it looks at three things and combines
61-
them into a single distance score:
62-
63-
1. **IRI match** *(weight: 60%)* — Do the two classes explicitly point to the
64-
same universal identifier? For example, both saying "this is the same as
65-
schema.org's Person" is a perfect match.
66-
67-
2. **Name similarity** *(weight: 15%)* — Do the class names mean similar things?
68-
Uses AI embeddings (the same math that powers modern search) to compare, so
69-
`Investigator` and `Researcher` register as similar even though the letters
70-
are different.
71-
72-
3. **Definition similarity** *(weight: 25%)* — Do the plain-English descriptions
73-
describe similar things? Same embedding approach.
74-
75-
You can play with these weights live on the **Concepts** page of the website —
76-
slide them around and watch alignments recompute in real time.
11+
**Distance score** — 0.0 = identical, 1.0 = unrelated. Computed from IRI match (60%), name embeddings (15%), and definition embeddings (25%). Adjustable live on the Concepts page.
7712

7813
---
7914

80-
## Using the website
81-
82-
Go to **[sensein.group/NeuroGhost](https://sensein.group/NeuroGhost/)**.
83-
You'll see seven tabs:
15+
## Website
8416

85-
- **Concepts** — classes grouped by meaning. Aligned classes from different
86-
schemas collapse into a single card.
87-
- **Diff** — pick any two schemas, see which classes overlap and which are unique
88-
to each.
89-
- **Graph Schema** — how the underlying database is structured (for the curious).
90-
- **Transform** — convert a CSV from one schema's format to another.
91-
- **Query** — structured query builder + Claude-assisted natural language queries.
92-
- **Provenance** — a full timeline of every change to the registry, with per-class
93-
version diffs.
94-
- **Register** — submit a new schema.
95-
96-
Every view has download buttons. You can grab a single class, a whole schema,
97-
the entire registry, or a CSV diff between two schemas.
17+
**[sensein.group/NeuroGhost](https://sensein.group/NeuroGhost/)** — seven tabs: **Concepts**, **Diff**, **Graph Schema**, **Transform**, **Query**, **Provenance**, **Register**. Every view has download buttons.
9818

9919
---
10020

101-
## API Reference
102-
103-
Static JSON served by GitHub Pages — no auth, no rate limits, CORS open.
21+
## API
10422

105-
| Method | URL | Description | Status |
106-
|--------|-----|-------------|--------|
107-
| `GET` | [`/data/registry.json`](https://sensein.group/NeuroGhost/data/registry.json) | Full registry: all schemas, classes, properties, alignments | ✅ Live |
108-
| `GET` | [`/data/versions/{version}.json`](https://sensein.group/NeuroGhost/data/versions/1.7.0.json) | Frozen snapshot at a specific registry version | ✅ Live |
109-
| `GET` | [`/data/provenance.json`](https://sensein.group/NeuroGhost/data/provenance.json) | Changelog of every schema ingestion | ✅ Live |
110-
| `GET` | `/api/transform?from={schema}&to={schema}` | Field-to-field mapping between two schemas | 🔜 Planned |
111-
| `POST` | `/api/transform` | Transform a dataset from one schema format to another | 🔜 Planned |
23+
Static JSON via GitHub Pages — no auth, no rate limits, CORS open.
11224

113-
---
25+
| Method | URL | Status |
26+
|--------|-----|--------|
27+
| `GET` | [`/data/registry.json`](https://sensein.group/NeuroGhost/data/registry.json) | ✅ Live |
28+
| `GET` | [`/data/versions/{version}.json`](https://sensein.group/NeuroGhost/data/versions/1.7.0.json) | ✅ Live |
29+
| `GET` | [`/data/provenance.json`](https://sensein.group/NeuroGhost/data/provenance.json) | ✅ Live |
30+
| `GET` | `/api/transform?from={schema}&to={schema}` | 🔜 Planned |
31+
| `POST` | `/api/transform` | 🔜 Planned |
11432

115-
### `GET /data/registry.json`
33+
`distance`: **0.0** = identical · **1.0** = unrelated.
11634

117-
```bash
118-
curl https://sensein.group/NeuroGhost/data/registry.json
119-
```
35+
<details>
36+
<summary>Response shapes</summary>
12037

38+
**`GET /data/registry.json`**
12139
```json
12240
{
12341
"registry_version": "1.7.0",
12442
"generated_at": "2026-07-23T12:40:24Z",
125-
"sources": [
126-
{ "label": "bbqs", "version": "1.0.0", "class_count": 29 }
127-
],
128-
"classes": [
129-
{
130-
"hash_id": "sha256:abc123...",
131-
"iri": "https://registry.sensein.io/obj/Subject",
132-
"name": "Subject",
133-
"definition": "A research participant.",
134-
"source": "bbqs",
135-
"properties": [
136-
{ "hash_id": "sha256:def456...", "name": "age", "value_range": "xsd:integer", "multivalued": false, "required": false }
137-
],
138-
"alignments": [
139-
{ "target_name": "Participant", "target_source": "bids", "distance": 0.12, "method": "composite" }
140-
]
141-
}
142-
]
43+
"sources": [{ "label": "bbqs", "version": "1.0.0", "class_count": 29 }],
44+
"classes": [{
45+
"hash_id": "sha256:abc123...",
46+
"iri": "https://registry.sensein.io/obj/Subject",
47+
"name": "Subject",
48+
"definition": "A research participant.",
49+
"sources": ["bbqs"],
50+
"properties": [{ "hash_id": "sha256:def456...", "name": "age", "value_range": "xsd:integer" }],
51+
"alignments": [{ "target_name": "Participant", "distance": 0.12, "method": "composite" }]
52+
}]
14353
}
14454
```
14555

146-
`distance`: **0.0** = identical · **1.0** = unrelated.
147-
148-
---
149-
150-
### `GET /data/versions/{version}.json`
151-
152-
Same shape as `registry.json`. Snapshots are permanent — they never change.
153-
154-
```bash
155-
curl https://sensein.group/NeuroGhost/data/versions/1.2.0.json
156-
```
157-
158-
---
159-
160-
### `GET /data/provenance.json`
161-
162-
```bash
163-
curl https://sensein.group/NeuroGhost/data/provenance.json
164-
```
165-
166-
---
167-
168-
### `GET /api/transform?from={schema}&to={schema}` *(planned)*
169-
170-
Returns the pre-computed field mapping between two schemas. Derived from the
171-
alignment graph; can be served as a static file with no extra infrastructure.
172-
173-
```bash
174-
curl "https://sensein.group/NeuroGhost/api/transform?from=bbqs&to=bids"
175-
```
176-
56+
**`GET /api/transform?from=bbqs&to=bids`** *(planned)*
17757
```json
17858
{
17959
"from": "bbqs", "to": "bids",
180-
"mappings": [
181-
{
182-
"from_class": "Subject", "to_class": "Participant", "distance": 0.12,
183-
"field_mappings": [
184-
{ "from_field": "subject_id", "to_field": "participant_id", "confidence": 0.85 },
185-
{ "from_field": "age", "to_field": "age", "confidence": 0.97 }
186-
]
187-
}
188-
]
60+
"mappings": [{
61+
"from_class": "Subject", "to_class": "Participant", "distance": 0.12,
62+
"field_mappings": [
63+
{ "from_field": "subject_id", "to_field": "participant_id", "confidence": 0.85 }
64+
]
65+
}]
18966
}
19067
```
19168

192-
---
193-
194-
### `POST /api/transform` *(planned)*
195-
196-
Send a record in one schema's format; receive it remapped to another.
197-
Requires a serverless compute layer (Cloudflare Worker or equivalent).
198-
69+
**`POST /api/transform`** *(planned — needs serverless layer)*
19970
```bash
20071
curl -X POST https://sensein.group/NeuroGhost/api/transform \
20172
-H "Content-Type: application/json" \
20273
-d '{ "from": "bbqs", "to": "bids", "data": { "subject_id": "sub-01", "age": 24 } }'
20374
```
204-
205-
```json
206-
{ "from": "bbqs", "to": "bids", "result": { "participant_id": "sub-01", "age": 24 }, "unmapped_fields": [], "warnings": [] }
207-
```
75+
</details>
20876

20977
---
21078

211-
## Adding your own schema
79+
## Adding a schema
21280

213-
The easiest way, no setup required:
81+
1. Write a LinkML `.yml` file (copy `schemas/bbqs.yml` as a template).
82+
2. Go to the [Register tab](https://sensein.group/NeuroGhost/), paste your YAML, click **Open GitHub Issue**.
83+
3. A GitHub Action validates, ingests, aligns, and archives it within minutes.
21484

215-
1. Write your schema as a LinkML `.yml` file (see the
216-
[LinkML tutorial](https://linkml.io/linkml/intro/tutorial01.html) or copy
217-
`schemas/bbqs.yml` from this repo as a template).
218-
2. Go to the [Register tab](https://sensein.group/NeuroGhost/) on the website.
219-
3. Paste your YAML, give it a name, click **Open GitHub Issue**.
220-
4. A GitHub Issue opens in a new tab, pre-filled. Click **Submit**.
221-
5. Within a couple of minutes, an automated workflow will:
222-
- Validate your schema
223-
- Add it to the registry graph
224-
- Compute alignments against every other schema
225-
- Bump the registry version
226-
- Archive a permanent snapshot
227-
- Comment on your issue with a link to your schema in the browser
228-
229-
That's it. No installation, no pull request, no reviewers required.
85+
No installation, no pull request, no reviewers required.
23086

23187
---
23288

233-
## Running it locally (optional)
234-
235-
Only needed if you want to develop the registry itself, or bulk-load schemas
236-
outside the web flow.
89+
## Running locally
23790

23891
```bash
23992
git clone https://github.com/sensein/NeuroGhost.git
24093
cd NeuroGhost
24194
pip install -r requirements.txt
24295
```
24396

244-
**Full rebuild** (seeds schema.org, fetches external schemas, ingests everything, aligns, exports):
245-
246-
```bash
247-
python neuro_ghost/pipeline.py --fresh
248-
```
249-
250-
**Local schemas only** (skips the external network fetch):
251-
25297
```bash
253-
python neuro_ghost/pipeline.py --fresh --skip-converters
98+
python neuro_ghost/pipeline.py --fresh # full rebuild
99+
python neuro_ghost/pipeline.py --fresh --skip-converters # local schemas only
100+
python neuro_ghost/pipeline.py --skip-converters --schemas schemas/bbqs.yml # one schema
254101
```
255102

256-
**One schema** (incremental, adds to an existing DB):
103+
Options: `--fresh` (wipe DB), `--skip-converters` (skip BIDS/NWB/DANDI/openMINDS/AIND fetch), `--schemas FILE`, `--bump major|minor|patch`, `--agent TEXT`.
257104

258-
```bash
259-
python neuro_ghost/pipeline.py --skip-converters --schemas schemas/bbqs.yml
260-
```
105+
Open `index.html` in a browser when done.
261106

262-
```
263-
Options:
264-
--fresh Delete the DB first (fixes version-mismatch errors)
265-
--skip-converters Skip fetching BIDS, NWB, DANDI, openMINDS, AIND
266-
--schemas FILE One or more specific .yml files to ingest
267-
--bump Version bump type: major / minor / patch [default: minor]
268-
--agent TEXT Name recorded in provenance [default: local]
269-
```
270-
271-
Then open `index.html` in a browser to see the local snapshot.
272-
273-
274-
## Under the hood
275-
276-
Nothing exotic:
107+
---
277108

278-
- **[LadybugDB](https://ladybugdb.com/)** — embedded graph database that stores
279-
every class, property, and relationship. Runs as a Python package, no server.
280-
- **[LinkML](https://linkml.io/)** — the human-friendly schema format everyone
281-
writes their vocabularies in.
282-
- **[sentence-transformers](https://sbert.net/)** — the local embedding model
283-
(`all-MiniLM-L6-v2`) that powers semantic distance.
284-
- **Static HTML + GitHub Pages** — the website is one file, no framework.
285-
- **GitHub Actions** — the automation that runs on every schema submission.
109+
## Stack
286110

287-
Every design decision favors "nothing to run, nothing to maintain." The
288-
registry lives in a single GitHub repo. The website lives on GitHub Pages. The
289-
database is rebuilt fresh in CI from the source `.yml` files every time
290-
something changes.
111+
- **[LadybugDB](https://ladybugdb.com/)** — embedded graph DB, no server
112+
- **[LinkML](https://linkml.io/)** — schema format
113+
- **[sentence-transformers](https://sbert.net/)** — `all-MiniLM-L6-v2` for semantic distance
114+
- **Static HTML + GitHub Pages** — one-file frontend, no framework
115+
- **GitHub Actions** — CI/CD on every schema submission
291116

292117
---
293118

294119
## Contributing
295120

296-
- **Register a schema:** use the [Register tab](https://sensein.group/NeuroGhost/) on the site.
297-
- **Report an issue or suggest a feature:** [open an issue](https://github.com/sensein/NeuroGhost/issues/new).
298-
- **Improve the tooling:** PRs welcome, especially around the distance function.
299-
300-
---
301-
302-
## License
121+
- Register a schema via the [Register tab](https://sensein.group/NeuroGhost/).
122+
- [Open an issue](https://github.com/sensein/NeuroGhost/issues/new) to report bugs or suggest features.
123+
- PRs welcome, especially around the distance function.
303124

304-
CC0-1.0 — public domain. Fork it, remix it, run your own registry.
125+
**License:** CC0-1.0 — public domain.

0 commit comments

Comments
 (0)