CEOS calvalportal glossary ingest

[do-me]

As discussed with @strobpr: We want to do a systematic ingest of the 436 terms of the legacy cal val https://calvalportal.ceos.org/web/guest/t-d_wiki

• ingest all terms as they are
• if there is an acronym in the term move it to synonym section
• default: add a new definition to a file/term unless one of the existing definitions is very similar or identical, in that case just add the source
• if the term does not exist, create a new markdown file for it
• tag the ingested terms properly with "to be discussed" "calval ingest" "to be defined" if the definition is empty
• tag the terms with two suitable CEOS groups from https://ceos.org/ourwork/. These will be responsible for discussing.

At the same time this will serve as a pilot for ingesting other glossaries.

[do-me]

I used Claude Code (max) for orchestration and a local model (qwen3-next-80b - qwen35b works just as well but switching off thinking mode is slightly more complex, preferred the instruction-tuned model) to review all terms individually and ensure a smooth ingest of the calval glossary. There were plenty of things to fix from spelling, grammar, acronyms as terms, duplicates (e.g. API), broken html and so on. Find the detailed summary below. Looking forward to your feedback!

---

CalVal Portal Ingest Report

Overview

Systematic ingest of the legacy CEOS CalVal glossary (calvalportal.ceos.org/t-d_wiki) into the EO Glossary Docusaurus project.

Metric	Count
CalVal source terms	432
Pre-ingest glossary	151 terms
Post-ingest glossary	509 terms

Pipeline

1. HTML scrape — Downloaded all 432 term pages (rate-limited, 5 req/s) and parsed the wiki-body content (Definition, Source, Comments, Category) using BeautifulSoup. Hyperlinks preserved as markdown.
2. Name matching — Matched CalVal terms against existing glossary by slug and title normalization. Found 37 name matches out of 432.
3. Definition deduplication — Used SequenceMatcher text similarity (threshold 0.45) to decide merge vs. add for the 37 matches:
   • 12 merged (sim ≥ 0.45): definitions similar enough → CalVal sources added to existing entry
   • 25 added (sim < 0.45): definitions meaningfully different → added as new numbered definition
4. New file creation — 395 new term files generated from unmatched CalVal terms.
5. CEOS group assignment — Deterministic mapping from CalVal's 4 categories to working groups.
6. Quality review — LLM-based review (qwen3-next-80b, 4 parallel requests) flagged 333 definitions, followed by automated fixes.

Actions taken

Action	Count
New markdown files created	395
Existing definitions merged (source added)	12
New definitions added to existing terms	25
Terms deleted (non-EO / problematic)	19
Redirect terms → converted to synonyms	8
Duplicate terms merged	4
Definitions rewritten (grammar + length)	227
Acronyms extracted → synonyms	18
Frontmatter descriptions rebuilt	161
Underscore titles fixed	7
Lowercase titles capitalized	2

Tagging

Every ingested term received 5 tags:

Tag type	Values
Term class	core (default for ingested terms)
Status	to be discussed (with definition) or to be defined (without)
Provenance	calval ingest
CEOS groups (×2)	Based on CalVal category

Category → Group mapping

CalVal Category	Terms	Groups
4. Data Stewardship words	371	WGISS, WGCV
1. Earth Observation system words	22	WGCV, WGClimate
3. Observation properties words	39	WGCV, WGClimate
2. Measurement and Metrology words	24	WGCV, WGISS

Group tag distribution (across 394 calval-tagged terms)

Group	Count
WGCV	394
WGISS	338
WGClimate	56

Synonym handling

37 synonyms added across the glossary:

• 18 from acronym extraction: e.g. "Climate Data Record (CDR)" → title "Climate Data Record", synonym "CDR"
• 8 from redirect merges: e.g. "Sensor Data" → synonym on Instrument Data; "CAL/VAL Data" → synonym on Calibration Data
• 7 from cleanup: e.g. "Dead Band (or neutral zone)" → synonym "Neutral Zone"; "FCDR (Fundamental Climate Data Record)" → synonym "FCDR"
• 4 from duplicate merges: e.g. "Data Formats" → synonym on Data Format; "Standing Order" → synonym on Subscription

Definition quality

The LLM review (qwen3-next-80b) flagged 333 of 455 calval definitions:

Flag	Count	Action
GRAMMAR	65	Rewritten to be substitutable (no "X is..." starts)
LONG	164	Condensed to 1–2 sentences, details moved to Notes
Problematic	104	19 deleted; 85 kept for community review

Definition merge examples

Term	Similarity	Decision
EO Space Data	0.98	Merge (nearly identical)
Near Real Time Data	0.89	Merge
Data	0.61	Merge
Uncertainty	0.59	Merge
Measurand	0.57	Merge
Process	0.55	Merge
Archive	0.54	Merge
User	0.53	Merge
Catalogue	0.49	Merge
Granule	0.48	Merge
Traceability	0.47	Merge
Long Term Preservation	0.46	Merge
Quantity	0.43	Add new definition
Preservation	0.35	Add new definition
Product	0.34	Add new definition
Representativeness	0.32	Add new definition
Verification	0.30	Add new definition
Metadata	0.28	Add new definition
Instrument Data	0.28	Add new definition
Auxiliary Data	0.25	Add new definition
Collection	0.23	Add new definition
Validation	0.21	Add new definition
Standard Uncertainty	0.20	Add new definition
Stability	0.19	Add new definition
Accuracy	0.18	Add new definition
Latency	0.18	Add new definition
Sensor	0.17	Add new definition
Calibration	0.14	Add new definition
Information	0.11	Add new definition
Property	0.08	Add new definition
Quality Indicator (QI)	0.07	Add new definition
Reproducibility	0.06	Add new definition
Source Data	0.04	Add new definition
Quality Indicator	0.04	Add new definition
Ancillary Data	0.02	Add new definition
Resolution	0.01	Add new definition
Temporal Resolution	0.01	Add new definition

Deleted terms (19)

Non-EO concepts, generic IT terms, or organizational jargon:

Benefit Hypothesis, Broad Network Access, Chunking, Dashboard, Data Use Monitoring and Administration, Establish, Measured Service, Multiple Selection and Download, On-demand Self-Service, Order Monitoring and Administration, Rapid Elasticity, Readiness Level, Resource Pooling, Restricted Data, Scalability, Statistics and Reporting, Subscription Monitoring and Administration, Tiers of Stewardship, Tools.

Problematic terms (104 flagged, 88 kept for review)

The LLM flagged 104 definitions as problematic — not proper concept definitions. These were triaged into two groups:

• Deleted: 19 terms (listed above) — clearly not EO glossary material.
• Kept for community review: 88 terms — domain-relevant but poorly defined. These remain in the glossary tagged to be discussed and need attention from the assigned working groups.

The 88 kept terms fall into five categories:

OAIS-specific definitions (10)

Definitions written in terms of OAIS (Open Archival Information System) internals rather than standing alone as general concepts:

• Access Collection
• Archive Recommendation Package
• Common Services
• Dissemination Information Package
• Packaging Information
• Preservation Description Information
• Repackaging
• Replication
• Submission Agreement
• Submission Information Package

System/implementation-specific (21)

Definitions that describe a particular system (e.g. PDGS, CLASS, ESOC) rather than the abstract concept:

• Bulk Dissemination
• Certification Services
• Circulation
• Directory
• Directory Service
• Event-based Order
• Guide
• Guide Service
• Intelligent Filtering
• Inventory System
• Logical Volume
• Operational Scenario
• Operations Center
• Operator
• Predicted Orbit
• Preliminary Orbit
• Repository
• Self-Service Access Model
• Subscription
• User Support
• Virtual Archive Information Package

Vague, circular, or incomplete (20)

Definitions that are self-referential, confuse related concepts, or were cut off mid-sentence:

• Application Programming Interface
• Bias Correction
• Cloud Optimized Formats
• Data Democracy
• Data Set Series
• Discovery
• Heterogeneous Data
• Knowledge Transfer File
• Maturity Matrix
• Outreach
• Point-to-Area Representativeness
• Primary Data
• Property
• Request
• Sensor
• Stability
• Standardisation
• Technology Stewards
• Temporal Resolution
• Uncertainty

Wrong scope (13)

Definitions that are organizational/administrative rather than EO-technical:

• Administrative Data
• Archive Organization
• Data Architecture
• Data File
• Data Model
• Data Operator
• EO Space Data Holders and Archive Owners
• Inter-Project Agreement
• Interface Control Document
• Metrics
• Operational Product
• User Category
• User Management

Process/workflow descriptions instead of definitions (24)

Definitions that describe what something does rather than what it is:

• Aggregation
• Analytics Optimized Data Store
• Artifact
• Bias
• Bulk Order
• Common Data Elements
• Consolidation
• Data Acquisition
• Data Maturity
• Homogenisation
• Payload Data Ground Segment
• Preservation Workflow
• Preserved Data Set Content
• Processing Levels
• Quantity
• Refreshment
• Research Product
• Resolution
• Transcription
• Transformation
• Uncertainty (measurement)
• Uncertainty of Measurement Method
• Value-added Product
• Version

What's left for the community

• 88 problematic terms kept but in need of discussion — definitions describe specific systems or workflows rather than defining EO concepts
• 10 terms tagged to be defined — placeholders without definitions
• All 394 calval-ingested terms are tagged to be discussed for review by the assigned working groups

[do-me]

@strobpr Ingestion completed and live (requires deep refresh of your browser page to reload all assets, else just open a private browser tab). Let me know what you think, happy to iterate on it.

We should probably add a note to https://calvalportal.ceos.org/web/guest/t-d_wiki? so everyone know that the main development will happen on GitHub in the future.

[strobpr]

Great Stuff, really impressive! Many thanks, @do-me!
We have to think about a way that makes approval by PoCs really simple and fast (kind of single click). Any ideas?

[do-me]

I think that's doable but definitely complicated. As an online interface it would require complex authentication and there might be race-conditions if two reviewers work at the same time.

Just a gut feeling but it might be easier to create something like a local text editor plugin or a simple branch one can work with locally where you click -> edits the tag in the opened file. Then one final "Commit" button would commit and push the PR.
I'll give it a bit more thought.