Skip to content

CONTRACT_CORE.md

Latest commit

 

History

History
105 lines (73 loc) · 3.01 KB

CONTRACT_CORE.md

File metadata and controls

105 lines (73 loc) · 3.01 KB

Lode Core Model — Contract

This document defines the core data model and invariants for Lode. It is authoritative for all implementations and adapters.

Lode defines persistence structure only. It does not define execution, planning, scheduling, or optimization.

Goals

  1. Immutable, self-describing snapshots.
  2. Explicit, persisted metadata.
  3. Linear history per dataset.
  4. Backend-agnostic layout semantics.

Non-goals

  • Query planning or execution.
  • Background compaction or mutation.
  • Backend-specific behavioral flags.

Core Entities

Dataset

Logical collection of snapshots. A dataset has a stable DatasetID.

Snapshot

Immutable point-in-time state of a dataset. Each snapshot has a stable SnapshotID and (optionally) a parent snapshot ID.

Manifest

The authoritative description of a snapshot and its data files.

Manifest Requirements

Manifests MUST be self-describing and persisted.

Minimum required fields:

  • schema name + version
  • dataset ID
  • snapshot ID
  • creation time
  • explicit metadata object (see below)
  • list of files with sizes and checksums (when configured)
  • parent snapshot ID (when applicable)
  • row/event count (total data units in snapshot)
  • min/max timestamp (when data units implement Timestamped; omit if not applicable) Optional fields:
  • codec name (omit when no codec is configured)
  • per-file statistics (when the codec reports them via StatisticalCodec; omit when not available)

Per-File Statistics

FileRef MAY contain per-file statistics reported by the codec.

  • Statistics are codec-agnostic: any codec may report them via the StatisticalCodec interface.
  • When a codec reports statistics, they MUST be persisted on the FileRef.
  • When a codec does not report statistics, the stats field MUST be omitted.
  • Statistics values MUST be JSON-serializable.
  • Statistics MUST NOT be inferred; they are reported by the codec from observed data.
  • Per-file statistics include: row count, and per-column min, max, null count, and distinct count.
  • Distinct count is optional; zero means not computed.

Checksum Rules

  • Checksum computation is opt-in and explicit.
  • When a checksum component is configured, manifests MUST record:
    • the checksum component name, and
    • a checksum value for each file written by the dataset.
  • When no checksum component is configured, checksum fields MUST be omitted.

Manifests are immutable once written.

Metadata Rules

  • Metadata MUST be explicit on every snapshot.
  • nil metadata is coalesced to empty ({}) at the API boundary.
  • Empty metadata ({}) is valid and MUST be persisted as an explicit object.
  • Metadata values MUST be JSON-serializable.
  • Manifests always contain a non-nil metadata map.

History & Immutability

  • Snapshot history is strictly linear (single head).
  • Snapshots are immutable after commit.
  • Data files are immutable after write.
  • Commits create new state; they do not mutate existing state.

Design Invariant

Lode stores facts, not interpretations.