Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 31 additions & 40 deletions CIP-0165/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ License: CC-BY-4.0

## Abstract

This proposal defines the Simple Canonical Ledger State (SCLS), a stable, versioned, and verifiable file format for representing the Cardano ledger state. It specifies a segmented binary container with deterministic CBOR encodings, per-chunk commitments, and a manifest that enables identical snapshots across implementations, supports external tools (e.g., Mithril), and future-proofs distribution and verification of state.
This proposal defines the Standard Canonical Ledger State (SCLS), a stable, versioned, and verifiable file format for representing the Cardano ledger state. It specifies a segmented binary container with deterministic CBOR encodings, per-chunk commitments, and a manifest that enables identical snapshots across implementations, supports external tools (e.g., Mithril), and future-proofs distribution and verification of state.

This CIP specifies a canonical interchange format for the ledger state. It does not define, prescribe, or constrain the internal storage or representation of the ledger state within any node implementation. Internal formats remain an implementation detail; the canonical format applies only to export, interchange, and verification of the ledger state consistency.

Expand All @@ -44,7 +44,7 @@ The concrete use-case scenarios for this CIP are:

## Specification

The Simple Canonical Ledger State (SCLS) is a segmented file format for Cardano ledger states, designed to support streaming and verifiability. Records are sequential, each tagged by type and independently verifiable by hash. Multi-byte values use network byte order (big-endian).
The Standard Canonical Ledger State (SCLS) is a segmented file format for Cardano ledger states, designed to support streaming and verifiability. Records are sequential, each tagged by type and independently verifiable by hash. Multi-byte values use network byte order (big-endian).

### File Structure

Expand All @@ -55,16 +55,16 @@ Unsupported record types are skipped; core data remains accessible.

### Record Types

| Code | Name | Purpose |
| ---- | -------- | ------------------------------------------------------- |
| 0x00 | HDR | File header: magic, version, network, namespaces |
| 0x01 | MANIFEST | Global commitments, chunk table, summary |
| 0x10 | CHUNK | Ordered entries with per-chunk footer + hash |
| 0x11 | DELTA | Incremental updates (overlay; last-writer-wins) |
| 0x20 | BLOOM | Per-chunk Bloom filter |
| 0x21 | INDEX | Optional key→offset or value-hash indexes |
| 0x30 | DIR | Directory footer with offsets to metadata/index regions |
| 0x31 | META | Opaque metadata entries (e.g., signatures, notes) |
| Code | Name | Purpose |
| ---- | -------- | ----------------------------------------------------------------------------- |
| 0x00 | HDR | File header: magic, version, network, namespaces |
| 0x01 | MANIFEST | Global commitments, chunk table, summary |
| 0x10 | CHUNK | Ordered entries with per-chunk footer + hash |
| 0x11 | DELTA | Incremental updates overlay (reserved for future) |
| 0x20 | BLOOM | Per-chunk Bloom filter (reserved for future) |
| 0x21 | INDEX | Optional key→offset or value-hash indexes |
| 0x30 | DIR | Directory footer with offsets to metadata/index regions (reserved for future) |
| 0x31 | META | Opaque metadata entries (e.g., signatures, notes) |

Proposed file layout:

Expand Down Expand Up @@ -169,10 +169,7 @@ When calculating and verifying hashes, it's build over the uncompressed data.

#### DELTA Record

```text
TODO: exact contents of the DELTA record will be defined later, currently we describe
a high-level proposal.
```
Index record is reserved for the future use, in case if search in the file will be required by the downstream, most notably node that wants to generate the file incremetally.

**Purpose:** Delta records are used to build iterative updates, when base format is created and we want to store additional transactions in a fast way. Delta records are designed to be compatible with UTxO-HD, LSM-Tree or other storage types where it's possible to stream list of updates.

Expand All @@ -199,7 +196,7 @@ All updates are written in the following way:

#### BLOOM Record

TODO: define details or move to future work, (we propose to define exact format and properties, after the first milestone, when basic data will be implemented and tested. Then based on the benchmarks we could define exact properties we want to see)
Bloom record is reserved for the future use, in case if search in the file will be required by the downstream.

**Purpose:** additional information for allowing fast search and negative search.

Expand All @@ -212,9 +209,7 @@ TODO: define details or move to future work, (we propose to define exact format

#### INDEX Record

```text
TODO: define structure, (we propose to define exact format and properties, after the first milestone, when basic data will be implemented and tested. Then based on the benchmarks we could define exact properties we want to see)
```
Index record is reserved for the future use, in case if search in the file will be required by the downstream.

**Purpose:** allows fast search based on the value of the entries.

Expand All @@ -226,7 +221,7 @@ The general idea is that we may want to write a query to the raw data using comm

### Directory Record

TODO: define structure, (we propose to define exact format and properties, after the first milestone, when basic data will be implemented and tested. Then based on the benchmarks we could define exact properties we want to see)
Directory record is reserved for the future use, in case if index records or delta records will be implemented.

**Purpose**: If a file has index records then they will be stored after the records with actual data, and directory record allow a fast way to find them. Directory record is intended to be the last record of the file and has a fixed size footer.

Expand Down Expand Up @@ -262,22 +257,21 @@ In order to provide types of the values and be able to store and verify only par

Each logical table/type is a namespace identified by a canonical string (e.g., `"utxo"`, `"gov"`).

| Shortname | Content |
| ------------ | ------------------------------- |
| utxo/v0 | UTxOs |
| stake/v0 | Stake delegation |
| rewards/v0 | Reward accounts |
| params/v0 | Protocol parameters |
| pots/v0 | Accounting pots (reserves etc.) |
| spo/v0 | SPO state |
| drep/v0 | DRep state |
| gov/v0 | Governance action state |
| hdr/v0 | Header state (e.g. nonces) |
| Shortname | Content |
| -------------------- | ------------------------------- |
| blocks/v0 | Blocks created |
| gov/committee/v0 | Governance action state |
| gov/constitution/v0 | Constitution |
| gov/pparams/v0 | Protocol parameters |
| gov/proposals/v0 | Update proposals |
| pool_stake/v0 | Stake delegation |
| nonce/v0 | Nonces |
| pots/v0 | Accounting pots (reserves etc.) |
| snapshots/v0 | snapshots |
| utxo/v0 | UTXOs |

New namespaces may and will be introduced in the future. With new eras and features, new types of the data will be introduced and stored. In order to define what data is stored in the SCLS file, tools fill the `HDR` record and define namespaces. The order of the namespaces does not change the signatures and other integrity data.

For future compatibility support we added version tag to the name, but it may be a subject of discussion

#### Entries

Data is stored in the list of `Entries`, each entry consist of the namespace and its data:
Expand Down Expand Up @@ -357,7 +351,6 @@ By contrast, CBOR has a defined deterministic encoding (see [RFC 8949](https://d

Importantly, RFC 8949 also defines a mapping between CBOR and JSON. This allows us to specify a JSON view of the format so that downstream applications can consume the data using standard JSON tooling, while the canonical form remains CBOR.


##### Multi-file or single file?

We considered using single file because it's more friendly to the producer, because it's possible to ensure required atomicity and durability properties, together with footers-in records, it's possible to validate that the data was actually written and is correct. In case of failure it's possible to find out exactly the place where the failure happened.
Expand Down Expand Up @@ -391,13 +384,11 @@ We are proposing adding additional records types:

Both changes will not change the structure of the file.

**Do we want to support entries without natural keys? If so how can we do that?**
**Do we want use human readable names or tags in the structures?**

There are three options that we see:
In the various part of the CDDL definitions we have big structures (e.g. parameters update). Such structures has on-wire format that is fixed in the chain data. In these cases we always have a choice either to keep on-wire tags or use human readable tags defined in CDDLs.

- In chunks records make key optional, that will support such values. That will change the spec, as we must allow many of such values in the chunks records.
- Keep value in key, and zero value.
- Create a separate record type for values without keys.
Using on-wire tags is less readable in code is generated from CDDL it may be less readable comparing to the names. It would be nice to deletage this question to the downstreams that may use SCLS type.

## Path to Active

Expand Down
25 changes: 14 additions & 11 deletions CIP-0165/namespaces/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,17 @@ This is directory of the supported namespaces.

Each namespace defines a non-intersecting slices of the data.

| Shortname | Content | Key size |
| --------- | ------------------------------- | -------- |
| utxo | UTXOs | TxIn(transaction + offset) |
| stake | Stake delegation | TBD |
| rewards | Reward accounts | TBD |
| params | Protocol parameters | TBD |
| pots | Accounting pots (reserves etc.) | TBD |
| stake_pools | Stake Pools State | TBD |
| drep | DRep state | TBD |
| gov | Governance action state | 0 |
| hdr | Header state (e.g. nonces) | TBD |
| Shortname | Content | Key size | Key description |
| -------------------- | ------------------------------- | -------- | --------------- |
| blocks/v0 | Blocks created | 36 | keyhash of the stake pool |
| gov/committee/v0 | Governance action state | 8 | epoch |
| gov/constitution/v0 | Constitution | 8 | epoch |
| gov/pparams/v0 | Protocol parameters | 4 | current, previous, or future |
| gov/proposals/v0 | Update proposals | 34 | address of the proposal in transactions |
| pool_stake/v0 | Stake delegation | 28 | stake pool keyhash |
| nonce/v0 | Nonces | 1 | zero key |
| pots/v0 | Accounting pots (reserves etc.) | 8 | epoch |
| snapshots/v0 | snapshots | 32 | key type, stage, value type (see docs) |
| utxo/v0 | UTXOs | 34 | utxo address in the transaction |

Key specifications are described in cddl specification comments.
26 changes: 26 additions & 0 deletions CIP-0165/namespaces/blocks_v0.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
; This file was auto-generated from huddle. Please do not modify it directly!

; Values for the blocks.
;
; Key definition:
;
; ```
; meta:
; endian: be
;
; seq:
; - id: key
; type: blocks
;
; types:
; block:
; seq:
; - id: keyhash_stakepool
; doc: keyhash of the stake pool
; size: 28
; - id: epoch
; doc: epoch
; type: u8
; ```
record_entry = int
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lacking a comment about the key


50 changes: 50 additions & 0 deletions CIP-0165/namespaces/gov_committee_v0.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
; This file was auto-generated from huddle. Please do not modify it directly!

; The key for the namespace
;
; ```
; meta:
; endian: be
;
; seq:
; - id: key
; type: gov_committee
;
; types:
; gov_committee:
; seq:
; - id: epoch
; doc: epoch
; type: u8
; ```
record_entry = committee

; Storage of the committee members
committee = {* credential => committee_authorization}

credential = [0, addr_keyhash// 1, script_hash]

addr_keyhash = hash28

hash28 = bytes .size 28

; To compute a script hash, note that you must prepend
; a tag to the bytes of the script before hashing.
; The tag is determined by the language.
; The tags in the Conway era are:
; - "\x00" for multisig scripts
; - "\x01" for Plutus V1 scripts
; - "\x02" for Plutus V2 scripts
; - "\x03" for Plutus V3 scripts
script_hash = hash28

; 0 - hot committee member
; 1 - resignation
committee_authorization = [0, credential// 1, anchor/ nil]

;
; Signed url
anchor = [anchor_url : url, anchor_data_hash : bytes]

url = text .size (0 .. 128)

41 changes: 41 additions & 0 deletions CIP-0165/namespaces/gov_constitution_v0.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
; This file was auto-generated from huddle. Please do not modify it directly!

; Constitution record entry
;
; ```
; meta:
; endian: be
;
; seq:
; - id: key
; type: gov_constitution
;
; gov_constitution:
; seq:
; - id: epoch
; doc: Current epoch.
; type: u8
; ```
record_entry = constitution

; address of the constition
constitution = [anchor, script_hash/ nil]

;
; Signed url
anchor = [anchor_url : url, anchor_data_hash : bytes]

url = text .size (0 .. 128)

; To compute a script hash, note that you must prepend
; a tag to the bytes of the script before hashing.
; The tag is determined by the language.
; The tags in the Conway era are:
; - "\x00" for multisig scripts
; - "\x01" for Plutus V1 scripts
; - "\x02" for Plutus V2 scripts
; - "\x03" for Plutus V3 scripts
script_hash = hash28

hash28 = bytes .size 28

Loading