Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

## [v1.1.0] - 2026-01-12

### Added

- **"Deep Integrity"** specification: defined how to bind metadata hashes to physical file checksums using the [File Info Extension](https://github.com/stac-extensions/file).
- `Dependencies` field to the extension header.
- **Merkle Proofs**: introduced the `merkle-proof` link relation and the JSON schema for proof objects to enable efficient verification.

### Changed

- **Hash Computation** rules to strongly recommend including `assets` (specifically `file:checksum`)
in the `merkle:object_hash` to ensure data integrity.
- Item, Collection, and Catalog examples to demonstrate usage with the `File Info Extension` and `Merkle Proof` links.

## [v1.0.0] - 2024-11-05

- move extension to stacchain org
Expand All @@ -29,7 +43,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- first release

[Unreleased]: https://github.com/stacchain/merkle-tree-stac-extension/tree/v1.0.0...main
[Unreleased]: https://github.com/stacchain/merkle-tree-stac-extension/tree/v1.1.0...main
[v1.1.0]: https://github.com/stacchain/merkle-tree-stac-extension/tree/v1.0.0...v1.1.0
[v1.0.0]: https://github.com/stacchain/merkle-tree-stac-extension/tree/v1.0.0-beta.2...v1.0.0
[v1.0.0-beta.2]: https://github.com/stacchain/merkle-tree-stac-extension/tree/v1.0.0-beta.1...v1.0.0-beta.2
[v1.0.0-beta.1]: https://github.com/stacchain/merkle-tree-stac-extension/tree/v1.0.0-beta.1
196 changes: 163 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,28 @@
# Merkle Tree Extension Specification

- **Title:** Merkle Tree
- **Identifier:** <https://stacchain.github.io/merkle-tree/v1.0.0/schema.json>
- **Identifier:** <https://stacchain.github.io/merkle-tree/v1.1.0/schema.json>
- **Field Name Prefix:** `merkle`
- **Scope:** Item, Collection, Catalog
- **Extension [Maturity Classification](https://github.com/radiantearth/stac-spec/tree/master/extensions/README.md#extension-maturity):** Proposal
- **Dependencies:**
- [File Info Extension](https://github.com/stac-extensions/file) (Recommended for Deep Integrity)
- **Owner**: @jonhealy1

This extension specifies a way to ensure metadata integrity for STAC Items, Collections, and Catalogs by encoding them in a Merkle tree via
hashing. Each STAC object (Item, Collection, or Catalog) is hashed using a hash function (e.g., SHA-256), and this hash is stored in the
object's properties under the `merkle:object_hash` field. Details concerning the methods used for hashing are stored in a separate object
called `merkle:hash_method`. To produce the Merkle root identifier for a Collection or Catalog, the hashes from its child objects are taken
into account. This process ensures the integrity of all STAC objects within the hierarchy.
This extension specifies a way to ensure metadata integrity for STAC Items, Collections, and Catalogs by encoding them in a Merkle tree via hashing.
Each STAC object is hashed using a hash function (e.g., SHA-256),
and this hash is stored in the object's properties under the `merkle:object_hash` field.
Details concerning the methods used for hashing are stored in a separate object called `merkle:hash_method`.

To produce the Merkle root identifier for a Collection or Catalog, the hashes from its child objects are aggregated.
This structure allows for the generation of **Merkle Proofs**,
which enable efficient verification of any individual STAC object against the Collection Root without requiring access to the entire dataset.

To achieve **Deep Integrity** (verifying not just the metadata,
but the data files themselves), this extension utilizes fields defined
in the **File Info Extension**. By binding the `merkle:object_hash` to
the `file:checksum` of the assets, the cryptographic proof extends to
the binary files themselves.

- **Examples:**
- [Item example](examples/item.json): Shows the basic usage of the extension in a STAC Item.
Expand All @@ -20,14 +31,32 @@ into account. This process ensures the integrity of all STAC objects within the
- [JSON Schema](json-schema/schema.json)
- [Changelog](./CHANGELOG.md)

## Table of Contents

- [Fields](#fields)
- [Additional Field Information](#additional-field-information)
- [Hash Method Object](#hash-method-object)
- [Deep Integrity & File Hashing](#deep-integrity--file-hashing)
- [Merkle Proofs](#merkle-proofs)
- [Linking to a Proof](#linking-to-a-proof)
- [Proof Object Schema](#proof-object-schema)
- [Example Proof](#example-proof)
- [Computing Hashes and Merkle Roots](#computing-hashes-and-merkle-roots)
- [Computing `merkle:object_hash`](#computing-merkleobject_hash)
- [Computing `merkle:root`](#computing-merkleroot)
- [Computing the Merkle Proof](#computing-the-merkle-proof)
- [Examples](#examples)
- [Relation types](#relation-types)
- [Contributing](#contributing)

## Fields

The fields in the table below can be used in these parts of STAC documents:

- [x] Catalogs
- [x] Collections
- [x] Item Properties (incl. Summaries in Collections)
- [ ] Assets (for both Collections and Items, incl. Item Asset Definitions in Collections)
- [x] Assets (via File Info Extension recommendation)
- [ ] Links

| Field Name | Type | Description |
Expand Down Expand Up @@ -77,16 +106,91 @@ The `merkle:hash_method` object provides details about the hash computation meth
| `ordering` | string | **REQUIRED** (for Collections and Catalogs). Describes how the hashes are ordered when building the Merkle tree (e.g., "ascending by hash value"). |
| `description` | string | Optional. Additional details or notes about the hash computation method, such as serialization format or any special considerations. |

## Deep Integrity & File Hashing

To ensure the integrity of the underlying data files (e.g., Cloud Optimized GeoTIFFs), implementations are **STRONGLY RECOMMENDED** to use the
[File Info Extension](https://github.com/stac-extensions/file) in conjunction with this extension.

1. **Implement File Info Extension:** The Item MUST include the File Info Extension URI
(`https://stac-extensions.github.io/file/v2.1.0/schema.json`) in its `stac_extensions` list.
2. **Calculate File Hash:** Compute the checksum of the physical asset file.
- *Note:* The File Info Extension requires checksums to be **Multihash** compliant (hexadecimal string).
3. **Store in Asset:** Store this value in the Asset definition using the `file:checksum` field.
4. **Include in Merkle Hash:** Ensure the `assets` object (specifically the `file:checksum` fields) is included in the `merkle:object_hash`
calculation.

This binds the physical file to the metadata. If the file on S3 is swapped or corrupted, its checksum changes, invalidating the Asset metadata,
which invalidates the Item hash, which invalidates the Collection Root.

## Merkle Proofs

While `merkle:hash_method` allows a user to verify an individual object's hash,
verifying that an object belongs to a specific Collection Root would typically
require access to the entire collection (to reconstruct the full tree).
To enable efficient verification (O(log n)) without downloading the full
dataset, implementations SHOULD provide a **Merkle Proof**
(also known as an audit path or witness).

### Linking to a Proof
STAC objects SHOULD provide a link to their proof using the
`merkle-proof` relation. This link may point to a static file
(e.g., on S3) or a dynamic API endpoint.

```json
"links": [
{
"rel": "merkle-proof",
"href": "https://example.com/collections/c1/items/i1/proof.json",
"type": "application/json",
"title": "Merkle Inclusion Proof"
}
]
```

### Proof Object Schema

The resource at the `merkle-proof` link MUST return a JSON object with the following structure:

| Field Name | Type | Description |
| :--- | :--- | :--- |
| `target_hash` | string | **REQUIRED.** The `merkle:object_hash` of the Item being proven. |
| `root` | string | **REQUIRED.** The `merkle:root` of the Collection/Catalog this Item belongs to. |
| `path` | [Proof Node] | **REQUIRED.** An ordered list of sibling hashes required to reconstruct the root. |

### Example Proof

This JSON allows a client to take the Item's hash, combine it with the hashes in path, and confirm the result matches root.

```json
{
"target_hash": "3a7bd3e2360a8e7d9f5b1c2d4e6f7890abcdef1234567890abcdef1234567890",
"root": "abc123def4567890abcdef1234567890abcdef1234567890abcdef1234567890",
"path": [
{
"position": "right",
"hash": "8f434346648f6b96df89dda901c5176b10a6d83961dd3c1ac88b59b2dc327aa4"
},
{
"position": "left",
"hash": "2b26b46c6b3c9657f86494796342f7c00e62059c4033757f446574676528d223"
}
]
}
```

## Computing Hashes and Merkle Roots

### Computing `merkle:object_hash`

1. **Prepare Metadata:**
- Include all fields specified in `merkle:hash_method.fields`.
- If `fields` is `"*"` or `"all"`, include all fields of the object.
- **Recommendation:** Explicitly include `assets` to capture `file:checksum` values.
- **Warning:** It is recommended to **exclude** `links` from the hash, as dynamic URLs
(e.g., API endpoints, signed URLs) can change without altering the underlying data,
which would invalidate the Merkle Root.
- If `fields` is `"*"` or `"all"`, include all fields of the object (implementers should be cautious of the warning above).
2. **Serialize Metadata:**
- Use canonical JSON serialization with sorted keys and consistent formatting.
- Ensure consistent encoding (e.g., UTF-8).
- Use canonical JSON serialization (RFC 8785) with sorted keys to ensure reproducibility.
3. **Compute Hash:**
- Apply the specified cryptographic hash function (e.g., SHA-256) to the serialized metadata.

Expand All @@ -99,33 +203,51 @@ The `merkle:hash_method` object provides details about the hash computation meth
- Order the hashes according to the method specified in `merkle:hash_method.ordering`.
3. **Build Merkle Tree:**
- Pairwise hash the ordered hashes, proceeding up the tree until a single hash remains—the `merkle:root`.
4. **Include `merkle:hash_method`:**
- Specify the method used in the Collection's or Catalog's `merkle:hash_method` field, including any details necessary for users
to replicate the process.

### Computing the Merkle Proof

To generate a proof for a specific child object (Target) within a tree:

1. **Identify Target:** Locate the `merkle:object_hash` of the Target object in the sorted list of all children hashes.
2. **Initialize Path:** Create an empty list for the `path`.
3. **Traverse Tree:**
- Pair the Target hash with its **Sibling** (the adjacent hash in the sorted list).
- **Record Sibling:** Add the Sibling's `hash` and relative `position` ("left" or "right") to the `path`.
- **Hash Upwards:** Compute the parent hash by concatenating and hashing the pair.
- **Repeat:** Treat this new parent hash as the current Target and repeat the process until the **Root** is reached.
4. **Finalize:** The resulting ordered list is the `path`.

## Examples

### Item Example
### Item Example (With File Integrity)

```jsonc
{
"type": "Feature",
"stac_version": "1.1.0",
"stac_extensions": [
"https://stac-extensions.github.io/file/v2.1.0/schema.json",
"https://stacchain.github.io/merkle-tree/v1.1.0/schema.json"
],
"id": "item-001",
"properties": {
"datetime": "2024-10-15T12:00:00Z",
"merkle:object_hash": "3a7bd3e2360a8e7d9f5b1c2d4e6f7890abcdef1234567890abcdef1234567890"
// ... other properties
},
"geometry": {
// ... geometry definition
"assets": {
"visual": {
"href": "https://storage.example.com/item-001.tif",
"type": "image/tiff; application=geotiff; profile=cloud-optimized",
"file:checksum": "12204a1b..." // Multihash (SHA-256)
}
},
"links": [
// ... item links
],
"assets": {
// ... item assets
}
{
"rel": "merkle-proof",
"href": "https://storage.example.com/proofs/item-001.json",
"type": "application/json"
}
]
}
```

Expand All @@ -141,18 +263,26 @@ The `merkle:hash_method` object provides details about the hash computation meth
"merkle:root": "abc123def4567890abcdef1234567890abcdef1234567890abcdef1234567890",
"merkle:hash_method": {
"function": "sha256",
"fields": ["*"],
"fields": ["properties", "assets", "geometry"],
"ordering": "ascending",
"description": "Computed by including merkle:object_hash values in ascending order and building the Merkle tree."
},
"extent": {
// ... spatial and temporal extent
"description": "Includes assets to ensure file integrity via file:checksum."
},
"links": [
// ... collection links
"stac_extensions": [
"https://stac-extensions.github.io/file/v2.1.0/schema.json",
"https://stacchain.github.io/merkle-tree/v1.1.0/schema.json"
],
"license": "proprietary",
"stac_extensions": ["https://stacchain.github.io/merkle/v1.0.0/schema.json"]
"links": [
{
"rel": "root",
"href": "https://example.com/catalog-001.json"
},
{
"rel": "merkle-proof",
"href": "https://example.com/proofs/collection-123.json",
"type": "application/json",
"title": "Proof of membership in Master Catalog"
}
]
}
```

Expand All @@ -168,7 +298,7 @@ The `merkle:hash_method` object provides details about the hash computation meth
"merkle:root": "f1e2d3c4b5a67890abcdef1234567890abcdef1234567890abcdef1234567890",
"merkle:hash_method": {
"function": "sha256",
"fields": ["*"],
"fields": ["id","title","description"],
"ordering": "ascending",
"description": "Computed by including merkle:object_hash values of child objects in ascending order and building the Merkle tree."
},
Expand All @@ -182,7 +312,7 @@ The `merkle:hash_method` object provides details about the hash computation meth
"href": "collection-456.json"
}
],
"stac_extensions": ["https://stacchain.github.io/merkle/v1.0.0/schema.json"]
"stac_extensions": ["https://stacchain.github.io/merkle-tree/v1.1.0/schema.json"]
}
```

Expand Down
22 changes: 19 additions & 3 deletions examples/collection.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"stac_version": "1.1.0",
"stac_extensions": [
"https://stac-extensions.github.io/file/v2.1.0/schema.json",
"https://stacchain.github.io/merkle-tree/v1.0.0/schema.json"
],
"type": "Collection",
Expand Down Expand Up @@ -39,9 +40,14 @@
"description": "Computed by including merkle:object_hash values in ascending order and building the Merkle tree."
},
"assets": {
"example": {
"href": "https://example.com/examples/file.xyz",
"template:new_field": "test"
"metadata": {
"href": "https://example.com/examples/metadata.xml",
"type": "application/xml",
"title": "ISO 19115 Metadata",
"roles": [
"metadata"
],
"file:checksum": "12204a1b0b3e2360a8e7d9f5b1c2d4e6f7890abcdef1234567890abcdef123456"
}
},
"summaries": {
Expand All @@ -58,6 +64,16 @@
{
"href": "https://example.com/examples/item.json",
"rel": "item"
},
{
"rel": "root",
"href": "https://example.com/catalog-001.json"
},
{
"rel": "merkle-proof",
"href": "https://example.com/proofs/collection-123.json",
"type": "application/json",
"title": "Proof of membership in Master Catalog"
}
]
}
9 changes: 8 additions & 1 deletion examples/item.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"stac_version": "1.1.0",
"stac_extensions": [
"https://stac-extensions.github.io/file/v2.1.0/schema.json",
"https://stacchain.github.io/merkle-tree/v1.0.0/schema.json"
],
"type": "Feature",
Expand Down Expand Up @@ -46,12 +47,18 @@
{
"href": "https://example.com/examples/item.json",
"rel": "self"
},
{
"rel": "merkle-proof",
"href": "https://storage.example.com/proofs/item-001.json",
"type": "application/json"
}
],
"assets": {
"data": {
"href": "https://example.com/examples/file.xyz",
"template:new_field": "test"
"type": "image/tiff; application=geotiff",
"file:checksum": "12204a1b0b3e2360a8e7d9f5b1c2d4e6f7890abcdef1234567890abcdef123456"
}
}
}
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
"format-examples": "stac-node-validator . --format --schemaMap https://stac-extensions.github.io/template/v1.0.0/schema.json=./json-schema/schema.json"
},
"dependencies": {
"remark-cli": "^12.0.0",
"remark-cli": "^12.0.1",
"remark-gfm": "^4.0.0",
"remark-lint": "^9.1.2",
"remark-lint-no-html": "^3.1.2",
Expand Down