Skip to content

Commit ede57af

Browse files
hanabi1224hanabi1224
committed
address comments
1 parent d730636 commit ede57af

1 file changed

Lines changed: 117 additions & 17 deletions

File tree

‎FRCs/frc-0108.md‎

Lines changed: 117 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ a Filecoin node with a Filecoin snapshot.
2020
## Abstract
2121
<!--A short (~200 words) description of the technical issue being addressed.-->
2222

23-
The Filecoin ecosystem existed for years without specifying the snapshot format. That was fine until the advent of F3 and the resulting need to update the format in a coordinated way.
23+
The Filecoin ecosystem existed for years without specifying the snapshot format. That was fine until the advent of F3 and the resulting need to update the format in a coordinated way.
2424

2525
This document outlines:
2626
- "v1": the original accepted format found implementations through 2025 and
@@ -37,18 +37,19 @@ The time cost and the network bandwidth usage for a new Filecoin node to catch u
3737
We propose the below changes to the Filecoin CAR snapshot format.
3838

3939
- Change CAR root to be a CID that points to a CBOR-encoded [`SnapshotMetadata`](#snapshotmetadata) struct that is stored as the first data block in the CAR.
40-
- Store the raw [`F3Snapshot`](#f3snapshot) bytes as the second data block in the CAR when `F3Data != nil` in the metadata.
40+
- Store the raw [`F3Data`](#f3data) bytes as the second data block in the CAR when `F3Data != nil` in the metadata.
4141

4242
### SnapshotMetadata
4343

4444
```go
4545
type SnapshotMetadata {
46+
Version uint64 // required, format version for SnapshotMetadata. Only "2" is supported since "v1" was implied in the original format that predates `SnapshotMetadata`.
4647
HeadTipsetKey []Cid // required
47-
F3Data *Cid // optional
48+
F3Data *Cid // optional, points to F3Data structure. The only supported codec is "RAW" (0x55).
4849
}
4950
```
5051

51-
### F3Snapshot
52+
### F3Data
5253

5354
An F3 snapshot contains one header block and N data blocks (where N>0) in the below format:
5455

@@ -58,24 +59,31 @@ A header block is a CBOR-encoded [`F3SnapshotHeader`](#f3snapshotheader) with a
5859

5960
`[varint-encoded byte length of "CBOR-encoded F3SnapshotHeader"] [CBOR-encoded F3SnapshotHeader]`
6061

62+
A data block is a CBOR-encoded [`FinalityCertificate`](#finalitycertificate) with a length prefix in the below format:
63+
64+
`[varint-encoded byte length] [CBOR-encoded FinalityCertificate]`
65+
66+
Notes:
67+
- `FinalityCertificate`s should be ordered by `GPBFTInstance` in ascending order for sequential validation and intermediate power table calculation. This also ensures deterministic generation of F3 snapshot from a given F3 finality certificate chain.
68+
- The first and last `FinalityCertificate` instances should match those in the header, respectively.
69+
- This [CARv1](https://ipld.io/specs/transport/car/carv1)-like format is ideal for dumping blocks via streaming reads as the Header can be loaded first and minimal state is required for ongoing parsing.
70+
6171
### F3SnapshotHeader
6272

6373
```go
6474
type F3SnapshotHeader struct {
6575
Version uint64
66-
FirstInstance uint64 // The first FinalityCertificate.GPBFTInstance in the "data blocks" that follow the header.
67-
LatestInstance uint64 // The last FinalityCertificate.GPBFTInstance in the "data blocks" that follow the header.
76+
FirstInstance uint64
77+
LatestInstance uint64
6878
InitialPowerTable gpbft.PowerEntries
6979
}
7080
```
7181

72-
A data block is a CBOR-encoded [`FinalityCertificate`](#finalitycertificate) with a length prefix in the below format:
73-
74-
`[varint-encoded length] [CBOR-encoded FinalityCertificate]`
75-
7682
### FinalityCertificate
7783

7884
```go
85+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/certs/certs.go#L34>
86+
//
7987
// FinalityCertificate represents a single finalized GPBFT instance.
8088
type FinalityCertificate struct {
8189
// The GPBFT instance to which this finality certificate corresponds.
@@ -94,18 +102,109 @@ type FinalityCertificate struct {
94102
// used to validate the next finality certificate. Sorted by ParticipantID, ascending.
95103
PowerTableDelta PowerTableDiff `json:"PowerTableDelta,omitempty"`
96104
}
105+
106+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/gpbft/chain.go#L194>
107+
//
108+
// A chain of tipsets comprising a base (the last finalised tipset from which the chain extends).
109+
// and (possibly empty) suffix.
110+
// Tipsets are assumed to be built contiguously on each other,
111+
// though epochs may be missing due to null rounds.
112+
// The zero value is not a valid chain, and represents a "bottom" value
113+
// when used in a Granite message.
114+
type ECChain struct {
115+
TipSets []*TipSet
116+
117+
key ECChainKey `cborgen:"ignore"`
118+
keyLazyLoader sync.Once `cborgen:"ignore"`
119+
}
120+
121+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/gpbft/types.go#L123>
122+
type SupplementalData struct {
123+
// Commitments is the Merkle-tree of instance-specific commitments. Currently
124+
// empty but this will eventually include things like snark-friendly power-table
125+
// commitments.
126+
Commitments [32]byte `cborgen:"maxlen=32"`
127+
// PowerTable is the DagCBOR-blake2b256 CID of the power table used to validate
128+
// the next instance, taking lookback into account.
129+
PowerTable cid.Cid // []PowerEntry
130+
}
131+
132+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/certs/certs.go#L31>
133+
type PowerTableDiff []PowerTableDelta
134+
135+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/certs/certs.go#L16>
136+
//
137+
// PowerTableDelta represents a single power table change between GPBFT instances. If the resulting
138+
// power is 0 after applying the delta, the participant is removed from the power table.
139+
type PowerTableDelta struct {
140+
// Participant with changed power
141+
ParticipantID gpbft.ActorID
142+
// Change in power from base (signed).
143+
PowerDelta gpbft.StoragePower
144+
// New signing key if relevant (else empty)
145+
SigningKey gpbft.PubKey `cborgen:"maxlen=48"`
146+
}
147+
148+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/gpbft/types.go#L15>
149+
type ActorID uint64
150+
151+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/gpbft/types.go#L17>
152+
type StoragePower = big.Int
153+
154+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/gpbft/types.go#L19>
155+
type PubKey []byte
156+
157+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/gpbft/chain.go#L52>
158+
//
159+
// TipSet represents a single EC tipset.
160+
type TipSet struct {
161+
// The EC epoch (strictly increasing).
162+
Epoch int64
163+
// The tipset's key (canonically ordered concatenated block-header CIDs).
164+
Key TipSetKey `cborgen:"maxlen=760"` // 20 * 38B
165+
// Blake2b256-32 CID of the CBOR-encoded power table.
166+
PowerTable cid.Cid
167+
// Keccak256 root hash of the commitments merkle tree.
168+
Commitments [32]byte `cborgen:"maxlen=32"`
169+
}
170+
171+
// Defined at <https://github.com/filecoin-project/go-f3/blob/v0.8.7/gpbft/chain.go#L20>
172+
//
173+
// TipSetKey is the canonically ordered concatenation of the block CIDs in a tipset.
174+
type TipSetKey = []byte
97175
```
98176

99-
Notes:
100-
- `FinalityCertificate`s should be ordered by `GPBFTInstance` in ascending order, thus they can be validated and intermediate power tables can be generated while data blocks are being read in a stream.
101-
- The first and last `FinalityCertificate` instances should match those in the header, respectively.
177+
### Filecoin snapshot v1 format
178+
179+
We define the existing Filecoin snapshot format as v1 here for future reference.
180+
181+
Filecoin snapshot v1 is in [CARv1](https://ipld.io/specs/transport/car/carv1/) format.
182+
183+
The roots array in the `CarHeader` stores the tipset keys of the chain head in the snapshot.
184+
185+
The data blocks are chain IPLD blocks generated in a deterministic depth-first traversal order during chain export. Thus a snapshot is deterministic for a given chain head and the number of state trees to include. (The details of the chain traversal algorithm can be found in Filecoin node implementations)
186+
187+
```go
188+
type CarHeader struct {
189+
version Int
190+
roots [&Any]
191+
}
192+
```
102193

103194
## Backwards Compatibility
104195
<!--All FIPs that introduce backwards incompatibilities must include a section describing these incompatibilities and their severity. The FIP must explain how the author proposes to deal with these incompatibilities. FIP submissions without a sufficient backwards compatibility treatise may be rejected outright.-->
105196

106-
- A Filecoin node should try to read a snapshot CAR in the proposed format. A failure to successfully decode the blocked referenced as as the CAR's single root using the schema presented above, a snapshot reader may fallback to the old format and maintain backward compatibility. Additional failures to decode original snapshot format would indicate a fatal error.
107-
- CLI options for implementations like Forest and Lotus should remain unchanged to make it transparent to the node users.
108-
- The code change in all Filecoin nodes should be shipped with a network upgrade, and the Filecoin snapshot providers should only start publishing with the new format after the mainnet upgrade finishes to avoid potential errors during snapshot import for node users.
197+
- A Filecoin node should try to read a snapshot CAR in the proposed format. A failure to successfully decode the block referenced as the CAR's single root using the schema presented above, a snapshot reader may fallback to the old format and maintain backward compatibility. Additional failures to decode original snapshot format would indicate a fatal error.
198+
- CLI options for implementations like Forest, Lotus and Venus should remain unchanged to make it transparent to the node users.
199+
- The code change in all Filecoin nodes should be shipped with a network upgrade, and the Filecoin snapshot providers should only start publishing with the new format after the mainnet upgrade finishes to avoid potential errors during snapshot import for node users. That is to say:
200+
- Before NV27,
201+
- a node reads both v1 and v2 snapshots
202+
- a node generates v1 snapshot by default
203+
- node provides publish and host v1 snapshots.
204+
- After NV27,
205+
- a node reads both v1 and v2 snapshots
206+
- a node generates v2 snapshot by default
207+
- node providers publish and host v2 snapshots.
109208

110209
## Test Cases
111210
<!--Test cases for an implementation are mandatory for FIPs affecting consensus changes. Other FIPs can choose to include links to test cases if applicable.-->
@@ -116,7 +215,8 @@ Notes:
116215
This change has minimal security implications as the additional F3 data are also stored in the node database, unencrypted. Key considerations:
117216

118217
- **Integrity**: The F3 snapshot can be validated.
119-
- **Performance** The F3 snapshot data blocks can be read, validated and imported in a stream.
218+
- **Performance** The F3 snapshot data blocks can be read, validated and imported in a stream, without requiring to hold the entire finality certificate chain in the RAM. To facilitate this, it might require some new API(s) in the CAR reader package.
219+
- **Cyclic structure** Not applicable. The F3 snapshot does not build any cyclic graph during import and export, only a single block(certificate) is required to be held in the RAM.
120220

121221
The change does not introduce new attack vectors or modify existing security properties of the protocol.
122222

0 commit comments

Comments
 (0)