This is a request to allow users of the MPT to GC events that are played into the MPT.
VIndex operates as a verifiable secondary index over append-only transparency logs. Incoming log entries are processed by mapping functions and staged in an append-only Write-Ahead Log (WAL). Batches of mapped keys are applied to an MPT.
While in-memory mpt.Snap(version) calls advance the active serving state, MPT disk snapshots are compacted and persisted to disk asynchronously in the background. If the VIndex daemon restarts or crashes, the MPT on disk will reload at the state of the last completed, fsync'd compaction snapshot.
To bridge any delta between the loaded MPT disk snapshot and the latest committed Output Log state on startup, VIndex replays missing entries from its WAL. However, to prevent unbounded WAL disk growth, we want to periodically prune older WAL records. Pruning WAL entries up to an active serving version is unsafe because an unpersisted MPT state would lose its replay data if a crash occurs before the next snapshot is fsync'd.
// PersistedVersion returns the latest version (InputLogSize) that has been
// completely written and fsync'd to disk in a persistent snapshot.
PersistedVersion() int64
A method like this allows VIndex to safely prune only WAL records that are no longer needed. Is this something that you'd be interested in exposing?
This is a request to allow users of the MPT to GC events that are played into the MPT.
VIndex operates as a verifiable secondary index over append-only transparency logs. Incoming log entries are processed by mapping functions and staged in an append-only Write-Ahead Log (WAL). Batches of mapped keys are applied to an MPT.
While in-memory mpt.Snap(version) calls advance the active serving state, MPT disk snapshots are compacted and persisted to disk asynchronously in the background. If the VIndex daemon restarts or crashes, the MPT on disk will reload at the state of the last completed, fsync'd compaction snapshot.
To bridge any delta between the loaded MPT disk snapshot and the latest committed Output Log state on startup, VIndex replays missing entries from its WAL. However, to prevent unbounded WAL disk growth, we want to periodically prune older WAL records. Pruning WAL entries up to an active serving version is unsafe because an unpersisted MPT state would lose its replay data if a crash occurs before the next snapshot is fsync'd.
A method like this allows VIndex to safely prune only WAL records that are no longer needed. Is this something that you'd be interested in exposing?