Skip to content

Draft local state query serialization tests #1501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 27 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
f29e0e9
Update index-state
jasagredo May 27, 2025
5271869
Update to newer ledger
jasagredo May 27, 2025
244c65e
Provide backwards-compatible codecs for PParams and CompactGenesis
jasagredo May 27, 2025
bb82d84
NEEDS NEW NTC VERSION: New NTC versions for Cardano and Shelley
jasagredo May 27, 2025
b4eb858
New LedgerState golden files for pre-Conway eras
jasagredo May 27, 2025
3d4540e
Fix LedgerConfig encoding
jasagredo May 27, 2025
3f500de
New golden files for CardanoNodeToClientVersion17
jasagredo May 27, 2025
9ea62ae
New/Modify golden files for byron and shelley
jasagredo May 27, 2025
901c357
Use NTC21
jasagredo May 28, 2025
5fad2fe
Use lenses when generating transactions
jasagredo May 28, 2025
8dfdd6a
Move Legacy types to their own modules
jasagredo May 28, 2025
f1afee4
Missing golden files for LedgerConfig
jasagredo May 28, 2025
fcc0a41
Nix SHAs
jasagredo May 28, 2025
36481b1
Add base CDDL
jasagredo May 23, 2025
6de9ade
Add NodeToNode CDDLs
jasagredo May 23, 2025
d9ce40d
Add Disk CDDLs
jasagredo May 23, 2025
67f8cb3
Use StandardCrypto on block generation in Cardano and Shelley
jasagredo May 23, 2025
3b8e847
Add CDDL testing to golden and roundtrip tests
jasagredo May 23, 2025
9c9284f
Run CDDL tests in `cardano-test`
jasagredo May 23, 2025
13a512e
Attempt to add the relevant tools to ci
jasagredo May 23, 2025
19bd27e
flake.nix: add cuddle to dev shell
geo2a May 23, 2025
04ec1c0
Use Byron CDDLs
jasagredo May 27, 2025
184477b
Update Ledger for a correct conway.pool_metadata
jasagredo May 28, 2025
f03513d
Some cleanup from code-review
jasagredo May 28, 2025
e142fc8
Add cddl to dev shell
ch1bo May 7, 2025
9961c68
Add cardano-blueprint submodule
ch1bo May 7, 2025
8e232a0
Draft a first set of tests for query/result serialization
ch1bo May 7, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,22 @@ jobs:
- name: Build projects [build]
run: cabal build all -j

- name: Install cddlc
run: gem install cddlc

- name: Clone cuddle with validator
uses: actions/checkout
with:
repo: 'input-output-hk/cuddle'
ref: 'js/validator'
path: 'cuddle'

- name: Install cuddle
run: |
( cd $GITHUB_WORKSPACE/cuddle
cabal install --ignore-project exe:cuddle
)

- name: Test
if: matrix.test-set == 'all'
run: cabal test all -j --test-show-details=streaming
Expand Down
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[submodule "cardano-blueprint"]
path = cardano-blueprint
url = [email protected]:cardano-scaling/cardano-blueprint.git
58 changes: 43 additions & 15 deletions cabal.project
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,9 @@ repository cardano-haskell-packages
-- update either of these.
index-state:
-- Bump this if you need newer packages from Hackage
, hackage.haskell.org 2025-05-15T07:59:00Z
, hackage.haskell.org 2025-05-26T13:28:18Z
-- Bump this if you need newer packages from CHaP
, cardano-haskell-packages 2025-05-15T08:36:14Z
, cardano-haskell-packages 2025-05-26T19:51:53Z

packages:
ouroboros-consensus
Expand All @@ -40,11 +40,6 @@ package ouroboros-network
-- https://github.com/IntersectMBO/ouroboros-network/issues/4927 for context).
flags: -txsubmission-delay

-- We need to disable bitvec's SIMD for now, as it breaks during cross compilation.
if(os(windows))
constraints:
bitvec -simd

if impl (ghc >= 9.12)
allow-newer:
-- Bounds will be relaxed in next release
Expand All @@ -54,13 +49,46 @@ if impl (ghc >= 9.12)
, ouroboros-network-protocols:base
, ouroboros-network:base

-- https://github.com/phadej/vec/issues/118
, bin:base
, fin:base
, ral:base

-- https://github.com/haskellari/tree-diff/issues/97
, tree-diff:base

-- https://github.com/kapralVV/Unique/issues/11
, Unique:hashable

source-repository-package
type: git
location: https://github.com/IntersectMBO/ouroboros-network
tag: 3e8d3b4b8c87ead794876c62d7fe25f32efb5142
-- sha256: sha256-uKrxpHhcNGzKUSeA4zIdkslfDFIJjkCsHUg+FUef1yE=
subdir:
ouroboros-network-api

source-repository-package
type: git
location: https://github.com/IntersectMBO/cardano-ledger
tag: 75656430492b34e2300704eafeb77e1d5001671a
--sha256: sha256-m/Glsryh4aJBsI4ifIe4H3ZkcX6K7EGLEHCGGVzOzyI=
subdir:
eras/allegra/impl
eras/alonzo/impl
eras/alonzo/test-suite
eras/babbage/impl
eras/babbage/test-suite
eras/byron/chain/executable-spec
eras/byron/crypto
eras/byron/ledger/executable-spec
eras/byron/ledger/impl
eras/conway/impl
eras/conway/test-suite
eras/mary/impl
eras/shelley/impl
eras/shelley-ma/test-suite
eras/shelley/test-suite
libs/cardano-data
libs/cardano-ledger-api
libs/cardano-ledger-binary
libs/cardano-ledger-core
libs/cardano-ledger-test
libs/cardano-protocol-tpraos
libs/constrained-generators
libs/non-integral
libs/set-algebra
libs/small-steps
libs/vector-map
1 change: 1 addition & 0 deletions cardano-blueprint
Submodule cardano-blueprint added at 1b15ab
12 changes: 6 additions & 6 deletions flake.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions nix/shell.nix
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@ hsPkgs.shellFor {
pkgs.ghcid
pkgs.xrefcheck
pkgs.fourmolu
pkgs.cuddle

# testing cddls
pkgs.cddl

# release management
(pkgs.scriv.overridePythonAttrs (_oldAttrs: {
Expand Down
9 changes: 9 additions & 0 deletions nix/tools.nix
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,15 @@ in

fourmolu = tool "fourmolu" "0.18.0.0" { };

cuddle = tool "cuddle" "git" {
src = final.fetchFromGitHub {
owner = "input-output-hk";
repo = "cuddle";
rev = "43050522b2c3326dc2bcb95a3fde852bce5bc729";
hash = "sha256-S3GJBmvBmnbdb7tD2Fq9FNr9Z8iuT/eWwRpRxq9is10=";
};
};

haskellBuildUtils = prev.haskellBuildUtils.override {
inherit (final.hsPkgs.args) compiler-nix-name;
index-state = tool-index-state;
Expand Down
53 changes: 53 additions & 0 deletions ouroboros-consensus-cardano/cddl/base.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
telescope7<byron, shelley, allegra, mary, alonzo, babbage, conway>
= [pastEra, pastEra, pastEra, pastEra, pastEra, pastEra, currentEra<conway>] /
[pastEra, pastEra, pastEra, pastEra, pastEra, currentEra<babbage>] /
[pastEra, pastEra, pastEra, pastEra, currentEra<alonzo>] /
[pastEra, pastEra, pastEra, currentEra<mary>] /
[pastEra, pastEra, currentEra<allegra>] /
[pastEra, currentEra<shelley>] /
[currentEra<byron>]

ns7<byron, shelley, allegra, mary, alonzo, babbage, conway>
= [6, conway] /
[5, babbage] /
[4, alonzo] /
[3, mary] /
[2, allegra] /
[1, shelley] /
[0, byron]

;; Blockchain types
pastEra = [bound, bound]
currentEra<st> = [bound, st]
bound = [relativeTime, slotno, epochno]
eraIdx = word8
individualPoolStake = [stake, hash]
nonce = [0] / [1, hash]
point = [] / [ slotno, hash ]
poolDistr = map<keyhash, individualPoolStake>
slotno = word64
stake = rational

withOrigin<v> = [] / [v]
withOriginTH<v> = [0] / [1, v]

;; Collections
either<x, y> = [0, x] / [1, y]
map<x, y> = { * x => y }
maybe<x> = [] / [x]
seq<x> = [*23 x] / [24* x] ; encoded with indefinite-length encoding
set<x> = #6.258([* x])

;; Types from other packages
blockno = word64
epochno = word64
coin = word64
rational = [int, int]
keyhash = bstr .size 28
hash = bstr .size 32
relativeTime = int

;; Base word types
word8 = uint .size 1
word32 = uint .size 4
word64 = uint .size 8
15 changes: 15 additions & 0 deletions ouroboros-consensus-cardano/cddl/disk/block.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
cardanoBlock = byron.block
/ [2, shelley.block]
/ [3, allegra.block]
/ [4, mary.block]
/ [5, alonzo.block]
/ [6, babbage.block]
/ [7, conway.block]

;# import byron as byron
;# import shelley as shelley
;# import allegra as allegra
;# import mary as mary
;# import alonzo as alonzo
;# import babbage as babbage
;# import conway as conway
9 changes: 9 additions & 0 deletions ouroboros-consensus-cardano/cddl/disk/snapshot.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
ledgerStateSnapshot =
[snapshotEncodingVersion1, extLedgerState]

snapshotEncodingVersion1 = 1

extLedgerState = [ledgerState, headerState]

;# import ledgerstate
;# import headerstate
23 changes: 23 additions & 0 deletions ouroboros-consensus-cardano/cddl/disk/snapshot/headerstate.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
headerState =
[withOrigin<headerStateTip>, headerStateChainDep]

headerStateTip =
ns7<byronAnnTip, annTip, annTip, annTip, annTip, annTip, annTip>

byronAnnTip = [slotno, hash, blockno, bool]
annTip = [slotno, hash, blockno]

headerStateChainDep =
telescope7<versionedPbftState,
versionedTPraosState,
versionedTPraosState,
versionedTPraosState,
versionedTPraosState,
versionedPraosState,
versionedPraosState>

versionedPbftState = [serializationFormat1, {* keyhash => [* slotno]}]

;# import base
;# import praos
;# import tpraos
30 changes: 30 additions & 0 deletions ouroboros-consensus-cardano/cddl/disk/snapshot/ledgerstate.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
ledgerState =
telescope7<byron.ledgerstate,
versionedShelleyLedgerState<shelley.ledgerstate>,
versionedShelleyLedgerState<allegra.ledgerstate>,
versionedShelleyLedgerState<mary.ledgerstate>,
versionedShelleyLedgerState<alonzo.ledgerstate>,
versionedShelleyLedgerState<babbage.ledgerstate>,
versionedShelleyLedgerState<conway.ledgerstate>>

versionedShelleyLedgerState<era> = [ shelleyVersion2, shelleyLedgerState<era> ]

shelleyVersion2 = 2

shelleyLedgerState<era> = [ withOrigin<shelleyTip>, era, shelleyTransition ]

shelleyTip = [slotno, blockno, hash]

shelleyTransition = word32

; TODO these should be imports from the ledger however they do not
; provide these definitions yet.
byron.ledgerstate = any
shelley.ledgerstate = any
allegra.ledgerstate = any
mary.ledgerstate = any
alonzo.ledgerstate = any
babbage.ledgerstate = any
conway.ledgerstate = any

;# import base
13 changes: 13 additions & 0 deletions ouroboros-consensus-cardano/cddl/disk/snapshot/praos.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
versionedPraosState = [praosVersion, praosState]

praosVersion = 0

praosState = [withOrigin<slotno>,
{* keyhash => word64},
nonce,
nonce,
nonce,
nonce,
nonce]

;# import base
12 changes: 12 additions & 0 deletions ouroboros-consensus-cardano/cddl/disk/snapshot/tpraos.cddl
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
versionedTPraosState =
[serializationFormat1, [withOriginTH<slotno>, tpraosState]]

tpraosState = [prtclState, ticknState, nonce]

prtclState = [{* keyhash => word64}, nonce, nonce]

ticknState = [nonce, nonce]

serializationFormat1 = 1

;# import base
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
serialisedCardanoBlock = #6.24(bytes .cbor cardanoBlock)

;# import cardanoBlock from block
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
point = base.point

;# import base as base
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
header
= ns7<byronHeader,
serialisedShelleyHeader<shelley.header>,
serialisedShelleyHeader<allegra.header>,
serialisedShelleyHeader<mary.header>,
serialisedShelleyHeader<alonzo.header>,
serialisedShelleyHeader<babbage.header>,
serialisedShelleyHeader<conway.header>>

byronHeader = [byronRegularIdx, #6.24(bytes .cbor byron.blockhead)]
/ [byronBoundaryIdx, #6.24(bytes .cbor byron.ebbhead)]

byronBoundaryIdx = [0, word32]
byronRegularIdx = [1, word32]

serialisedShelleyHeader<era> = #6.24(bytes .cbor era)

;# include byron as byron
;# include shelley as shelley
;# include allegra as allegra
;# include mary as mary
;# include alonzo as alonzo
;# include babbage as babbage
;# include conway as conway
;# import base
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
point = base.point

;# import base as base
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
tip = [ base.point, base.blockno ]

;# import base as base
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
tx =
ns7<byron.tx,
serialisedShelleyTx<shelley.transaction>,
serialisedShelleyTx<allegra.transaction>,
serialisedShelleyTx<mary.transaction>,
serialisedShelleyTx<alonzo.transaction>,
serialisedShelleyTx<babbage.transaction>,
serialisedShelleyTx<conway.transaction>>

serialisedShelleyTx<era> = #6.24(bytes .cbor era)

;# include byron as byron
;# include shelley as shelley
;# include allegra as allegra
;# include mary as mary
;# include alonzo as alonzo
;# include babbage as babbage
;# include conway as conway
;# import base
Loading
Loading