Ledger state query api example

.github/workflows/check-examples.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+set -uo pipefail
+
+# Check consistency of .cbor files against .cddl in the repository.
+
+# TODO: check whether 'cddl' is installed
+
+# Check that given cbor is valid against all given cddls files concatenated
+function check_cbor() {
+    local cbor_file=$1
+    shift
+    local cddl_files=("$@")
+    local cddl=$(cat "${cddl_files[@]}")
+    # NOTE: cddl spams stdout on invalid cbor, so we drop everything after '-- cannot complete'
+    out=$(cddl <(echo "${cddl}") validate <(xxd -r -p "${cbor_file}") 2> /dev/null | \
+            awk -v RS='--' 'NR==1{print} NR==2{exit}') 
+    if [ $? -ne 0 ]; then
+        echo "Invalid cbor file: ${cbor_file}"
+        echo "Error: ${out%--*}"
+        printf "CDDL: \n${cddl}"
+        exit 1
+    fi
+}
+
+cd src/api
+check_cbor examples/getSystemStart/query.cbor cddl/local-state-query.cddl cddl/getSystemStart.cddl
+check_cbor examples/getSystemStart/result.cbor cddl/local-state-query.cddl cddl/getSystemStart.cddl

.github/workflows/check-examples.yaml

@@ -0,0 +1,29 @@
+name: Check cbor examples
+
+on:
+  pull_request:
+  push:
+    branches:
+    - main
+  schedule:
+    # Everyday at 4:00 AM
+    - cron: "0 4 * * *"
+
+jobs:
+  check-cbor-examples:
+    name: Check cbor examples
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+
+    - name: Install nix
+      uses: cachix/install-nix-action@V27
+      with:
+        extra_nix_config: |
+          accept-flake-config = true
+          log-lines = 1000
+
+    - name: Enter nix shell
+      uses: nicknovitski/nix-develop@v1
+
+    - run: ./.github/workflows/check-examples.sh

flake.nix

@@ -25,14 +25,18 @@
                 mdbook-katex
                 mdbook-alerts
                 mdbook-toc
+
+                # to validate cbor / cddl
+                cddl
+                xxd
               ];
               phases = [ "unpackPhase" "buildPhase" ];
               buildPhase = ''
                 mdbook build -d $out
               '';
             }
           );
-        defaultPackage = packages.mdbook;
+        packages.default = packages.mdbook;
       }
     );
 }

src/api/cddl/getSystemStart.cddl

@@ -1,5 +1,6 @@
-query = 1
-result = [year, dayOfYear, timeOfDayPico]
+$query /= [1]
+$result /= [year, dayOfYear, timeOfDayPico]
+
 year = bigint
-dayOfYear = int
+dayOfYear = uint
 timeOfDayPico = bigint

src/api/cddl/local-state-query.cddl

@@ -18,17 +18,14 @@ acquireFailurePointNotOnChain = 1
 failure      = acquireFailurePointTooOld
              / acquireFailurePointNotOnChain
 
-query        = any
-result       = any
-
 msgAcquire   = [0, point]
              / [8]
              / [10]
 msgAcquired  = [1]
 msgFailure   = [2, failure]
 ; ANCHOR: api
-msgQuery     = [3, query]
-msgResult    = [4, result]
+msgQuery     = [3, $query]
+msgResult    = [4, $result]
 ; ANCHOR_END: api
 msgRelease   = [5]
 msgReAcquire = [6, point]

src/api/examples/getSystemStart/result.cbor

@@ -1 +1 @@
-820483c2581e65fea62360470c59141d0ba6cc897f99e050184606937264a1f8c5026abc3b3a5d754770442481c3581e50670ee65e805e3cc5aadf6619e791db8b1c2237dd918ba3b6818e7c258a
+820483c2581ecca45aff5ff13d202641114479adc1b566b8a778514878b6b8c7410aa4a41a1fbe21aac2581e9de64341ff22211c81b5479ea4ad812682e3db52f36447dfc107a992e49d

src/api/node-to-client.md

@@ -90,6 +90,9 @@ _Since: v9_
 
 Query the chain's start time as a `UTCTime`.
 
+> [!CAUTION]
+> Is this the Byron or Shelly start time? 
+
 ```cddl
 {{#include cddl/local-state-query.cddl:api}}
 ```
@@ -110,6 +113,10 @@ Example response:
 {{#include examples/getSystemStart/result.cbor}}
 ```
 
+> [!WARNING]
+> TODO: Create more realistic examples and use a tool to probabilistically verify more data.
+> TODO: Also use CBOR diagnostic notation for examples
+
 > [!CAUTION]
 > FIXME: While I experimented in using the network / consensus cddl parts above, `time` would be defined in the CDDL prelude (a number, assuming seconds since epoch), but is actually incorrect and the result is serialized using `ToCBOR UTCTime` following this cddl:
 > ```cddl

src/logbook.md

@@ -1,5 +1,52 @@
 Logbook about `cardano-blueprint` that contains thinking, discussions, pains, joys, events, and experiences that happen on a daily basis. It is supposed to be a kind of [Stream of consciousness](https://en.wikipedia.org/wiki/Stream_of_consciousness) that can later be searched, linked to or reviewed. It may also be used as a very informal decision log.
 
+## 2025-05-09
+
+By @ch1bo on writing local state query serialization tests in `ouroboros-consensus`
+
+- If we want to use `cddlc`, we need to package it using `nix` to make it available to CI.
+- Cuddle based tests in `cardano-ledger` are way faster than the ones running the ruby-based `cddl` tool.
+- The LocalStateQuery API description and example cbor I envisioned to have in the cardano-blueprint would be *full messages* as sent over the wire.
+- After moving the test suite into `ouroboros-consensus-cardano` tests I seem to have everything in scope, network decoders and `CardanoBlock` which selects all the right things.
+- Making sure the right type class instances are in scope was a bit tricky (needed imports like `import Ouroboros.Consensus.Shelley.Ledger.SupportsProtocol ()`)
+- Finally I have a working roundtrip test for queries and when doing it for results (and some `typed-protocols` annoyance) I came across this weird thing:
+  - After correctly decoding the `GetSystemStart` result from:
+
+        820483c2581e65fea62360470c59141d0ba6cc897f99e050184606937264a1f8c5026abc3b3a5d754770442481c3581e50670ee65e805e3cc5aadf6619e791db8b1c2237dd918ba3b6818e7c258a
+
+  - Re-encoding it as `MsgResult` produces:
+
+        820483c2581e65fea62360470c59141d0ba6cc897f99e050184606937264a1f8c5026abc01c3581e50670ee65e805e3cc5aadf6619e791db8b1c2237dd918ba3b6818e7c258a
+
+  - In diagnostic notation:
+
+        [
+            4,
+            [
+                2(h'65fea62360470c59141d0ba6cc897f99e050184606937264a1f8c5026abc'),
+                -4205646576720553090_3,
+                3(h'50670ee65e805e3cc5aadf6619e791db8b1c2237dd918ba3b6818e7c258a'),
+            ],
+        ]
+
+    becomes
+
+        [
+            4,
+            [
+                2(h'65fea62360470c59141d0ba6cc897f99e050184606937264a1f8c5026abc'),
+                1,
+                3(h'50670ee65e805e3cc5aadf6619e791db8b1c2237dd918ba3b6818e7c258a'),
+            ],
+        ]
+
+  - Overflow on the second field of `UTCTime`!?
+
+  - Turns out the `FromCBOR UTCTime` is using [`fromOrdinalDate`](https://hackage.haskell.org/package/time-1.14/docs/Data-Time-Calendar-OrdinalDate.html#v:fromOrdinalDate) which clips `dayOfYear` (the second field, an `Int`) to valid range `[1, 366]`
+- So in fact `dayOfYear` in `cddl` should not be an `int` (with which the example was generated)!
+- Made a CI job to check cbor against cddl in `cardano-blueprint`, switched `dayOfYear` to `uint` and regenerate example. Now it fails because the generated `532554154` is clamped to `366`.
+- Are roundtrips even required? As long as we can decode what comes from the CDDL and what we encode is conformant that might be good enough?
+
 ## 2025-03-25
 
 By @ch1bo