CDDL based Node2Client LocalStateQuery API

[ch1bo]

Why

Users repeatedly have asked for a reliable API reference of the LocalStateQuery protocol available on the Node-to-Client (N2C) socket of the cardano-node.

The cardano-node implements this API through the ouroboros-network, ouroboros-consensus and cardano-ledger components, where the resulting protocol is comprised by messages and their binary CBOR encodings from these three components.

There are multiple clients/servers with varying completeness of this API:

• cardano-api
• ogmios
• pallas
• gouroboros and dingo
• ouroboros-miniprotocols-ts (even contains .cddl files).

Implementing them resulted in users asking for this in multiple issues scattered across repositories:

• [FEAT] - Provide CDDL specs for ledger state queries IntersectMBO/ouroboros-consensus#439 (links to reverse engineering)
• CDDL specs for ledger state queries IntersectMBO/cardano-ledger#4396
• Clarity on ledger state queries IntersectMBO/ouroboros-consensus#246
• Write a CDDL spec for LocalStateQuery and LocalTxSubmission IntersectMBO/ouroboros-consensus#1370 (good comments on approach)
• Comprehensive information about Cardano wire protocol IntersectMBO/ouroboros-network#5049
• [FR] - Provide comprehensive CDDL grammar for local tx-submission protocol IntersectMBO/cardano-node#5139 (asks about tx submission, but good comments on why)
• Document serialisation used by Cardano IntersectMBO/ouroboros-consensus#7

This is a meta-issue to collect these issues and tackle them in course of the cardano-blueprint initiative. On top of these issues, there has been also repeated questions on communication channels about this API.

What

An API reference for the N2C protocols, especially the LocalStateQuery should be:

1. human readable to discover available functionality
2. machine readable to test and ensure consistency with implementations
3. easy to maintain

Concretely this means we need:

• A high-level API description like in ogmios
• An explanation of the protocol (acquiring, querying, ...) with messages defined as CDDL
• A definite list of all queries available
  • can be searched
  • can be linked to
• Each query holds an explanation (to answer questions like these)
• Each query and its possible responses are defined as a CDDL
  • Can be used to encoder/decoder implementations
  • Notable implementations like pallas, gouroboros and dingo have test suites to check conformance
• The API definition is automatically checked against the cardano-node reference implementation
  • Contributors to ouroboros-network, ouroboros-consensus and cardano-ledger get notified about inconsistencies
• The API definition is versioned, such that
  • it is clear which API is available in a given cardano-node version (or any of other client/server implementations)
  • modifications of the API can be done freely before releasing components

How

As also discussed in this and this issue, the concrete API is a combination of the ouroboros-network protocol, that delegates handling of a list of queries defined by ouroboros-consensus, which in turn often relies on cardano-ledger to provide the actual query results.

Current situation

• ouroboros-network defines a CDDL for localStateQueryMessage in local-state-query.cddl

  • This is using query = any and result = any placeholders

• ouroboros-consensus does not define a CDDL ledger queries, there is this PR though: Refine the CDDL spec of the LocalStateQuery mini-protocol IntersectMBO/ouroboros-network#5074, where (shelley era) query messages are defined in data instance BlockQuery and encoding/decoding is provided by encodeShelleyQuery and encodeShelleyResult.

  • Depending on the result type, the cardano-ledger is involved in providing the response.
  • Consensus "native" results are not explicitly covered by a CDDL

• cardano-ledger generates a composite CDDL for the conway era using cuddle into conway.cddl (with "huddle" source here)

  • It is not always clear which schema is used, e.g. the UTxO result of GetUTxOByTxIn query (needs to know how the Haskell type UTxO is mapped to CBOR)
  • This does not hold everything needed to describe query results, e.g. PParams seems not to be covered

Concrete solution idea

Create a binary, CDDL-based equivalence of the ledger state query documentation of ogmios:

• Written in markdown
• Rendered and hosted as static website
• Enumerates all queries and explains them
  • references .cddl files (see below)
  • (optional) includes examples / test data
• Create individual .cddl files for each query/result
  • References the relevant rules from cardano-ledger's conway.cddl
  • Using CDDL module extension
  • Add tests to ouroboros-consensus that test each Query
    • This requires the individual query .cddl files to be composed with conway.cddl
• Provide instructions to do similar tests for client implementations
  • (optional) Show how to generate arbitrary CBOR with cuddle to test client-side decoders
• Create a full, composite .cddl of the overall protocol
  • Can be used to explain all messages received on the wire
  • Enumerates all possible query and result combinations
  • Is based on ouroboros-network's local-state-query.cddl
    • (optional) Update ouroboros-network tests to use this composite cddl?
  • This may also require to be composed with conway.cddl
  • Using CDDL module extension

Steps

• Create an example local state query API description and CDDL #29
• Validate example with cardano-node components
• Validate example with pallas or gouroboros clients
  • Cardano blueprint localstatequery tests txpipe/pallas#638
  • Cardano blueprint localstate query tests blinklabs-io/gouroboros#992
• Create full API description and CDDLs
  • Add CDDLs IntersectMBO/ouroboros-consensus#1422

TBD

• Latest version and era is sufficient for a first pass?

[ch1bo]

Sketches on status quo:

and goal of this issue:

[jasagredo]

I have a proposal for a CDDL spec structure. They could either live in a new repo or the cardano-blueprint one could be used, I will assume they live in a separate repo cardano-cddl.

1. The cardano-cddl repo has different folders for the different layers of a node that emit serialized data. Note that the way these specs are generated is not specified (manually writing them or using huddle would be acceptable), but in the end the result is just plain .cddl files:

   1. network: contains the .cddl files for the messages of the mini-protocols.
   2. storage: contains the .cddl files for disk storage (ledger snapshots, immutable and volatile files)
   3. consensus: contains the hard-fork-wrapped data used in network and storage (cardano header, cardano point, cardano block, cardano transaction).
   4. ledger: contains a folder for each era with the .cddl files for all types involved in blocks, headers, and ledger states, used by the files in the consensus folder.
   5. Possibly a prelude file with common definitions would be useful.

2. Question: what to do with different codec versions?

3. The CDDL specs make use of https://www.ietf.org/archive/id/draft-ietf-cbor-cddl-modules-02.html for module hierarchy. For example, a block in Consensus would simply be:

   cardanoBlock = byron.block
             / [2, shelley.block]
             / [3, allegra.block]
             / [4, mary.block]
             / [5, alonzo.block]
             / [6, babbage.block]
             / [7, conway.block]
   
   ;# import byron as byron
   ;# import shelley as shelley
   ;# import allegra as allegra
   ;# import mary as mary
   ;# import alonzo as alonzo
   ;# import babbage as babbage
   ;# import conway as conway
   

   where the <era>.block definition belongs to each one of the ledger files.

4. Instructions for generating data from such files and checking cbor data against them is provided in cardano-cddl, possibly with the standard cddl ruby tool.

5. All Cardano implementations shall declare whether they are compliant with the CDDL specs, by testing their serialized data against them. I think it should be possible for an implementation to declare they support part of the spec (for example only certain queries).
   Implementations will usually import the cardano-cddl spec as a git submodule.

6. When there is a change in a codec or new versions are added in an implementation, the developers on such implementation should propose such change to the cardano-cddl repository.
   Say for example I change how points are serialized in ouroboros-consensus, I would propose such change to the cardano-cddl spec, and once accepted I update the ref for cardano-cddl in consensus and test against the updated spec. If I changed the serialization and didn't propose the change to the spec, I would be making my implementation not-compliant with the official CDDLs anymore.

7. Question: Does this imply that we will need versions for the official spec CDDLs?

8. Question: Who decides what changes are accepted in the official CDDLs spec?

9. The cardano-blueprint project then links to the cddl files where appropriate, without explicitly dumping .cddl files but instead describing the behavior in prose.

We can discuss this further on the blueprint office hours or in an ad-hoc call @ch1bo.

[ch1bo]

@jasagredo Great find on the module extension for CDDLs. Using that makes sense to me.

[dnadales]

The API definition is automatically checked against the cardano-node reference implementation

What kind of checks do you have in mind for this @ch1bo. And in particular, would the checks described by @jasagredo in this comment suffice?

[dnadales]

This is another question we need to answer: is it ok if the CDDL specs live in the repositories used by the reference Cardano implementation (https://github.com/input-output-hk/cardano-node) or should they live in another repo (if so which one)?

[lehins]

When there is a change in a codec or new versions are added in an implementation, the developers on such implementation should propose such change to the cardano-cddl repository.
Say for example I change how points are serialized in ouroboros-consensus, I would propose such change to the cardano-cddl spec, and once accepted I update the ref for cardano-cddl in consensus and test against the updated spec. If I changed the serialization and didn't propose the change to the spec, I would be making my implementation not-compliant with the official CDDLs anymore.

This is not going to work as workflow for ledger. When we define a new era and start adding features then CDDL for transactions changes rapidly. The only way I can see it could work is if we do not publish any changes to the cardano-cddl spec for a new era until that era is pretty much ready.

That being said, I am not really sure about the whole submission of CDDL changes and wait for approval? Presumably it is the maintainers of the project that will now the best what CDDL should look like for what they are maintaining. Of course, providing more exposure and ability for the community to give feedback on the changes will be nice. But, seeking approval of someone who is not an expert on the CDDL for all of Cardano makes little sense to me.

@jasagredo FYI, the reason why I was suggesting that all core teams could use huddle for specifying CDDL is because a repo like cardan-cddl could be automatically populated from the latest CHaP releases of all of our packages. That would be the simplest way to have a central repo with CDDL with the least amount of human intervention.

[ch1bo]

FYI, the reason why I was suggesting that all core teams could use huddle for specifying CDDL is because a repo like cardan-cddl could be automatically populated from the latest CHaP releases of all of our packages. That would be the simplest way to have a central repo with CDDL with the least amount of human intervention.

@lehins Let's do that for cardano-ledger as that component is using huddle already. How would you go about it?

[jasagredo]

I imagine the usual flow will be:

• Ledger implements a new era with any changes needed without looking at the CDDLs. This new era is itself not reflected yet in the CDDL so the previous era CDDLs will still work.
• Once the new era is done, Ledger proposes the changes to the cardano-cddl, with the new era constructs and the new hard-fork tag for consensus wherever it fits.
• Such changes are reviewed and approved by the ledger-consensus-network maintainers, presumably with no objections. Community has a chance to give feedback. Other implementations also have a chance to chime in.
• The changes to the cardano-cddl are merged. A new "version of the spec" is therefore stablished.
• Ledger/Consensus/Network update their git-submodule ref to cardano-cddl and create new golden files/tests.
• New versions of Ledger/Consensus/Network are released with the new era and compliant with the new cddl spec.

[jasagredo]

A couple of pain points with Huddle that were raised on last week's Blueprint office hours were:

• Comments on particular fields or choices are gone. Doesn't huddle support those? It only supports comments for top-level bindings?
• The sorting of the definitions is lexicographic, so one has to jump around the file to find the right definitions.

Perhaps these two could be solved in huddle?

[ch1bo]

I imagine the usual flow will be:

I can see this also working for new versions of things (protocols, eras) where things are just labeled as "staging" or "unrelased". No need to limit ourselves to do everything in one big merge. If we are using submodules, implementations are depending on a specific revision anyways and can signal compatibility that way.