diff --git a/docs/website/contents/for-developers/HardForks.md b/docs/website/contents/for-developers/HardForks.md new file mode 100644 index 0000000000..02d9b39182 --- /dev/null +++ b/docs/website/contents/for-developers/HardForks.md @@ -0,0 +1,134 @@ +# Hard Forks in Cardano + +## Introduction + +This document details how the Cardano node handles a [hard fork](https://en.wikipedia.org/wiki/Fork_(blockchain)#Hard_fork), and why it handles hard forks that way. + +Cardano mainnet is intended to hard fork when and only when the major component of the [`protocol version`](https://github.com/IntersectMBO/cardano-ledger/blob/2ff5fa4e8c6b773748799981ef44ef6a62cd5c92/libs/cardano-ledger-core/src/Cardano/Ledger/Core/PParams.hs#L350) changes. +The protocol version can only change due to on-chain governance [^updating-protver], and in practice the major protocol version must only ever increase by one [^PVP-style]. + +- The requirement that a major protocol version increment implies a hard fork is merely the traditional semantics of the major component of a version, ie _backwards-incompatibility_ (aka _breaking_ change). + +- The requirement of on-chain governance for each hard fork means that --- as of the arrival of decentralized governance to the Cardano mainnet chain --- it's the community that decides when to transition to the next iteration of the protocol rules and/or ledger rules, ie when to abandon any nodes that have not yet updated their software in preparation for the hard fork. + +There are other Cardano networks besides mainnet, notably the testnets. +By their essential nature, though, testnets should also use mainnet's mapping between each major protocol version number and the corresponding pair of protocol rules and ledger rules. +On the other hand, these other networks must be able to hard fork at different times than mainnet does --- the testnets should have hard forked successfully before mainnet hard forks! +Since the testnet governance and mainnet governance are independent, the determination of the hard fork timing via on-chain governance immediately enables it to vary on different networks' chains. + +Beyond those fundamentals, the community of Cardano developers must also decide how to implement the changes that underly a hard fork, how to organize them within the nodes' specifications and implementations [^multiple-nodes]. + +## Concrete Dynamics + +A hard fork in Cardano proceeds in phases. + +- *Implementation*. + Cardano developers introduce some backwards-incompatible changes to the protocol and/or ledger rules in the node software [^change-decisions]. +- *Release*. + As a decentralized network, it is impractical to rely on all Cardano node operators simultaneously updating their software. + Therefore, a new release of the software implements the new rules _and also_ the old rules. + That release continues to use the old rules --- even for fresh blocks --- until the community ultimately chooses to switch to the new rules by increasing the major protocol version parameter via on-chain governance [^frame-rule]. +- *Adoption*. + It is crucial that the community does not increment the major protocol version until a vast majority of (honest) stake has upgraded their nodes to the new software. + Stake pool operators should therefore not vote to increment before upgrading their nodes to the new software. + +Because the protocol version is determined on-chain, it's perfectly plausible for divergent chains to increment the major version at different times. +The Cardano node ought to handle that scenario gracefully, if only to bound degradation during atypical scenarios (eg low participation within the limits of Praos's _dynamic availability_ feature). +However, branching realities are disruptive even within blockchain communities, so the current design additionally ensures that even temporarily divergent chains (aka _short forks_) will agree about when to increment the major version as long as mainnet enjoys high participation from its honest majority of stake. + +## Eras + +When Cardano developers design and implement the changes for a hard fork, they determine whether the changes are easy to maintain as variations within the existing code. +If so, the hard fork is ultimately implemented by some conditionals within the code that branch on the major protocol version. +If the changes are not so simple, the Cardano developers instead introduce a new _era_ within the code [^roadmap-disambiguation]. +(See "Which changes require a new era?" below --- this determination is done within the context of the existing implementation's architecture, not a vacuum.) + +In the intended usage, each era defines some new types and rules that are independent within the code from the existing eras. +For example, the [Conway era introduces](https://docs.cardano.org/about-cardano/evolution/upgrades/chang) many new parts of the ledger state, transaction types, corresponding rules, etc that the preceding era does not have, and so the architects decided to explicitly separate the Conway code from the old code as a new era. +The Ledger and Consensus infrastructure code is defined so that each era can flexibly reuse whichever of the older eras' features it does not supersede. + +Several eras coexist sequentially on the Cardano chain; a hard fork that takes one step within that sequence is called an _era transition_. +Every intentional hard fork is either an era transition or an _intra-era hard fork_ [^accidentals]. +Intra-era hard forks only introduce changes with a small effect on the code, eg a few easy-to-maintain conditionals. +Some have been bugfixes that were small by chance, and others have been changes that were only small because they had been anticipated and intentionally deferred when the era was originally designed. + +Eras are a straight-forward mechanism that enables node developers to separate the concerns arising from features that are added to or removed from Cardano over time. +The separation amongst eras inherently simplifies the development of the new features, and it also helps ensure that the Cardano node can sync the entire historical chain [^mithril] --- as well as refuse invalid alternative histories. + +### Which changes require a new era? + +It's ultimately the judgment of the node developers whether some changes should be achieved with an era transition rather than an intra-era hard fork. +In other words, eras are not the only to organize non-trivial backwards-incompatible changes in Cardano. +However, only a single node implementation has existed for several years now, and aspects of its existing design make some backwards-incompatible changes prohibitively costly (ie redesigns) unless they are delineated by an era transition. + +The Cardano developers have so far used very expressive static typing in order to provide the safety guarantees that Cardano is known for. +It's plausible that an implementation that instead used a language more suited to dynamic types could more frequently use intra-era hard forks rather than era transitions or avoid the concept of eras altogether and rely solely on logic that branches on the major protocol version. +However, we strongly believe that overall development and assurance would significantly degrade without the benefits of such precise types. + +The first era transition (from Byron to Shelley) is a notable example. +It was the first era transition we experienced, and no subsequent transition changes nearly so much. +The two code bases are drastically different, so it's hard to argue this should have been an intra-era hard fork. +In particular, this is the only era transition that changed the duration of slots and the number of slots in an epoch. +In the current codebase, any hard fork that changes those per-era constants must be an era transition, since the Consensus infrastructure was explicitly designed with that restriction as a simplifying assumption. +A different design might relax that constraint, but it has not been restrictive so far: a second change to these constants has been discussed only rarely and never actually proposed, and deferring them until whatever justifies the next era transition wouldn't obviously be prohibitively disruptive. + +### How wide-reaching are eras? + +Eras are ultimately an implementation detail of the node, a consequence of architectural trade-offs. +The node's behaviors currently reflects the eras in only two ways. + +Specifically, the node must be able to forecast the data necessary to validate a useful prefix of some header chain from the intersection of that header chain and the node's current selection, including whether each header arrived too early. + +- The familiar and intuitive notion of era is useful for organization in general, and so the eras' names might appear within parameter names in configuration files, specification documents, announcements, etc. + + However, the era transitions themselves should generally be considered as "just a big hard fork". + The foundational data is instead the protocol version, especially because not all hard forks are era transitions. + (See [this GitHub comment](https://github.com/IntersectMBO/ouroboros-consensus/issues/416#issuecomment-2669347315) for the details of how this lesson was learned.) + +- Some codecs reflect Cardano's specific era structure. + That's not strictly necessary, but a tagged union approach has made it trivial to combine the era-specific codecs into a full Cardano codec. + Specifically, top-level codecs fundamentally branch on a simple number located near the beginning of the bytestring and dispatch accordingly to the corresponding era's codecs. + + *Remark*. + These eras tags are not present in any codecs for bytes that determine hashes used on chain. + For example, suppose the only difference between some transaction's serialization in two different eras is the numeric value of the era tag. + In that case --- unless the eras use different hashing functions --- both serializations would be assigned the same transaction ID, since the function that assigns transaction IDs is applied to the bytes inside of the envelope that carries the era tag. + +## Forecasting, forewarning + +Hard forks can affect the validity of block headers, and so must be sufficiently forecastable. +For this reason, hard forks must not happen less than 36 hr (on mainnet) after the voting deadline, ie hard forks must have at least 36 hrs of forewarning --- just like other protocol parameter updates that affect headers (eg maximum block size). + +TODO how to argue that hard forks or era transitions needs more than that? + +[^updating-protver]: *Remark*. + Prior to Conway, the protocol version was just another protocol parameter; see Section 5 "Protocol Parameters" of the "A Formal Specification of the Cardano Ledger, Deliverable SL-D5" document for more details --- its latest revision can be accessed via the hyperlink in the "Formal Specification" column of the "Shelley" row of the documentation table at the top of the [README](https://github.com/IntersectMBO/cardano-ledger/blob/master/README.md) of the [`cardano-ledger` repository](https://github.com/IntersectMBO/cardano-ledger). + As of Conway's [CIP-1694](https://github.com/cardano-foundation/CIPs/tree/master/CIP-1694) implementation, the protocol version has special treatment compared to other protocol parameters, since it so clearly requires a preceding software upgrade. + The logic is the same at a high-level, but the details are tuned to ensure a hard fork could only ever surprise a very small amount of stake. + +[^PVP-style]: *Remark*. + The Cardano protocol version has two components, so-called major and minor. + The lesser component has always been zero on mainnet. + We might eventually see a use for it, but at the moment the only plausible idea we have is for both components to track backwards-incompatible changes, as in the [Haskell Package Versioning Policy](https://pvp.haskell.org/), eg. + +[^multiple-nodes]: *Remark*. + There has only been one Cardano node so far, but others are now in development. + The different nodes _must_ be able to exchange messages, but it is not yet clear how the multiple teams will cooperate on decisions such as how to specify and implement some backwards-incompatible change. + Cardano Improvement Proposals (aka [CIPs](https://cips.cardano.org/)) will certainly be involved in such alignment efforts but are not necessarily sufficient. + +[^change-decisions]: *Remark*. + At time of writing, these are almost always going to be (possibly indirectly) related to a Cardano Improvement Proposal (aka [CIP](https://cips.cardano.org/)). + +[^frame-rule]: *Clarification*. + There's no other technical reason for the community to increment the major protocol version. + +[^roadmap-disambiguation]: *Disambiguation*. + The [Cardano Roadmap](https://roadmap.cardano.org/en/) also defines some very broad "eras": Byron, Shelley, Goguen, Basho, and Voltaire. + These are more general than the eras that divide the actual evolution of the protocol rules and ledger rules within the node. + [CIP-0059](https://github.com/cardano-foundation/CIPs/blob/master/CIP-0059/feature-table.md) maintains a table showing the correspondence; it lists roadmap eras in the "Phase" column. + +[^accidentals]: *Clarification*. + Accidental hard forks (eg due to a bug) would necessarily arise within a single era, but they are not _intra-era hard forks_ because they weren't gated by a protocol version change. + +[^mithril]: *Clarification*. + Mechanisms such as [Mithril](https://github.com/input-output-hk/mithril) would provide alternatives to the node, but the node itself should always be able to revalidate (or merely reapply) all the blocks if needed/desired. diff --git a/docs/website/contents/for-developers/ProtocolVersioning.md b/docs/website/contents/for-developers/ProtocolVersioning.md new file mode 100644 index 0000000000..820e4e8b04 --- /dev/null +++ b/docs/website/contents/for-developers/ProtocolVersioning.md @@ -0,0 +1,155 @@ +# Introduction + +The Cardano protocol will change over time. +The community innovates applications and updates feature priorities, researchers refine Ouroboros, and developers improve the code. + +This document describes how Cardano accomodates these changes as a community, as a software engineering project, and as a distributed system. + +## The Protocol Version + +Each change to the Cardano protocol advances through the following steps before influencing the network's chain. + +- Someone (you!) specifies a change that is backwards-incompatible but worthwhile. + For example, it could be a [Cardano Improvement Proposal](https://cips.cardano.org/) (CIP). +- The Technical Steering Committee and Ouroboros researchers address the worth, safety, and future consequences of the proposed solution. +- Cardano developers prepare a software release that includes the changes but only enables them for chains on which the protocol version has been incremented. +- The community uses on-chain governance to increment the protocol version, thereby putting the change into effect. + (Stake pool operators must not vote to increment the protocol version before updating their software!) + +Cardano philosophy mandates that the community has the final say regarding if and when to adopt some protocol change. +They accordingly control the final step above: developers determine the exact meaning of each protocol version, but only the community increments the protocol version on mainnet. + +It is crucial that the community does not increment the protocol version before enough nodes have upgraded their software. +Each protocol version increment corresponds to a _hard fork_, because the changes are backwards-incompatible: nodes running the old code will not be able to adopt blocks minted by the new code. +Even if some block content happened to be valid in both the old and the new protocol, each software release explicitly refuses any block that (extends a ledger state that) has a protocol version greater than that release was intended to implement — see the MaxMajorPV constant. + +Even software released well after mainnet increments the protocol version must implement the protocol both with and without that increment's changes. +The Cardano node must correctly process blocks with older protocol versions, eg when catching up to mainnet for the first time or after being offline for a long time. +So every protocol change must remain conditional within all future releases, instead of only within the particular release that mainnet used for that increment of the protocol version. +(Exceptions are possible if the community decides to somehow truncate/compress a prefix of the historical chain.) + +## When to Increment the Protocol Version? + +Fundamentally, the protocol version must be incremented for some change to the protocol when an old node and a new node that received the same messages might disagree on which block is the best valid block, aka maintain consensus. +A more precise and actionable answer will be given after introducing more context. + +### The Cardano Abstraction Ladder + +For the sake of this discussion, Cardano fixes a ladder of abstractions between the following two extrema. + +- *Opacity* (bottom rung). + The Cardano node exchanges bytestrings with peers in the network and with local users (eg wallets). + At any given time, the node can determine whether such a bytestring is valid, and that predicate depends on previously exchanged bytestrings and the passage of time. + +- *Transparency* (top rung). + Users interact with Cardano nodes to interpret the current state of the network and to evolve it according to their goals. + +An analogous spectrum applies to any computer system that humans interact with, and there are longstanding traditions of how to organize the rungs between these two. +The Cardano node is internally organized as the following ladder. +The overall intent is for every Cardano use case to be satisfied by exchanging messages with nodes, so messages are the focus of some foundational rungs. + +- Opaque Rung (bottom). +- CBOR Rung. + - The _generic data model_ of [RFC 8949 Concise Binary Object Representation](https://cbor.io/). +- Message Data Rung. + - A data model for the content of Cardano messages that is sufficiently abstract for Cardano developers to easily interpret them in a consistent and useful way (eg algebraic data types). + - Note that this content includes transaction IDs, transactions, block headers, blocks, queries, query replies, etc, but not the node's internal state [^state-data-model]. + - Some notable fields within this data model are hashes computed in terms of the CBOR Rung. +- Message Validity Rung. + - The key difference from the Message Data Rung is that this rung excludes messages with invalid content. + Users usually only consider valid messages, and the honest node should certainly never send invalid messages. + - In practice, this rung requires the honest node to maintain as state a summary of all relevant messages that is sufficient to determine which messages to send [^theory-coalgebra]. + - Query replies are an awkward fit for this rung, since the node only ever sends them without ever receiving them. + But the honest node must only send valid messages, which in this case means a correct reply. + - Another important category of messages the honest node must send are messages that indicate which chain it has selected. + This rung also therefore must resolve Praos's inherent short forks. +- There are other rungs, such as the rung that abstracts from the set of nodes to the network as a whole, but they are beyond the scope of this document. +- Transparent Rung (top). + +### The Cardano Message Data Onion + +The Message Data Rung is organized as the layers of an onion. +So are some other rungs, but their structure is beyond the scope of this document. + +- [Ledger Layer](https://github.com/IntersectMBO/cardano-ledger/). + - The notion of _era_ is an organizing concept within the Ledger Layer --- more on this below. + - For each era, the content of its blocks, transactions, queries, and query responses. +- [Consensus Layer](https://github.com/IntersectMBO/ouroboros-consensus/). + - The content of the few queries and their responses that either do not depend on the ledger era or else regard when prior era transitions happened, which the Ledger Layer does not record. + - An additional envelope wrapping block headers, blocks, transactions, era-specific queries, and era-specific query responses. + The envelope indicates a particular era via an _era tag_ [^era-tag-parse], which affects the validity of the object but not its identifying hash. +- [Network Layer](https://github.com/IntersectMBO/ouroboros-network) (outermost) [^mux-layer]. + - ChainSync, BlockFetch, TxSubmission, LocalStateQuery, etc; see "Chapter 3 Mini Protocols" within the [Ouroboros Network Specification](https://ouroboros-network.cardano.intersectmbo.org/pdfs/network-spec/network-spec.pdf) [^network-pdf-link]. + - The content of each mini protocol message, excluding the payload. + +### Precise Answer + +The protocol version must be incremented if an old node and a new node would otherwise disagree on any of the following. + +- *Chain Validity Predicates*, the validity predicates within the Message Validity Rung for block headers and blocks. +- *Chain Order*, the [partial order](https://en.wikipedia.org/wiki/Partially_ordered_set) within the Message Validity Rung that determines one chain is better than another. + Tiebreakers are a gray area, since Praos does not inherently rely on them. +- *Chain Data Model*, the parts of the Message Data Rung used to define the Chain Validity Predicates and the Chain Order. + - A change to the data model of the node's maintained state --- as opposed to its messages --- would not inherently trigger this. + - However, if such a change were meaningful (ie not just a refactor/optimization/etc), then some Chain Validity Predicates would necessarily also change. +- *Chain Codecs*, the parts of the mapping from the CBOR Rung to the Chain Data Model that affect the hashes therein. + Two nodes won't even be able to exchange transactions, block headers, and/or blocks if they disagree on their identifying hashes. + +The most common change in Cardano's protocol so far has been the addition of a new kind/variant of transaction, which triggers all of these except the Chain Order. +The Chain Order has only changed from Praos's length to Ouroboros Genesis's length-soon-after-the-intersection, which is slight. +On the other hand, the tiebreaker has changed, and out of caution those changes were bundled up with unrelated changes that independently required a protocol version increment. + +It's also worth noting that the Consensus Layer and Network Layer of the Message Data Rung can change without incrementing the protocol version. +The most common example is that queries can be changed without triggering any of the above, since queries do not contain any of the Chain Data Model. +Another example is that the structure of the Consensus envelope could change. +Or the mini protocols --- excluding their payload --- could change. +All such changes are conditional on the per-connection version negotiated by the Handshake mini protocol instead of the protocol version. + +## Ledger Eras + +Any Cardano node implemented in a strongly-typed language is likely going to declare distinct types for Chain Data Models that differ between protocol versions; more precise types yield stronger assurances from the typechecker. +For the sake of such implementations, the ledger introduces a new era whenever the Chain Data Model changes. +The [sequence of eras](https://github.com/cardano-foundation/CIPs/blob/master/CIP-0059/feature-table.md) at the time of writing are: Byron [^byron-halves], Shelley, Allegra, Mary, Alonzo, Babbage, and Conway. + +A Cardano node implementation or other tooling implemented in a language prone to dynamic typing might find it convenient to ignore the era tags for the most part. +However, the era tags must still be validated according to the intended function from Cardano protocol versions to Ledger eras, since a strongly-typed node would fail to accept a block with the wrong era tag. + +Protocol changes that do not also introduce a new era are known as _intra-era hard forks_. +These are certainly sound if they only change the Chain Validity Predicate, but they generally could do more. +But so far, Cardano has forbidden intra-era hard forks from changing the Chain Codecs. +This could be allowed in either of two ways. + +- *NoEnvelope*. + The Consensus envelope could be entirely eliminated if an object's slot could be always be determined even without knowing the protocol version. + The slot could then be used to determine the appropriate protocol version, based on the node's state. + The downside here is significant: programs that do not maintain the node's state (since that is costly to do) would not necessarily be able to parse block headers, blocks, and transactions, since they wouldn't necessarily know the mapping from slot to protocol version (since it's determined by on-chain governance). +- *HideEras*. + The Consensus envelope could store the protocol version instead of the era tag. + At that point, the era tag would be an internal implementation detail of strongly-typed node implementations, and could even freely differ between them. + The only downside is that the Consensus envelope's byte size might grow slightly, but it'll still be dwarfed by the Ledger Layer portion of the wrapped block header/block/transaction. + - Notably, the Ledger already maintains an invariant that the CBOR of UTxO are forward-compatible, so the storage of each UTxO does not need to include an era tag. + That's the only object in the system that might be of comparable size as the era tag and numerous enough to incur significant costs. + - The code would also need to store the mapping from protocol version to era tags, but that is known at compile-time --- there is the slight complication that the mapping might vary between chains (eg testnets), but they already tend to have different configuration files. + +It might be plausible for the NoEnvelope option to restrict stateless programs to assume they're downstream of a node that had annotated the blocks and/or transactions with the correct protocol version. +At that point, though, this option is basically the same as HideEras, except that nodes would exclude the Consensus envelope when exchanging with other nodes. + +[^state-data-model]: *Remark*. + The state maintained by the Message Validity Rung also has a data model, but, in contrast to messages, it is an internal implementation detail of the node. + Different node implementations could have different state data models, but still implement the same protocol, ie send/receive/accept/reject the exact same messages. + +[^theory-coalgebra]: *Remark*. + In theory, on the other hand, that summary state could merely be the sequence of valid received and sent messages along with their timestamps. + +[^era-tag-parse]: *Remark*. + In particular, it's the only part of the parsed object for which the codec does not depend on the era. + +[^mux-layer]: *Disclaimer*. + This Network Layer of the Cardano Message Data Onion actually contains an additional layer, the Mux Layer, but that is transparent to the rest of system and beyond the scope of this doucment. + +[^network-pdf-link]: *Note*. + If that link breaks, the document can usually be obtained via the "The Shelley Networking Protocol" hyperlink in the `README` file of the [`ouroboros-network` repository](https://github.com/IntersectMBO/ouroboros-network). + +[^byron-halves]: *Historical Note*. + Byron originally had two halves (Ouroboros Classic with Epoch Boundary Blocks versus round-robin with no EBBs), but today's node does not differentiate the two. + It uses the same Chain Validity Predicate for both: the Permissive BFT protocol, which essentially ignores any EBBs. diff --git a/docs/website/sidebars.js b/docs/website/sidebars.js index f186b0754e..94a17c64d1 100644 --- a/docs/website/sidebars.js +++ b/docs/website/sidebars.js @@ -39,6 +39,8 @@ const sidebars = { 'for-developers/Ticking', 'for-developers/CivicTime', 'for-developers/AbstractProtocol', + 'for-developers/HardForks', + 'for-developers/ProtocolVersioning', 'for-developers/AddingAnEra', 'for-developers/ChainSync', 'for-developers/HardWonWisdom',