|
| 1 | +# Contributing to the `cardano-node` project |
| 2 | + |
| 3 | +The `cardano-node` development is primarily based on the Nix |
| 4 | +infrastructure (https://nixos.org/), which enables packaging, CI, |
| 5 | +development environments and deployments. |
| 6 | + |
| 7 | +On how to set up Nix for `cardano-node` development, please see |
| 8 | +[Building Cardano Node with |
| 9 | +nix](https://github.com/input-output-hk/cardano-node-wiki/wiki/building-the-node-using-nix). |
| 10 | + |
| 11 | +## Roles and Responsibilities |
| 12 | + |
| 13 | +We maintain a [CODEOWNERS file][CODEOWNERS] which provides |
| 14 | +information who should review a contributing PR. Note that you might |
| 15 | +need to get approvals from all code owners (even though GitHub doesn't |
| 16 | +give a way to enforce it). |
| 17 | + |
| 18 | +[CODEOWNERS]: <https://github.com/intersectmbo/cardano-node/blob/master//CODEOWNERS> |
| 19 | + |
| 20 | +## Testing |
| 21 | + |
| 22 | +`cardano-node` is essentially a container which implements several |
| 23 | +components such networking, consensus, and storage. These components |
| 24 | +have individual test coverage. The node goes through integration and |
| 25 | +release testing by Devops/QA while automated CLI tests are ongoing |
| 26 | +alongside development. |
| 27 | + |
| 28 | +Developers on `cardano-node` can [launch their own |
| 29 | +testnets](https://developers.cardano.org/docs/get-started/cardano-testnet) |
| 30 | +or [run the chairman |
| 31 | +tests](https://github.com/input-output-hk/cardano-node-wiki/wiki/running-chairman-tests) |
| 32 | +locally. |
| 33 | + |
| 34 | +## Updating dependencies |
| 35 | + |
| 36 | +### ... from Hackage |
| 37 | + |
| 38 | +Updating package dependencies from Hackage should work like normal in a |
| 39 | +Haskell project. The most important thing to note is that we pin the |
| 40 | +`index-state` of the Hackage package index in `cabal.project`. This |
| 41 | +means that cabal will always see Hackage “as if” it was that time, |
| 42 | +ensuring reproducibility. But it also means that if you need a package |
| 43 | +version that was released *after* that time, you need to bump the |
| 44 | +`index-state` (and to run `cabal update` locally). |
| 45 | + |
| 46 | +Because of how we use Nix to manage our Haskell build, whenever you do this you |
| 47 | +will also need to pull in the Nix equivalent of the newer `index-state`. You can |
| 48 | +do this by running `nix flake update hackageNix`. |
| 49 | + |
| 50 | +### ... from the Cardano package repository |
| 51 | + |
| 52 | +Many Cardano packages are not on Hackage and are instead in the [Cardano |
| 53 | +package |
| 54 | +repository](https://chap.intersectmbo.org). Getting new packages from |
| 55 | +there works much like getting them from Hackage. The differences are |
| 56 | +that it has an independent `index-state`, and that there is a different |
| 57 | +Nix command you need to run afterwards: |
| 58 | +`nix flake update CHaP`. |
| 59 | + |
| 60 | +### Using unreleased versions of dependencies |
| 61 | + |
| 62 | +Sometimes we need to use an unreleased version of one of our dependencies, |
| 63 | +either to fix an issue in a package that is not under our control, or to |
| 64 | +experiment with a pre-release version of one of our own packages. You can use a |
| 65 | +[`source-repository-package`](https://cabal.readthedocs.io/en/latest/cabal-project-description-file.html#taking-a-dependency-from-a-source-code-repository) |
| 66 | +stanza to pull in the unreleased version. Try only to do this for a short time, |
| 67 | +as it does not play very well with tooling, and will interfere with the ability |
| 68 | +to release the node itself. |
| 69 | + |
| 70 | +For packages that we do not control, we can end up in a situation where we have |
| 71 | +a fork that looks like it will be long-lived or permanent (e.g. the maintainer |
| 72 | +is unresponsive, or the change has been merged but not released). In that case, |
| 73 | +[instructions](https://github.com/IntersectMBO/cardano-haskell-packages?tab=readme-ov-file#how-to-add-a-patched-versions-of-a-hackage-package) |
| 74 | +are provided for releasing a patched version to CHaP, which allows us to remove |
| 75 | +the `source-repository-package` stanza. |
| 76 | + |
| 77 | +## Releasing a version of the node |
| 78 | + |
| 79 | +(There is much more to say here, this is just a small fragment) |
| 80 | + |
| 81 | +### ... to the Cardano package repository |
| 82 | + |
| 83 | +When releasing a new version of the node, it and the other packages in |
| 84 | +this repository should be released to the [Cardano package |
| 85 | +repository](https://github.com/input-output-hk/cardano-haskell-packages). |
| 86 | +See the README for instructions, including a script to automate most of |
| 87 | +the process. Please note that libraries need bounds on the version of |
| 88 | +their dependencies to avoid bitrot and be effectively reusable. |
| 89 | + |
| 90 | +## Workbench: running a cluster |
| 91 | + |
| 92 | +You can quickly spin up a local cluster (on Linux and Darwin), based on |
| 93 | +any of a wide variety of configurations, and put it under a transaction |
| 94 | +generation workload -- using the `workbench` environment: |
| 95 | + |
| 96 | +1. Optional: choose a workbench profile: |
| 97 | + - `default` stands for a light-state, 6-node cluster, under |
| 98 | + transaction saturation workload, ~30min runtime. |
| 99 | + - `ci-test-hydra` is the profile run in the node CI -- very light, |
| 100 | + just two nodes, Plutus transaction worklooad, =\< 3min runtime. |
| 101 | + - `devops` is an unloaded profile (no transaction workload) with |
| 102 | + short slots -- `0.2` sec. |
| 103 | + - ...and many more -- which can be: - listed by name with |
| 104 | + `make ps` - inspected with e.g. |
| 105 | + `cabal run cardano-profile -- by-name-pretty default-coay` |
| 106 | + (default profile with Conway era suffix) |
| 107 | + |
| 108 | +2. Optional: select mode of operation, by optionally providing a suffix: |
| 109 | + - no suffix (default) -- just enter the workbench shell, allowing |
| 110 | + you to run `start-cluster` at any time. Binaries will be built |
| 111 | + locally, by `cabal`. |
| 112 | + - `-nix` suffix -- same as before, but all binaries will be built by |
| 113 | + Nix or downloaded from Hydra CI cache directly. |
| 114 | + - `-prof` and `-profnix` suffixes -- same as both before, but all |
| 115 | + binaries will be built such that GHC profiling is enabled. |
| 116 | + - ...there are other modes as per |
| 117 | + [lib.mk](https://github.com/intersectmbo/cardano-node/tree/master/lib.mk#L34-L44) |
| 118 | + |
| 119 | +3. Enter the workbench shell for the chosen profile & mode: |
| 120 | + `make <PROFILE-NAME>` or `make <PROFILE-NAME>-<SUFFIX>` (when there |
| 121 | + is a suffix). |
| 122 | + |
| 123 | +4. Optional: start cluster: |
| 124 | + Depending on the chosen mode, your cluster might already start, or |
| 125 | + you are expected to start it yourself, using `start-cluster`. |
| 126 | + |
| 127 | +The workbench services are available only inside the workbench shell. |
| 128 | + |
| 129 | +### Using Cabal |
| 130 | + |
| 131 | +By default, all binaries originating in the `cardano-node` repository |
| 132 | +are available to `cabal build` and `cabal run`, unless the workbench was |
| 133 | +entered using one of the pure `*nix` modes. Note that in all cases, the |
| 134 | +dependencies for the workbench are supplied through Nix and have been |
| 135 | +built/tested on CI. |
| 136 | + |
| 137 | +### **Dependency localisation** -or- *Cabal&Nix for painless cross-repository work* |
| 138 | + |
| 139 | +The Cabal workflow described above only extends to the repository-local |
| 140 | +packages. Therefore, ordinarily, to work on `cardano-node` dependencies |
| 141 | +in the context of the node itself, one needs to go through an expensive |
| 142 | +multi-step process -- with committing, pushing and re-pinning of the |
| 143 | +dependency changes. |
| 144 | + |
| 145 | +The **dependency localisation** workflow allows us to pick a subset of |
| 146 | +leaf dependencies of the `cardano-node` repository, and declare them |
| 147 | +*local* -- so they can be iterated upon using the `cabal build` / |
| 148 | +`cabal run` of `cardano-node` itself. This cuts development iteration |
| 149 | +time dramatically and enables effective cross-repo development of the |
| 150 | +full stack of Cardano packages. |
| 151 | + |
| 152 | +Without further ado (**NOTE**: *the order of steps is important!*): |
| 153 | + |
| 154 | +1. Ensure that your `cardano-node` checkout is clean, with no local |
| 155 | + modifications. Also, ensure that you start outside the node's Nix |
| 156 | + shell. |
| 157 | + |
| 158 | +2. Check out the repository with the dependencies, *beside* the `cardano-node` checkout. You have to check out the git revision of the dependency used by your `cardano-node` checkout -- as listed in `cardano-node/cabal.project`. |
| 159 | + - we'll assume the `ouroboros-consensus` repository |
| 160 | + |
| 161 | + - ..so a certain parent directory will include checkouts of both |
| 162 | + `ouroboros-consensus` and `cardano-node`, at the same level |
| 163 | + |
| 164 | + - ..and the git revision checked out in `ouroboros-consensus` will |
| 165 | + match the version of the `ouroboros-consensus` packages used |
| 166 | + currently |
| 167 | + |
| 168 | + - Extra point #1: you can localise/check out several dependency |
| 169 | + repositories |
| 170 | + |
| 171 | + - Extra point #2: for the dependencies that are not listed in `cabal.project` of the node -- how do you determine the version to check out? You can ask the workbench shell: |
| 172 | + 1. Temporarily enter the workbench shell |
| 173 | + 2. Look for the package version in `ghc-pkg list` |
| 174 | + 3. Use that version to determine the git revision of the |
| 175 | + dependency's repository (using a tag or some special knowledge |
| 176 | + about the version management of said dependency). |
| 177 | + |
| 178 | +3. Enter the workbench shell, as per instructions in previous sections |
| 179 | + -- or just a plain Nix shell. |
| 180 | + |
| 181 | +4. Ensure you can build `cardano-node` with Cabal: |
| 182 | + `cabal build exe:cardano-node`. If you can't something else is |
| 183 | + wrong. |
| 184 | + |
| 185 | +5. Determine the *leaf dependency set* you will have to work on. The *leaf dependency set* is defined to include the target package you want to modify, and its reverse dependencies -- that is, packages that depend on it (inside the dependency repository). |
| 186 | + - let's assume, for example, that you want to modify |
| 187 | + `ouroboros-consensus-protocol` |
| 188 | + |
| 189 | + - `ouroboros-consensus-protocol` is not a leaf dependency in itself, |
| 190 | + since `ouroboros-consensus-cardano` (of the same |
| 191 | + `ouroboros-consensus` repository) depends on it -- so the *leaf |
| 192 | + dependency set* will include both of them. |
| 193 | + |
| 194 | + - you might find out that you have to include a significant fraction |
| 195 | + of packages in `ouroboros-consensus` into this *leaf dependency set* |
| 196 | + -- do not despair. |
| 197 | + |
| 198 | + - if the *leaf dependency set* is hard to determine, you can use `cabal-plan` -- which is included in the workbench shell (which you, therefore, have to enter temporarily): |
| 199 | + ``` console |
| 200 | + [nix-shell:~/cardano-node]$ cabal-plan dot-png --revdep ouroboros-consensus-shelley |
| 201 | + ``` |
| 202 | + |
| 203 | + This command will produce a HUGE `deps.png` file, which will |
| 204 | + contain the entire chart of the project dependencies. The |
| 205 | + important part to look for will be the subset of packages |
| 206 | + highlighted in red -- those, which belong to the |
| 207 | + `ouroboros-consensus` repository. This will be the full *leaf |
| 208 | + dependency set*. |
| 209 | + |
| 210 | +6. Edit the `cardano-node/cabal.project` as follows: |
| 211 | + - for the *leaf dependency set* in the very beginning of the `cabal.project`, add their relative paths to the `packages:` section, e.g.: |
| 212 | + ``` console |
| 213 | + packages: |
| 214 | + ... |
| 215 | + trace-resources |
| 216 | + trace-forward |
| 217 | + ../ouroboros-consensus/ouroboros-consensus-protocol |
| 218 | + ../ouroboros-consensus/ouroboros-consensus-cardano |
| 219 | + ``` |
| 220 | + |
| 221 | +7. The two packages have now become **local** -- when you try |
| 222 | + `cabal build exe:cardano-node` now, you'll see that Cabal starts to |
| 223 | + build these dependencies you just localised. Hacking time! |
| 224 | + |
| 225 | +### Hoogle |
| 226 | + |
| 227 | +The workbench shell provides `hoogle`, with a local database for the |
| 228 | +full **non-local** set of package dependencies: |
| 229 | + |
| 230 | +``` console |
| 231 | +[nix-shell:~/cardano-node]$ hoogle search TxId |
| 232 | +Byron.Spec.Ledger.UTxO newtype TxId |
| 233 | +Byron.Spec.Ledger.UTxO TxId :: Hash -> TxId |
| 234 | +Cardano.Chain.UTxO type TxId = Hash Tx |
| 235 | +Cardano.Ledger.TxIn newtype TxId crypto |
| 236 | +Cardano.Ledger.TxIn TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto |
| 237 | +Cardano.Ledger.Shelley.API.Types newtype TxId crypto |
| 238 | +Cardano.Ledger.Shelley.API.Types TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto |
| 239 | +Cardano.Ledger.Shelley.Tx newtype TxId crypto |
| 240 | +Cardano.Ledger.Shelley.Tx TxId :: SafeHash crypto EraIndependentTxBody -> TxId crypto |
| 241 | +Ouroboros.Consensus.HardFork.Combinator data family TxId tx :: Type |
| 242 | +-- plus more results not shown, pass --count=20 to see more |
| 243 | +``` |
0 commit comments