cardano-testnet: choose a design to support user-provided files

[smelc]

Current situation

Right now in cardano-testnet, we offer the user to pass its own:

1. configuration file
2. genesis files

And we also propose not to pass any of these things:

cardano-node/cardano-testnet/src/Testnet/Start/Cardano.hs

Line 125 in 36161c9

getDefaultGenesisBatch

Then, depending on whether the user passed its own file or not (we refer to the former case as using "a custom file"), we:

1. generate a node configuration file if the user did not specify one
2. generate genesis files if the user did not specify custom ones (actually we overwrite the ones generated by create-testnet-data).

Problem

There's a design decision to be made when using files passed by the user, because there is a tension between avoiding confusion and providing more service (at the risk of some confusion).

Let me list the different scenarios. I am here ignoring the case where cardano-testnet generates all files (which is the case of most tests in cardano-testnet-test), because we're concerned with the scenario where the user wants some customization.

1. We avoid confusion at all costs and ask the user to provide the node configuration file and all genesis files. In terms of code, this would consist of merging those two arguments into one being data UserFiles = Nothing | Just NodeConfigFile and reading the genesis files from the corresponding fields in the node configuration file. In this setup, ideally, the sole purpose of createSPOGenesisAndFiles woul dbe to generate keys with create-testnet-data (but the genesis files created by create-testnet-data would be entirely discarded).
2. We keep being able to supply a node configuration separately from the genesis files. This is the current state: we generate a node configuration file if the user doesn't pass one, and the same for genesis files. The genesis files (shelley, alonzo, conway) should either all be supplied or all generated. As soon as you want to pass one custom genesis file, all generation-features provided by cardano-testnet for this genesis file (e.g. total supply in Shelley genesis file, dreps in Conway genesis file) are lost. In this scenario, we need to track that there is no conflict between fields of the node configuration file and the provided genesis files (either the node configuration field is absent, or it points to the genesis file provided by the user).
3. Like 2. but we have the possibility of specifying genesis files individually. So you can for example have the Shelley genesis file generated, while passing a custom Conway file. This was the initial state of cardano-testnet: Use user-specified genesis files (if any) instead of the one generated by create-testnet-data #6132, for example see this commit. There is a risk of confusion, because if you pass a custom genesis file, you lose the generation capabilities for this file. So, say, you want to pass a custom Shelley genesis file, then you'll lose that cardano-testnet fills it with the total supply.
4. Like 3., but we expose more flags to tune the generation behavior, so that users have less often the need to pass custom genesis files. For example right now, we don't expose create-testnet-data's option to create committee members at the cardano-testnet level (look how it is neither in the fields of CardanoTestnetOptions nor in the fields of GenesisOptions. This works well with create-testnet-data features that it takes templates as input and modifies them (see e.g. here for the committee members)

Opinion and UX

In terms of maintenance, option 1. above is simpler for the API/CLI team (there are less conflicts to check), and it is also the simplest design to convey to users. However, I (@smelc) believe we're not providing much value to the user if we do that. If, as soon as cardano-testnet's defaults doesn't suit you, you are tasked with providing the node configuration file and the three genesis files; well that's not very nice: we leave the user on his/her own.

To a lesser extent, the same applies to option 2. As soon as the user wants one custom genesis configuration file, he is tasked with coming up with three of them, losing much of our generation support. This is what happened to @sourabhxyz here and from his comment, this was surprising.

In terms of maintenance, option 3. is roughly similar to option 2. while providing the user with more fine-grained control. I also think that having one origin per genesis file is a more obvious API. We don't have to explain that passing one custom genesis file means having to pass the UserProvidedOrigin origin for the entire batch (for example I think this wasn't to obvious to @sourabhxyz in his initial reaction to this design). This option does require more checks between the node configuration file and the custom genesis files.

Option 4. is probably the nicest one to the user, but it requires that we streamline the design of CardanoTestnetOptions and GenesisOptions (for example, right now, why is the options for dreps in CardanoTestnetOptions? whereas it affects the Conway genesis). We would also need to explain/mention those options in cardano-testnet's main entry point (e.g. cardanoTestnet right now). With this option, we acknowledge that it is inevitable to have some confusion when providing a good service to the user. In my opinion, this possible source of confusion is acceptable, because users that are going to call cardanoTestnet programmatically (or call the cardano-testnet executable) are anyway power users and call deal with understanding diffs in genesis files.

[carbolymer]

I agree option number 4 is the nicest, but we would have to gather requirements beforehand, design nice interface for it and moreover it requires much more effort than other options. It would be nice to measure if there would be enough interest in such feature.

I think I'm leaning towards option number 3 if it's not too much development effort over options 2 or 1. It gives quite granular control to the user without making the feature overly complex.

3. Like 2. but we have the possibility of specifying genesis files individually. So you can for example have the Shelley genesis file generated, while passing a custom Conway file. This was the initial state of cardano-testnet: Use user-specified genesis files (if any) instead of the one generated by create-testnet-data #6132

I haven't followed development of the solution in #6132 closely, why was that idea dropped?

[smelc]

I haven't followed development of the solution in #6132 closely, why was that idea dropped?

@carbolymer> @Jimbo4350 preferred option 2. but I think he and I underestimated the consequences.

[smelc]

I think I'm leaning towards option number 3 if it's not too much development effort over options 2 or 1. It gives quite granular control to the user without making the feature overly complex.

@carbolymer> Yes me too, and once we implement it, I can probably unblock #6103 (which is blocked on the new test of the standalone cardano-testnet executable failing with Condition not met by deadline: Chain not extended, because the shelley genesis is not customized anymore I presume 🙃). Then we can do the first cardano-testnet release.

After that, ultimately we can do a version between option 3. and option 4. above where we clean up the distinction between CardanoTestnetOptions and GenesisOptions without too much effort: we solely add flags for controlling the number of dreps and the number of committee members (this is easy, because those have dedicated flags in create-testnet-data, so no need to fumble with changes to genesis files in cardano-testnet itself).

cc @Jimbo4350

[smelc]

I sketched version 3. here:

cardano-node/cardano-testnet/src/Testnet/Start/Cardano.hs

Line 158 in 3ad402c

cardanoTestnet :: ()

Genesis files can be provided individually and the node configuration file can be provided too. For both the node configuration file and the genesis file, I use the simple UserProvidedData type to carry the value (if any, it's basically similar to Maybe).

Handling of the node configuration file is done in Cardano.hs itself while handling of the genesis files is confined to Configuration.hs. It works as follows:

• If the user provided a genesis file, no template is passed to create-testnet-data and we simply overwrite create-testnet-data's default genesis with the one provided by the user: here.
• If the user did not provide a genesis file, we generate a default one and we pass it as a template to create-testnet-data here, so that create-testnet-data can make its amendments to it.

What I didn't do yet is to check the consistency between the node configuration file and the provided (or not) genesis files: if the node configuration file specifies paths to genesis files, we need to check those genesis files are the same as the one passed as values to createTestnet. If the node configuration file doesn't specify paths, but the user passed custom genesis files, we need to augment the node configuration file to reflect that. This will happen in Cardano.hs, at the start of createTestnet.

Also note that, in the end, we probably want cardanoTestnet to take paths to genesis files, instead of the Haskell value corresponding to the file on disk; but because we used to use Haskell values, I sticked to it for now, to be able to show this PoC faster. This can be changed later, it's mostly noise.

[Jimbo4350]

Everyone is underestimating the number of edge cases that will be generated by allowing the mixing of our defaults with users custom genesis files (especially a custom Shelley genesis). Try asking the ledger team what ranges of values will produce a network that chain extends to begin to understand the potential can of worms being opened. This will be further compounded by having to be sure we are overwriting things correctly which makes debugging even harder in an already complicated system.

We can have our cake and eat it too. We can have an option to dump the cardano-testnet defaults and then the user can specify those files and tweak them to their heart's content. This way we provide a configuration that is guaranteed to work and YOYO if you decide to dump the files and tweak them. I don't see the need to complicate our code further to handle this use case.

Regarding the SPO related file creation we should maintain the current behaviour. I.E The generation of the SPO related files regardless of whether we use the node configuration files or user specified node configuration files. If users want something different here we can address it as it comes up.

[smelc]

Everyone is underestimating the number of edge cases that will be generated by allowing the mixing of our defaults with users custom genesis files

Note that I'm not advocating for coming up with genesis files from scratch here 🙂 What I'm advocating for is being to mix passing a custom genesis file for a specific era (say Conway, as @sourabhxyz is doing in #6130) and letting files from eras being generated. This is a behavior that is more convenient for users, and it allows them to take one of our own default genesis file and just slightly tweak it.

We can have an option to dump the cardano-testnet defaults and then the user can specify those files and tweak them to their heart's content

Amen, we can do that, and it's orthogonal to the decision being taken in this issue.

Regarding the SPO related file creation we should maintain the current behaviour. I.E The generation of the SPO related files regardless of whether we use the node configuration files or user specified node configuration files. If users want something different here we can address it as it comes up

Agreed too 👍

[carbolymer]

Everyone is underestimating the number of edge cases that will be generated by allowing the mixing of our defaults with users custom genesis files (especially a custom Shelley genesis).

I think that the number of edge cases does not matter here - we shouldn't handle them. If we give more power to the user, by allowing him to use only one genesis file, the user is basically on their own. We shouldn't care if the user provides invalid configuration really. The job of cardano-testnet configuration generation should be providing only working defaults.

There have to be clear warnings that providing custom configuration in any form may result in broken network though.

[Jimbo4350]

@carbolymer this is how we are handling custom configurations.

We can have our cake and eat it too. We can have an option to dump the cardano-testnet defaults and then the user can specify those files and tweak them to their heart's content. This way we provide a configuration that is guaranteed to work and YOYO if you decide to dump the files and tweak them. I don't see the need to complicate our code further to handle this use case.

Users are going to complain to us when our defaults don't play nicely with their configurations and that is what I want to avoid. They will need to explicitly dump our defaults and then tweak them.

[smelc]

Option 1. chosen here and implemented by #6148.