Planning for automated governance

[eddyashton]

We'd like to remove some of the burden on operators, by "automating" some actions which are currently considered "governance" actions, requiring operator involvement. The initial targets here are around reconfigurations - when a node joins or leaves, there are situations where we'd like to do that automatically with no member-authored proposals.

More concretely, we think that operators generally desire a 3-node network, but don't have any further affinity for particular node identities (ie - they're all running in the same trust environment). If they know that node A needs to be replaced (for a code upgrade, or because its container is being restarted, or...), then a replacement node B should be able to say "I am a replacement for A" as part of its join process, and automatically produce the configuration changes (writes to the node tables) that would have happened from governance actions saying "retire A, and trust B".

Discussion points:

• Do we implement this by allowing nodes to participate in governance (benefitting from existing replay protection, auditability, standardisation and constitution flexibility), or add new endpoints for each automated-governance-action? There's a risk to the semantics of existing governance, if requests might come from nodes (slight API changes to indicate these came from nodes, unexpected errors in member-table lookups). Do we just sidestep this entirely by adding node identities to member tables (with metadata clarifying that they're nodes?)
• Is it possible to opt-out of any new automation behaviour for instances which want to retain consortium control? We think most deployments will prefer higher automation, but it does change the trust semantics.
• Atomicity - can we do more atomic reconfigurations (retirement + trust in a single transaction)? Should we delay joins until we have a 3-node network, and consider the impacts on f when doing automated reconfigurations?

I'll try to write up more proposals for discussion here. Other input welcome.

[achamayou]

There's a risk to the semantics of existing governance, if requests might come from nodes (slight API changes to indicate these came from nodes, unexpected errors in member-table lookups). Do we just sidestep this entirely by adding node identities to member tables (with metadata clarifying that they're nodes?)

Another way to side-step this is to look-up issuers of governance messages in both tables, but to identify them to the functions in the governance module. A backwards-compatible and unambiguous way to do this is to turn resolve(proposal, proposerId, votes) into resolve(proposal, memberProposerId, votes, nodeProposerId). That means existing constitutions are semantically unaffected, and new constitutions can allow nodes to pass proposals.

An alternative is resolve(proposal, proposerId, votes, proposerIsNode), which would effectively update existing constitutions to give the same weight to nodes as to existing members. I think this is unlikely to be what we want.

It's worth noting that allowing nodes to vote is distinct from allowing them to make proposals, and I am not convinced it is worth adding for the purposes we are discussing here. Nodes would only ever propose reconfigurations that should pass or fail on a purely automated basis.

[eddyashton]

I've mused a little about how these options react when a the code is updated but the constitution isn't (so the constitution suddenly gets a null or nodeID value for proposerId), but I think these all result in "proposal creation probably fails", which is reasonable (and edge-case - constitution should be updated in-sync).

A pretty strong argument against adding node IDs to the member_info table (purely so they can participate in governance) is that it causes a bunch of extra KV churn. We expect nodes to turn over more frequently than members, we already require several KV writes to make them valid nodes, and we'd ultimately be duplicating a chunk of this into member_info.

I'm also tempted to make a new resolve function for this purpose, resolve_node(proposal, nodeProposerId). Then it's simpler to say "there's no voting step" (resolve_node takes no votes, result of this function is a bool determining whether we call the existing apply()). This has the minimal risk for existing proposals, because the C++ code simply determines this function doesn't exist and returns an early error.

We can call that function either conditionally in a shared endpoint (after auth we know whether the submitter is a node or member), or in a separate endpoint (making the auth requirements for each clear and distinct, requiring a little more care to make sure we do all necessary work in both paths, but also making it easier for them to diverge as needed).

[achamayou]

I like resolve_node()/resolveNode(). It does split the governance flow somewhat, and it means that node-originated proposals cannot be subject to member votes, but that does not seem to prevent anything useful. I guess the effect of this is that proposals going through this path only ever exist in Passed/Rejected states in the KV/ledger, which is fine and overall the direction we are going in.

I am less keen on splitting POST /proposals because it means introducing a new governance message type and a bunch of associated API updates that aren't obviously worth the churn.

[eddyashton]

For nodes to submit COSE signed proposals like members do (with the goal of getting the current replay protection), they need to be able to get a timestamp to put in the created-at header.

[achamayou]

Yes, and a message type as well.

[eddyashton]

We're discussing automated DR in parallel, but I don't understand the constraints enough to know how it affects this. I think that's going to be a separate process, but if it requires nodes to join each other (with an attestation check) and then vote for some action on their preferred service, will we also want replay-and-audit protections?

[achamayou]

Because the votes are stored, and the voting can only happen once, as I understand it, replay protection should be relatively simple.