Planning for automated governance

[eddyashton]

We'd like to remove some of the burden on operators, by "automating" some actions which are currently considered "governance" actions, requiring operator involvement. The initial targets here are around reconfigurations - when a node joins or leaves, there are situations where we'd like to do that automatically with no member-authored proposals.

More concretely, we think that operators generally desire a 3-node network, but don't have any further affinity for particular node identities (ie - they're all running in the same trust environment). If they know that node A needs to be replaced (for a code upgrade, or because its container is being restarted, or...), then a replacement node B should be able to say "I am a replacement for A" as part of its join process, and automatically produce the configuration changes (writes to the node tables) that would have happened from governance actions saying "retire A, and trust B".

Discussion points:

• Do we implement this by allowing nodes to participate in governance (benefitting from existing replay protection, auditability, standardisation and constitution flexibility), or add new endpoints for each automated-governance-action? There's a risk to the semantics of existing governance, if requests might come from nodes (slight API changes to indicate these came from nodes, unexpected errors in member-table lookups). Do we just sidestep this entirely by adding node identities to member tables (with metadata clarifying that they're nodes?)
• Is it possible to opt-out of any new automation behaviour for instances which want to retain consortium control? We think most deployments will prefer higher automation, but it does change the trust semantics.
• Atomicity - can we do more atomic reconfigurations (retirement + trust in a single transaction)? Should we delay joins until we have a 3-node network, and consider the impacts on f when doing automated reconfigurations?

I'll try to write up more proposals for discussion here. Other input welcome.

[achamayou]

There's a risk to the semantics of existing governance, if requests might come from nodes (slight API changes to indicate these came from nodes, unexpected errors in member-table lookups). Do we just sidestep this entirely by adding node identities to member tables (with metadata clarifying that they're nodes?)

Another way to side-step this is to look-up issuers of governance messages in both tables, but to identify them to the functions in the governance module. A backwards-compatible and unambiguous way to do this is to turn resolve(proposal, proposerId, votes) into resolve(proposal, memberProposerId, votes, nodeProposerId). That means existing constitutions are semantically unaffected, and new constitutions can allow nodes to pass proposals.

An alternative is resolve(proposal, proposerId, votes, proposerIsNode), which would effectively update existing constitutions to give the same weight to nodes as to existing members. I think this is unlikely to be what we want.

It's worth noting that allowing nodes to vote is distinct from allowing them to make proposals, and I am not convinced it is worth adding for the purposes we are discussing here. Nodes would only ever propose reconfigurations that should pass or fail on a purely automated basis.

[eddyashton]

I've mused a little about how these options react when a the code is updated but the constitution isn't (so the constitution suddenly gets a null or nodeID value for proposerId), but I think these all result in "proposal creation probably fails", which is reasonable (and edge-case - constitution should be updated in-sync).

A pretty strong argument against adding node IDs to the member_info table (purely so they can participate in governance) is that it causes a bunch of extra KV churn. We expect nodes to turn over more frequently than members, we already require several KV writes to make them valid nodes, and we'd ultimately be duplicating a chunk of this into member_info.

I'm also tempted to make a new resolve function for this purpose, resolve_node(proposal, nodeProposerId). Then it's simpler to say "there's no voting step" (resolve_node takes no votes, result of this function is a bool determining whether we call the existing apply()). This has the minimal risk for existing proposals, because the C++ code simply determines this function doesn't exist and returns an early error.

We can call that function either conditionally in a shared endpoint (after auth we know whether the submitter is a node or member), or in a separate endpoint (making the auth requirements for each clear and distinct, requiring a little more care to make sure we do all necessary work in both paths, but also making it easier for them to diverge as needed).

[achamayou]

I like resolve_node()/resolveNode(). It does split the governance flow somewhat, and it means that node-originated proposals cannot be subject to member votes, but that does not seem to prevent anything useful. I guess the effect of this is that proposals going through this path only ever exist in Passed/Rejected states in the KV/ledger, which is fine and overall the direction we are going in.

I am less keen on splitting POST /proposals because it means introducing a new governance message type and a bunch of associated API updates that aren't obviously worth the churn.

[eddyashton]

For nodes to submit COSE signed proposals like members do (with the goal of getting the current replay protection), they need to be able to get a timestamp to put in the created-at header.

[achamayou]

Yes, and a message type as well.

[eddyashton]

We're discussing automated DR in parallel, but I don't understand the constraints enough to know how it affects this. I think that's going to be a separate process, but if it requires nodes to join each other (with an attestation check) and then vote for some action on their preferred service, will we also want replay-and-audit protections?

[achamayou]

Because the votes are stored, and the voting can only happen once, as I understand it, replay protection should be relatively simple.

[cjen1-msft]

Just to document a possible issue with using a canonical name for each node to identify who to retire.

The problem is how does a node know which node it is replacing, and if there are multiple replacements in the network, how do they decide which is the correct one to replace?

1. machine/VM level reconfiguration, moving X from M_1 (machine 1) to M_2
2. X_2 (X on M_2) starts, retires X_1
3. X_1 crashes before being notified that it has been retired, but M_1 is still active
4. X_1 restarts as a replacement for X, retiring X_2
5. M_1 is shut down, potentially without X_1 being able to notify the network

Solutions to this include

• Replacement is identity based (X_2 replaces X_1)
  • May not be sufficient information on M_2 to actually achieve this.
• When a node is retired by the network, but believes it should be valid it restarts after a timeout.
  • This would cause the above situation to eventually stabilise with X_2 as the node in the network.

[eddyashton]

I think the solution for this is to have both identity-based replacement and a "target number of nodes" in the constitution.

Before X_2 arrives, X_1 could arrive, and would be replace X ("you're both running on M_1, and X_1 is newer").

When X_2 arrives, it is a replacement for X (or X_1, if it had arrived) because it contains some piece of metadata (in its node_data), saying "I'm running on M_2, which is a replacement for M_1".

If X_1 arrives after X_2, it is ignored because we already have 3 nodes, and X_1 doesn't say it replaces anyone currently running. It has gone through the /join protocol and is listed as Pending, it hangs around here polling its proposal (potentially resubmitting it, doesn't really matter), eventually M_1 dies, and some later piece of garbage-collection removes the X_1: Pending from the KV.

[cjen1-msft]

So in this case, each node would have a one-step "I'm running on M_2 which is a replacement for M_1" relation, with the initial deployment having no predecessors.

I think that this is the most reliable way to deal with this, where we have a 2 level (always accept new joins for same machine, and partially ordered M_2 > M_1 replacments).

My remaining concern would be whether the orchestrator can maintain enough information to actually construct M_2 is a replace for M_1 (And I'm assuming that losing a step in the chain is sufficiently unlikely that we can delegate fixing that to the operator.)
However if this is all constitution driven, then additional fixes if this is insufficient should be easy enough.

[eddyashton]

whether the orchestrator can maintain enough information to actually construct M_2 is a replace for M_1

This is an interesting point. I'd assumed this was already a given, that the orchestrator has to be telling each node "You're M_1", and that the knowledge of replacement was simple to add. But I guess in the specific case where the "alpha/beta/gamma" IDs we've been discussing are machine names like M_1 and M_2, they may be automatically internally derivable (ie - from attestation, or in some other way picked up and auto-inserted into node_data), and not the concern of the orchestrator.

That's a possibility, but not a blocker.

This kind of reconfiguration is ultimately an externally-triggered process, whether by a real human or another layer of machine doesn't really matter - something external has the concept of "M_2 is replacing M_1". This external orchestrator may have to tell the service that directly (eg - another proposal or another endpoint), or they may be able to bury it in the node_data, but for the new nodes to join at all they need to tell the service somehow.

[cjen1-msft]

Just to be specific:
I am pretty sure that either it is explict or implicitly possible for a node to be told "you are M_2".
I'm not sure about the other side of that equation, "You are a replacement for M_1" since reasonable distributed schedulers could simply not even know that M_1 exists.

However I expect most schedulers use a total order of configuration operations.

[achamayou]

What's a scheduler here? All CCF write transactions (and this is one) are executed by a primary, with a total order of all transactions. Configuration operations are transactions. If the node can't get them committed, they get rolled back. There is no obstacle to expressing "You are a replacement for M_1" in a transaction.

[eddyashton]

After weighing some options I think we need to retain a 2-step join protocol, where the node first submits a /join request (containing their attestation, getting secrets in the idempotent response), and then somebody submits a proposal to transition that node from Pending to Trusted.

Trying to flatten these into a single node-created auto-join proposal has multiple problems:

• Every auto-join proposal would need to include the full attestation information, so that the initial "join" verification could be re-executed.
• The authn for the endpoint accepting this auto-join proposal would have to be very permissive, to allow requests from completely unknown identities (since the node would explicitly be unknown when they first submitted!).
• We'd have to revisit how a node retrieves secrets once it has joined, since the proposal response is not suitable (it's public, and we don't store it long enough to provide idempotent repeated responses).

So I think we retain the separate /join request and current state machine, and look simply at triggering the existing governance actions from a node. I see that as comprising 3 parts:

• Alter the governance endpoints/logic to accept proposals produced by a node rather than a member (alter-existing vs fork-new seems like a minor choice to me, and we can revisit later, but currently leaning towards alter-existing).
• Add code within a node's startup flow to create and submit its own proposals. This likely needs to be best-effort (need to co-exist with nodes and services which ignore this), but should have some retry logic. The content and format of these proposals will be determined by framework C++ code, so we need to work out what the operator hooks are (I think the existing node_data is sufficient, but TBC).
• Modify the constitution (probably new actions) to support auto-reconfiguration, based on known node IDs in node_data (target for PF, and other environments where we have stable identifiers) and perhaps a pure node-count target (interesting and more general, but not a current priority).

The first 2 of these look reasonably straightforward and well-defined, I intend to start hacking on these shortly. The third is where a lot of the current discussions are occurring, is slightly more in-flux; I'll add some follow-up thoughts below.

[eddyashton]

First thought is just an abstract-design observation:

Our previous approach to governance actions has been to try and create simple, clear actions, arguing they could be composed into more detailed actions by a governing member. So we have add_node and remove_node actions, and then envision atomic reconfiguration happening by a member adding X and removing Y in the same proposal. This is a model where the member has autonomy, they are the impetus and brain driving these decisions.

What we're discussing here removes the member entirely, and that autonomy needs to move to the service. The submitter of the proposal is a new node, they can only give information about themselves, but the decision of whether the new node should be trusted (and what that means for the network, including whether a separate node should be removed) must be made by the service they're joining. That decision is dynamic, with writes far more variable and dependent on KV state than previous constitution actions. We should aim to ensure that it remains deterministic on KV state, to be compliant with existing audit expectations.

[eddyashton]

We've had some discussions on down-detectors.

For generic reconfigurations, we need some down-detector to tell us which node should be replaced - we'd like a 3 node network, we have a 4th node in pending state: should they replace someone, and who? In many cases there are external services doing this work already that we could rely on (eg - they may have told us that one of the 3 is dying, and the top candidate to replace, Or they may tell us explicitly that 4 is a replacement for the current 2. The latter is what we expect in PF). When we rely on those external signals, we should be confident about the semantics (how reliable and timely are those indications).

We could also build an internal down-detector, based on our existing consensus heartbeats - node 4 should replace an existing node, if that node hasn't responded in some threshold. This model is more generic, and puts us in control of the semantics of that signal, but has a few critical difficulties:

• Primary-local decision, or a more complex protocol
• Bias towards the current primary looking 'live', or increased gossip traffic
• Non-KV decision, or additional KV writes to record down-detector changes
• Flow for the down-detector to trigger a reconfiguration is non-obvious. What do we do when a 4th node arrives while the service is healthy, we say "no replacement", but node 2 dies shortly after? Lots of edge cases in the semantics we need to write, rather than punting to someone else.

[achamayou]

That decision is dynamic, with writes far more variable and dependent on KV state than previous constitution actions. We should aim to ensure that it remains deterministic on KV state, to be compliant with existing audit expectations.

I am confused by this, it feels like it will still pretty much be add_node, and remove_node conditionally to checking node_data, so either a one or two-action proposal effectively. It's a bit more dynamic, but still consistent with the existing action style (transition_node_to_trusted_for_label, or even set_node_for_label).