storing and loading cluster persistent state

[bertschinger]

Some cluster state needs to be stored persistently so that the manager process can recall it after restarting.

This includes:

• a list of hosts, resource groups, and/or resources that are unmanaged
• a list of hosts that were fenced

Whenever this state changes, for example

• because the admin un-managed or re-managed a host
• because a host was fenced
• because an admin cleared a host fenced flag, making the host eligible for fencing again

the state file should be updated.

The state file should have a default location in /etc/halo but a custom location can be provided with an environment variable and/or command line argument.

I don't see any reason to require a fancy database format; I think a simple text file in a format like yaml or toml should suffice.

[bertschinger]

Design ideas:

• add an already_fenced (or similar name) field to struct Host
• add a store_state() method to struct Cluster that iterates through the hosts, getting a list of the fenced ones, and writes out the state to a file
• add a load_state() method to struct Cluster that loads up the stored state and call this during the constructor of Cluster
• the file format could be very simple:

fenced nodes: <host names in nodeset notation>

[tbautis]

It would nice to be able to add comments when unmanaging a resource so that other admins can be aware of why a resource was unmanaged (i.e., displaying the comments upon halo status). This will require storage of state for these comments, and so the storage format may need to be a "real" format (e.g., YAML, TOML, etc.).

[bertschinger]

After thinking about this more, I think this state file should be an append-only log, so that it stores a history of all state changes, rather than a snapshot of the current state.

It could use a CSV format. A simple example of what it might look like:

# timestamp,event,host_or_resource_id,comment
100,fence,oss01,connection timed out
103,unmanage,mdt07,admin disabling resource because of zpool issues
109,reset,oss01,admin rebooted
117,manage,mdt07,

The downside of this method is that when the manager starts, it will have to read the entire state history and "replay" it to arrive at the most recent cluster state. If the state log grows excessively large, this could take a while. I don't think it's likely to grow large enough to pose a significant issue, but if it does, the log could be trimmed / rotated.

[tbautis]

Per in-person discussion, we have decided to use TSV (tab-separated values) format for the state file.