Fault tolerance

[basilhendroff]

If the step-ca server were to fail, what are the implications for internal PKI? What steps can be implemented to mitigate this?

[LecrisUT]

You would probably have to be more specific on how/where the step-ca server is failing, but in general, if the server can be brought back up with the same root certificates and ca.json secrets, everything would be back to normal.

As long as you have access to the root certificate, you can reproduce almost everything in step-ca using openssl, so on that front, there is nothing much to worry about. You would also probably want to backup the root certificate offline, or even completely isolate it and use a manually generated long-term intermediate certificate instead of the root one on the machines.

Some things that would require reconfiguration are if the ca.json secrets are damaged. However these can be regenerated with step ca init --root root_ca.crt --key root_ca.key and the appropriate --provisioner, or step ca provisioner. Generally these should not affect the user experience or dependent applications in any way, just the admin and some downtime.

The minimum things you should backup are the root certificate and key, the ca.json containing all your specific configurations for that server, and the template files containing all special certificates issued ssh, S/MIME, etc.

If there are any specific parts you worry that can break, or you would want a more "no-jargon" explanation, let me know.

Edit: Oh, I forgot about CRLs. These should also be backed-up to make sure the revoked (compromised) certificates are no longer trusted, but the location is dependent on the backend. Also #731 is needed for the CRLs to be usable.

[basilhendroff]

@LecrisUT Thanks!

You would probably have to be more specific on how/where the step-ca server is failing

More of a general question at this stage. It seems to me that to recover from a catastrophic failure of the step-ca server, at a minimum I need a backup of everything below $(step path) as well as the password to decrypt the intermediate certificate private key. This can be restored to a new step-ca server.

During the downtime interval, given the short lifetime of (leaf?) certificates, what symptoms can I expect to see on the local network?

[LecrisUT]

Everything in step path are only user configurations, and they can be regenerated with step ca bootstrap. If you run the step-ca server locally, than it becomes more complicated, but locating and saving the files mentioned above (and the password) would be the minimum requirement. If you have root privilege, you should be installing step-ca as a service, usually done by the package manager, but the short version:

• creating a step or step-ca user
• installing step-ca (and step if you want) to /usr/bin, or equivalent. (owned by root and with chmod 755)
• copying all of the configurations to /etc/step and securing it for step or step-ca user (chown -R and chmod 700 /etc/step)
• creating the appropriate systemd service

During the downtime interval, given the short lifetime of (leaf?) certificates, what symptoms can I expect to see on the local network?

If it's less than 1 hour downtime you will likely have no problems at all. Otherwise most of the issues will appear in the automated renewed certificates, due to the certificate being expired, and it's very specific to the script used to automate. Simply regenerating them with step ca certificate will get you back on track, but you will have to save the original instruction used to create them, because I don't think they can be copied from the certificate yet.

As for the symptoms and how to monitor them:

• At minimum check step ca health for the server being down or inaccessible. Most healthcheck apps should be easily configurable to keep track of it
• Check for the provisioners Oauth, ssh, etc. Be careful that if they do not start prior to step-ca, step-ca will not start (FR: Allow startup with unreacheable provisioner #589 might have been fixed, but I haven't checked in in a while)
• Check that the leaf certificates have not been expired in the meantime. For user certificates like ssh, that's no issue since they are created from scratch each time you login, but for the automated ones I mentioned before, you have to implement your own health-check. step certificate needs-renewal (reference ExecCondition in [email protected]) should be able to catch those errors.

[basilhendroff]

That pretty much covers all my concerns. Thank you!