Document access recovery procedures

[KellerFuchs]

As @AlissaSquared pointed out in #2, we need to document how to regain access to the various services we use.

That includes:

• Atlantic.net:
  • the Atlantic account itself;
  • the shell boxes;
• DigitalOcean:
  • account;
  • service boxes;
• Github organisation;
• GlobalSign account.

[KellerFuchs]

For DO service boxes, the solution is simple: nuke and redeploy ... right?

[RyanSquared]

For the service boxes, we should have some format to manage authorized keys. Nuke and redeploy is only an option if the service doesn't have static data (pae.st does, so nuke & pave is a last-effort option).

[KellerFuchs]

@ChickenNuggers Good point regarding “static” data.
Otherwise, the SSH keys for the CoreOS boxes are already managed in Ansible; this is not to deal with the case “one admin lost access to some service” (we can already deal with that, and it is to some extent documented by New_admin.md). The goal is to document how to recover from “we collectively lost access to some server/service/whatever”.

[RyanSquared]

For the most part, users can re-produce the steps listed in New_admin for positions that are missing. However, there's a few issues that I've encountered:

Unfortunately with our current setup, I honestly don't think recovering any boxes on DigitalOcean without pulling in some DigitalOcean staff is possible for a few key points:

• Password login for CoreOS is disabled
• You'd have to boot into single-user mode (if possible) to gain access to authorized_keys.d
• DigitalOcean doesn't "know" how to use authorized_keys.d properly, so they just don't offer the ability to reset SSH keys.

I believe we'll find similar issues with Atlantic.net.

As for GlobalSign, I hope that won't be an issue for much longer once we get Let's Encrypt certificate registration automated. For GitHub, because all admins each have an account (and can be considered a "mesh"), it would take a major catastrophic failure to have to recover the organization. Account recovery for individual people will be done based on their email and managed by the user themself.