LUKS image will not unlock in some cases

[euanmillar]

We use cryptfs to create an encrypted file to store data. For some unknown reason, the encrypted file could not be mounted on reboot in Burkina Faso and Niue development environments.

The result of this effect on production could lead to countries losing 24 hours of data.

Root cause:

Password incorrect. Niue was an old country using 1.4 provisioned manually with the DISK_ENCRYPTION_KEY written to /root/disk-encryption-key.txt.

Yarn environment:init created an ENCRYPTION_KEY github secret, because it didnt exist before due to human error when creating the github env post 1.5

After this secret was created, the Provision action overwrote the old secret in /root/disk-encryption-key.txt

The server rebooted and could not mount as the secret was incorrect.

Only because we had a password manager with the old key, could we recover.

Resolution steps

Check countries existing encryption key can decrypt a partition:

We need to ensure countries have the right disk encryption key for each of their environments in Github Secrets. The right disk encryption key must be in their Pasword Manager software too

This command checks that the secret which is in /root/disk-encryption-key.txt can be used to decrypt the disk

sudo cryptsetup luksHeaderBackup $(losetup -j /cryptfs_file_sparse.img | cut -d\: -f1) --header-backup-file /tmp/luks-header.img
export `sudo cat /root/disk-encryption-key.txt`
echo $DISK_ENCRYPTION_KEY | sudo cryptsetup -d - luksOpen /tmp/luks-header.img test --test-passphrase && echo "Passphase is correct" || echo "PASSPHRASE IS NOT CORRECT!  There is a discrepancy between the disk ENCRYPTION_KEY stored in your text file and the one in Github Secrets - or both are incorrect.  After a server reboot your data could become inaccessible.  Check your password manager for the correct ENCRYPTION_KEY for this environment. Update the password in /root/disk-encryption-key.txt & in Github Secrets for this environment.  If you have lost your password, take immediate steps to provision a new server and restore data to it."

Change disk encryption t be an opt-in procedure:

We need to ensure that in version 1.9.1, disk encryption is entirely optional, with documentation on how to implement it and the associated risks.

opencrvs/opencrvs-countryconfig#1123
opencrvs/infrastructure#156

[euanmillar]

@tumbledwyer suggesting backing up the LUKS partition when we do regular backups might be a potential stop-gap? @adskyiproger

[tumbledwyer]

@tumbledwyer suggesting backing up the LUKS partition when we do regular backups might be a potential stop-gap? @adskyiproger

I actually meant to say the LUKS header.