Add support for base64:… arg for unlock

[AliSoftware]

Why?

To make it easier to unlock a repository locally given the base64 of the key (which is how the key would likely be shared between coworkers, e.g. this is how it's shared in our internal Secret Store), this adds support for git conceal unlock "base64:…" syntax.

How?

Similar to the git conceal unlock env:<VARNAME> syntax to read the base64-encoded key from an env var (ideal for CI), this git conceal unlock "base64:<base64key> syntax is most suitable for developers that need to unlock their local working copy from the base64 key they'd retrive from our Secret Store.

Without that change, providing the base64 key from the Secret Store to git conceal required to use something like echo "<base64key>" | base64 -d | git conceal unlock -, which worked but was not super natural. Another possible way was to export GIT_CONCEAL_KEY="<base64key>" then git conceal unlock env:GIT_CONCEAL_KEY, but that required setting a one-off env var for little benefit. With that new syntax, it'll be easier to unlock a repo from a base64-encoded key like the ones shared in Secret Store, making the onboarding instructions easier.

Testing

• Run cargo build --release from this PR's head branch
• Checkout the head branch of one of the PRs of a repo in which I've already prepared the migration to git-conceal, like of [AINFRA-1533] Adopt git-conceal in this repo woocommerce/woocommerce-android#14979 or [AINFRA-1539] [Internal] Migrate from configure_apply to git-conceal pocket-casts-android#4841, so you have a repo on which to test that command on
• Search "git-conceal" in the Secret Store and copy the base64 key for the corresponding repo
• From the working copy of the repo you picked for testing, run:

$ git conceal unlock "base64:$(pbpaste)"

(or alternatively, paste the key manually in your terminal in that command, i.e. git conceal unlock "base64:<paste key here>")

[iangmaia]

I imagine this isn't caused by this PR, but I've tried to run it in Automattic/pocket-casts-android#4841 and deliberately used a wrong key, just for testing:

git-conceal unlock "base64:c3VwcG9zZWRseS15b3VyLWJpbmFyeS1zZWNyZXRrZXk=" 

Then I got:

Re-checking out encrypted files...
  Will re-checkout: app/google-services.json
  Will re-checkout: automotive/google-services.json
  Will re-checkout: firebase.secrets.json
  Will re-checkout: google-upload-credentials.json
  Will re-checkout: release.keystore
  Will re-checkout: secret.properties
  Will re-checkout: sentry.properties
  Will re-checkout: wear/google-services.json
Error: Failed to re-checkout encrypted files

Caused by:
    git checkout HEAD -- <files> failed: exit status: 128
    stderr: Error: Failed to decrypt data
    
    Caused by:
        HMAC verification failed - the decryption key may be incorrect, or the file may have been tampered with. Please verify you're using the correct key for this repository.
    error: external filter '"(...)git-conceal" filter smudge' failed 1
    error: external filter '"(...)git-conceal" filter smudge' failed
    fatal: app/google-services.json: smudge filter git-conceal failed

Which led to some changed files:

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	deleted:    app/google-services.json
	deleted:    automotive/google-services.json
	deleted:    firebase.secrets.json
	deleted:    google-upload-credentials.json
	deleted:    release.keystore
	deleted:    secret.properties
	deleted:    sentry.properties
	deleted:    wear/google-services.json

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	automotive/google-services.json
	firebase.secrets.json
	google-upload-credentials.json
	release.keystore
	secret.properties
	sentry.properties
	wear/google-services.json

My first reflex was running git reset, which also failed:

git reset --hard
Error: Failed to decrypt data

Caused by:
    HMAC verification failed - the decryption key may be incorrect, or the file may have been tampered with. Please verify you're using the correct key for this repository.
error: external filter '"(...)git-conceal" filter smudge' failed 1
error: external filter '"(...)git-conceal" filter smudge' failed
fatal: app/google-services.json: smudge filter git-conceal failed

git-conceal lock failed and then git-conceal lock --force restored the repo to its locked state, and unlocking with the right key worked again.

[AliSoftware]

Yeah it is indeed not related to this PR, and kinda the expected behavior so far.

I agree that we should ideally improve on that unhappy path (ie make unlock automatically revert changes in the git index if it detected decryption failed with the provided key during re-checkout, as well as make lock --force take care of the git restore --staged + git restore of secret files to undo any odd state in those situations. Noted for a future improvement 👍

---

(PS: for the record that's also what you get with git-crypt and unlocking with a wrong key, except git-crypt doesn't use HMAC for integrity verification iinm so with git-crypt it might happen that you could be (un-)lucky enough to use a bad key that still doesn't crash the decryption cipher (resulting in garbage decrypted output, but that git-crypt wouldn't realize is invalid). And it's only if you happen to use an incorrect key that makes the decryption impossible that git-crypt will leave you in that odd git state.
While at least git-conceal has integrity verification with HMAC so detects if you're using the wrong key when decrypting a file and is thus guaranteed to fail in those cases (as opposed to report success but generate garbage)

[iangmaia]

git-crypt doesn't use HMAC for integrity verification iinm

at least git-conceal has integrity verification with HMAC so detects if you're using the wrong key when decrypting a file and is thus guaranteed to fail in those cases

That's a great improvement 👍

[AliSoftware]

Brainstorm: what would be a good name for a subcommand to get out of such a messy status after unlocking with the wrong key?

I'm thinking that, while we could execute such a revert/cleanup automatically during unlock if provided with wrong key, to make it transparent and restore to a good state in those cases, maybe it could also be useful to have a dedicated command for also running it manually in case such thing happen in other situations?
(The most likely being if someone switches from one git branch that uses a given key to a different branch in which the secrets have been encrypted with a different key because a key rotation happened in between)

Or maybe git lock --force (which was actually meant to handle those kind of situations too, but apparently isn't enough currently) would be that command? Though would be annoying to have to lock/unlock the repo just to get out of bad git state… although I think such a case should only happen in practice if decryption failed due to bad key, so maybe forcing to lock then re-unlock (with the right key) makes sense to get out of those?

[iangmaia]

Tested it on PCAndroid and it worked great 👍

[iangmaia]

@AliSoftware Oops, didn't realize we had auto-merge here (to give you chance of updating the docs for this comment) 🤦‍♂️

[AliSoftware]

I can create a follow-up PR for adding the tip about histognorespace 👍

---

In retrospect I'm starting to regret that the unlock command has evolved to require base64: prefix for providing the base64 key inline while making the fallback case (parameter without special prefix) be for reading the key from a file. Especially since passing the base64 key inline will be way more common use case in practice than passing a file.

So maybe I'll make a corrective PR that switches the format and:

• Interpret the parameter a file name only if prefixed with file: (or maybe with @, to borrow the syntax used by curl)
• Interpret the parameter as the base64 key directly if not prefixed by either env: nor @¹

That would make base64-value the default and file input the exception, instead of the other way around.

Footnotes

1. I checked and @ is not part of the Base64 alphabet so that wouldn't be a risk of conflict there ↩

[AliSoftware]

So maybe I'll make a corrective PR that switches the format and:

• Interpret the parameter a file name only if prefixed with file: (or maybe with @, to borrow the syntax used by curl)
• Interpret the parameter as the base64 key directly if not prefixed by either env: nor @

I've implemented that change in #14.

In fact, I decided to completely drop the option to read the raw binary key from a file, because:

• That was the less likely use case in practice—as most people will either provide the key via env var on CI, or by pasting it from Secret Store on local Macs
• It being the default syntax was sending wrong signals/suggestions (as in practice I'd recommend the users to avoid storing the binary key in a (potentially unprotected) file (especially if it could be at risk of being accidentally committed, if that file was saved in the working copy dir)
• In the unlikely event that the user really still need that use case, they can still use - (read from stdin) + shell indirection (< /path/to/file.bin) anyway. But at least that's not the "default, easy go-to command" anymore, so that doesn't risk suggesting it's a recommended option

So now after #14 it'll only be either - (stdin), or env:ENVVAR, or <base64encodedkey> without any prefix.