diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index c067d1a4..3fcd2add 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -9,6 +9,7 @@ env: VERSION_SOPS: 3.13.1 # renovate: github-releases=helmfile/vals VERSION_VALS: 0.44.0 + SOPS_DISABLE_VERSION_CHECK: "true" on: pull_request: diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..842922d4 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,155 @@ +# AGENTS.md + +This repository is `helm-secrets`, a Helm plugin implemented mostly in POSIX shell. It decrypts Helm value files on demand, supports secret references through multiple backends, and integrates with Helm downloader and post-renderer plugin mechanisms. + +## Project Map + +- `plugin.yaml` is the Helm 3 plugin manifest. It registers the `secrets` command and downloader protocols. +- `plugins/helm-secrets-cli`, `plugins/helm-secrets-getter`, and `plugins/helm-secrets-post-renderer` contain Helm 4 plugin manifests. Helm 4 currently treats CLI, getter, and post-renderer plugins as separate plugin types, so these manifests are split even though Helm 3 can declare the command and downloader protocols together in `plugin.yaml`. Keep versions and user-facing help synchronized with `plugin.yaml` and `scripts/commands/help.sh`. +- `scripts/run.sh` is the main plugin entrypoint. It initializes globals, loads libraries/backends, parses top-level `helm secrets` options, then dispatches to command scripts. +- `scripts/commands/` contains subcommands and Helm integration: + - `encrypt.sh`, `decrypt.sh`, `edit.sh` implement direct file operations. + - `helm.sh` wraps arbitrary Helm commands, decrypts `-f`, `--values`, `--set-file`, and decrypts literals in `--set`, `--set-string`, and `--set-json`. + - `downloader.sh` implements `secrets://`, `secrets+gpg-import://`, `secrets+gpg-import-kubernetes://`, `secrets+age-import://`, `secrets+age-import-kubernetes://`, and `secrets+literal://`. + - `post-renderer.sh` evaluates `vals` references in rendered manifests when `--evaluate-templates` is enabled. + - `help.sh` and `version.sh` provide user-facing CLI output. +- `scripts/lib/` contains shared shell functions for logging, traps, path handling, backend dispatch, file retrieval, HTTP downloads, and strict variable expansion. +- `scripts/lib/backends/` contains in-tree backends: + - `sops.sh` is the default backend. It can encrypt, decrypt, edit, and detect SOPS-encrypted content. + - `vals.sh` resolves `ref+...` references. It does not support encrypt or edit. + - `noop.sh` passes files through for tests and non-encrypted workflows. + - `_custom.sh` is a helper contract for out-of-tree backends. +- `scripts/lib/file/` abstracts value source retrieval: + - `local.sh` handles normal files. + - `http.sh` downloads `http://` and `https://` values with `curl` or `wget`. + - `custom.sh` delegates arbitrary `*://` sources to Helm through a tiny chart in `helm-values-getter`. +- `scripts/wrapper/` contains wrapper scripts for Windows and optional automatic `helm secrets` forwarding. +- `docs/` is the wiki-style documentation source. Update docs when CLI flags, environment variables, security behavior, or integration behavior changes. +- `examples/` contains sample charts and backend scripts for SOPS, vals, Argo CD, Terraform, and custom backends. +- `tests/` contains first-party Bats tests and assets. `tests/bats/` is vendored/submodule test tooling; do not edit it unless intentionally updating submodules. + +## Runtime Flow + +1. Helm invokes `scripts/run.sh` through a plugin manifest. +2. `run.sh` sets `HELM_BIN`, `SCRIPT_DIR`, `TMPDIR`, default backend (`sops`), quiet mode, decrypted file naming settings, and feature flags from `HELM_SECRETS_*` environment variables. +3. It loads `common.sh`, `expand_vars_strict.sh`, `file.sh`, `backend.sh`, and `http.sh`, then calls `load_secret_backend`. +4. Top-level arguments are parsed before dispatch. Global flags such as `--backend`, `--backend-args`, `--quiet`, `--ignore-missing-values`, `--evaluate-templates`, and `--decrypt-secrets-in-tmp-dir` affect later command behavior. +5. Direct commands call backend helpers through `backend.sh`. Wrapped Helm commands source `commands/helm.sh`. +6. `helm_wrapper` rewrites Helm arguments: + - For `-f`/`--values`, it fetches the source, decrypts encrypted files into `.dec` files or temp files, and passes decrypted paths to Helm. + - For `secrets://...` values, prefer passing the protocol URL through to Helm so Helm's downloader plugin handles it directly. Avoid resolving `secrets://` through `_file_get` in the wrapper unless there is a specific compatibility reason and regression coverage for trailing newlines. + - For `--set-file`, it decrypts file contents and preserves Helm key prefixes. + - For `--set`, `--set-string`, and `--set-json`, it resolves encrypted literal values and preserves escaped commas, lists, and trailing newlines. + - It records generated decrypted files and removes them in `_trap_hook`. +7. Downloader protocol handling in `downloader.sh` prints decrypted content to stdout for Helm downloader usage. Key-import protocols initialize temporary GPG homes or `SOPS_AGE_KEY_FILE` before decrypting. +8. If template evaluation is enabled, `helm.sh` injects a Helm post-renderer. Helm 3 invokes `helm secrets post-renderer`; Helm 4 uses the separate `secrets-post-renderer` plugin because Helm 4 plugin types are split. + +## Backend Contract + +Backends are dispatched by name through functions in `scripts/lib/backend.sh`. A backend named `foo` must provide: + +- `_foo_backend_is_file_encrypted FILE` +- `_foo_backend_is_encrypted` reading stdin +- `_foo_backend_encrypt_file TYPE INPUT OUTPUT` +- `_foo_backend_decrypt_file TYPE INPUT [OUTPUT]` +- `_foo_backend_decrypt_literal VALUE` +- `_foo_backend_edit_file TYPE INPUT` + +In-tree backend selection accepts `sops`, `vals`, and `noop`. Out-of-tree backends can be loaded by file path through `--backend` or `HELM_SECRETS_BACKEND`; they normally source `scripts/lib/backends/_custom.sh`. `HELM_SECRETS_ALLOWED_BACKENDS` restricts allowed backend names and is tested by the suite. + +Backend-specific binary overrides: + +- `HELM_SECRETS_SOPS_PATH` or legacy `HELM_SECRETS_SOPS_BIN` +- `HELM_SECRETS_VALS_PATH` +- `HELM_SECRETS_CURL_PATH` +- `HELM_SECRETS_WGET_PATH` +- `HELM_SECRETS_KUBECTL_PATH` + +## Important Environment Variables + +- `HELM_SECRETS_BACKEND`, `HELM_SECRETS_BACKEND_ARGS`, `HELM_SECRETS_ALLOWED_BACKENDS` +- `HELM_SECRETS_QUIET`; defaults to `true` in Argo CD when `ARGOCD_APP_NAME` is present +- `HELM_SECRETS_DEC_PREFIX`, `HELM_SECRETS_DEC_SUFFIX`, `HELM_SECRETS_DEC_DIR`, `HELM_SECRETS_DEC_TMP_DIR` +- `HELM_SECRETS_IGNORE_MISSING_VALUES` +- `HELM_SECRETS_EVALUATE_TEMPLATES`, `HELM_SECRETS_EVALUATE_TEMPLATES_DECODE_SECRETS` +- `HELM_SECRETS_DECRYPT_SECRETS_IN_TMP_DIR` +- `HELM_SECRETS_LOAD_GPG_KEYS` +- `HELM_SECRETS_URL_VARIABLE_EXPANSION` +- `HELM_SECRETS_VALUES_ALLOW_SYMLINKS`, `HELM_SECRETS_VALUES_ALLOW_ABSOLUTE_PATH`, `HELM_SECRETS_VALUES_ALLOW_PATH_TRAVERSAL` +- `HELM_SECRETS_ALLOW_GPG_IMPORT`, `HELM_SECRETS_ALLOW_GPG_IMPORT_KUBERNETES`, `HELM_SECRETS_ALLOW_AGE_IMPORT`, `HELM_SECRETS_ALLOW_AGE_IMPORT_KUBERNETES` +- `HELM_SECRETS_KEY_LOCATION_PREFIX` +- `HELM_SECRETS_WRAPPER_ENABLED`, `HELM_SECRETS_HELM_PATH`, `HELM_SECRET_WSL_INTEROP` + +When adding a user-visible variable, update `scripts/commands/help.sh`, Helm 4 CLI help in `plugins/helm-secrets-cli/plugin.yaml`, docs, and tests. + +## Portability Rules + +- First-party plugin scripts under `scripts/` are POSIX `sh`, not Bash. They are tested under dash, ash, bash-as-sh, zsh-as-sh, posh, macOS `/bin/sh`, Cygwin, and WSL. +- Avoid Bash-only syntax in `scripts/**/*.sh`: no arrays, `[[ ]]`, process substitution, `local`, `${var//...}`, or `pipefail`. +- Files under `tests/`, including Bats tests and test helpers, may use Bash syntax because the test suite runs under Bash/Bats. +- Preserve careful quoting. This project handles paths with spaces, special characters, Windows paths, WSL path conversion, escaped commas, Helm set lists, and trailing newlines. +- Be cautious with command substitutions: POSIX shells strip trailing newlines. Existing code uses sentinel characters where preserving newlines matters. +- `pipefail` is not available in POSIX `sh`, so command substitutions need extra care. Multiple commands inside `$()` are dangerous because only the last command controls the substitution exit code. If a later command must run after a checked command, preserve the status explicitly, for example `$(cmd; status=$?; cmd2; exit "$status")`. This pattern may not work the same way with pipelines because pipeline status is also limited without `pipefail`. +- Do not rely on GNU-only tools unless guarded. macOS `sed -i` differs; use `_sed_i`. +- Keep generated decrypted files cleaned up via traps. Avoid leaving `.dec` files outside explicit inline operations. +- Do not edit vendored `tests/bats/**` unless the task is to update Bats submodules. + +## Security-Sensitive Areas + +- `scripts/lib/file.sh` enforces optional restrictions on symlinks, absolute paths, and `..` path traversal. +- `scripts/commands/downloader.sh` controls whether GPG/age key imports and Kubernetes key imports are allowed. +- `HELM_SECRETS_KEY_LOCATION_PREFIX` restricts key file locations for import protocols. +- `scripts/lib/file/http.sh` can expand environment variables inside URLs only when `HELM_SECRETS_URL_VARIABLE_EXPANSION=true`; keep this opt-in. +- Avoid logging decrypted secret values. Many tests assert quiet behavior and cleanup. + +## Tests + +Use the vendored Bats runner when Bats is not installed: + +```sh +bash tests/bats/core/bin/bats -r tests/unit +``` + +Run backend-specific unit suites: + +```sh +HELM_SECRETS_BACKEND=sops bash tests/bats/core/bin/bats -r tests/unit +HELM_SECRETS_BACKEND=vals bash tests/bats/core/bin/bats -r tests/unit +``` + +Integration tests require a reachable Kubernetes cluster: + +```sh +bash tests/bats/core/bin/bats -r tests/it +``` + +Focused tests are usually enough while iterating, for example: + +```sh +bash tests/bats/core/bin/bats tests/unit/template.bats +bash tests/bats/core/bin/bats tests/unit/secret-backends.bats +``` + +When changing wrapper handling for `--set`, `--set-string`, `--set-json`, `--set-file`, or downloader protocols, run at least one focused `vals` backend test as well as the default `sops` path. The default backend skips several vals-specific literal/reference tests, so a local green run with only `sops` can miss CI failures. + +The test suite installs this repo as a Helm plugin into a temporary Helm home, imports test GPG keys from `tests/assets/gpg`, creates charts under `tests/.tmp/cache`, and copies value assets into each test temp directory. Unit tests do not require Kubernetes; integration tests cover install/upgrade/diff paths and Kubernetes key-import protocols. + +Before running Bats, make sure a `gpg-agent` is running for the test GPG operations. Terminate it after the tests, for example with `gpgconf --kill gpg-agent`, so test state does not leak into later runs. + +## CI Expectations + +CI runs shell linting and checkbashisms, then unit tests across Linux, macOS, Windows, Cygwin, WSL, multiple shells, Helm 3 and Helm 4, SOPS, and vals. Coverage rewrites `env sh` to `env bash` only for bashcov; do not copy that pattern into normal code. + +The current CI version pins are in `.github/workflows/ci.yaml`. + +## Change Guidelines + +- Keep behavior changes tightly scoped and add/adjust Bats tests near the affected behavior. +- Before committing, run `shfmt` and `shellcheck` on changed shell files. +- For new features or bug fixes only, add a line to `CHANGELOG.md`. +- For CLI behavior, update both Helm 3 and Helm 4 manifests/help text when needed. +- For backend changes, update `docs/Secret Backends.md` and backend tests. +- For downloader protocol or Argo CD behavior, update `docs/ArgoCD Integration.md`, `docs/ARGOCD.md`, and relevant template/install tests. +- For security defaults or restrictions, update `docs/Security in shared environments.md` and tests around `HELM_SECRETS_VALUES_ALLOW_*` or key import flags. +- Maintain LF endings except `scripts/wrapper/run.cmd`, which uses CRLF per `.editorconfig`. +- Do not commit generated decrypted files, coverage output, or `tests/.tmp` artifacts. diff --git a/CHANGELOG.md b/CHANGELOG.md index 95e35c1e..7fb85a26 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +## [4.7.7] - 2026-06-08 + ### Fixes +- fix: newline gets stripped if wrapper + secrets:// is used. ([#791](https://github.com/jkroepke/helm-secrets/pull/#791)) - fix: preserve trailing newlines in unchanged --set literals ([#788](https://github.com/jkroepke/helm-secrets/pull/#788)) ## [4.7.6] - 2026-04-04 @@ -417,7 +420,8 @@ Started a fork of https://github.com/zendesk/helm-secrets - Verbose output is now on stderr - Support all helm sub commands and plugins -[Unreleased]: https://github.com/kroepke/helm-secrets/compare/v4.7.6...HEAD +[Unreleased]: https://github.com/kroepke/helm-secrets/compare/v4.7.7...HEAD +[4.7.7]: https://github.com/kroepke/helm-secrets/compare/v4.7.6...v4.7.7 [4.7.6]: https://github.com/kroepke/helm-secrets/compare/v4.7.5...v4.7.6 [4.7.5]: https://github.com/kroepke/helm-secrets/compare/v4.7.4...v4.7.5 [4.7.4]: https://github.com/kroepke/helm-secrets/compare/v4.7.3...v4.7.4 diff --git a/scripts/commands/helm.sh b/scripts/commands/helm.sh index beb2cb30..28601ec9 100644 --- a/scripts/commands/helm.sh +++ b/scripts/commands/helm.sh @@ -219,6 +219,13 @@ helm_wrapper() { load_secret_backend "${DEFAULT_SECRET_BACKEND}" fi + case "${file}" in + secrets://*) + decrypted_files="${decrypted_files}${opt_prefix}${file}," + continue + ;; + esac + if ! real_file=$(_file_get "${file}"); then if [ "${IGNORE_MISSING_VALUES}" = "true" ]; then real_file="$(_mktemp)" diff --git a/tests/assets/values/sops/secrets.trailing-newline.yaml b/tests/assets/values/sops/secrets.trailing-newline.yaml new file mode 100644 index 00000000..2eec6af7 --- /dev/null +++ b/tests/assets/values/sops/secrets.trailing-newline.yaml @@ -0,0 +1,26 @@ +podAnnotations: + value1: ENC[AES256_GCM,data:D7nrkhAvRDlXzMYo,iv:LQGlPvYKLixtGTfeb0yuPkht7hxNBkdpZJwbSXXNL0g=,tag:eIY2F0NWztW4CqdS51gAyw==,type:str] + value2: ENC[AES256_GCM,data:7yF/ZGUPQbUWJ5bCEA==,iv:6p/H+fazFvkusM7U1/0JWV601jWzTuX7qaFvVn5alDU=,tag:+aaQ9gqbSAN7T8eqrsKrdw==,type:str] + value3: ENC[AES256_GCM,data:WpEh26SjOtJ+suCDoyk=,iv:/oksQ9Uv/zVXNv/U6FZeqDzaUK85hE50GxmBU2vW0Tg=,tag:6hwuMPDGtsBjnsl0rRFbAg==,type:str] + finalnewline: ENC[AES256_GCM,data:cNoKx+v7EyXtRN3F,iv:sKlcUq+4lDHTO11kqCHIP4dYNkTwOOiKyr9ULxYOyp8=,tag:451Nw4RpyMbRLBY6N6sSdQ==,type:str] +sops: + lastmodified: "2026-06-07T19:30:41Z" + mac: ENC[AES256_GCM,data:UZT5eBpk2QWegacOshOkOh51R29mGCwvo9E8ZoMtYiyv5I8IWCalLEJS5qmktK+lCdB6ZgSeeW33vWvR5eyVabIa0pvn4iDYttps1V9obwdsRfRKmt8HZgyzFlkJYiBpBbcSXDUjyX+T9CKpIPU71YmTAqjbKKRSv7nGLDplaFA=,iv:FodKMBsl6E1WQ5hjdAY1XkWDTrI3mRia6rZ1/Zlgh1o=,tag:xDrOkDxcx1nMUf+yLXhCWw==,type:str] + pgp: + - created_at: "2026-06-07T19:30:41Z" + enc: |- + -----BEGIN PGP MESSAGE----- + + hQEMA9ce5qCwOO4MAQf/RilGQ7T1+Jas0WbkKnoWWZKyxNGFctPNviNLaf2pAh1v + 9wLZPNNUKAnbpQ7PsKI3/jV3Jb+8MROCbW1P19J9We0Q2cWwixqkN6vdNHNVKTBi + 0UJyPQo69xygNOltjHoxE2SWiIgHhbonvxh0tKmCU8AFYqHRG/aEczfZAQDf6iMx + M2hV0/RGcQkJnV3oZPKopHBnITQH8U8khjwzF274Zqo0akqi48BAkVWa4V/vrdQN + KCHF9TYqzRBFrPHikag9UUZfB1D/hJxC0/Wka36lH+8CJ30JQ0KIyrvBkEOZLX/i + 0AtlgAEzPG41fxgmZCWj7urFJ2fNBLbaswgI8FqebtJeATyLrorBjS9tFrUa9X/c + MfA9jA+jbRZ1WsPSynxu7w30WwKtNSyvCDizAVDm7CkacsafOeN9ZV7f3CeqJSfR + jMEL1EcVPbtNTirWRKpwAqmv0MLVVvbxBwBRNsYIfA== + =xfTb + -----END PGP MESSAGE----- + fp: D6174A02027050E59C711075B430C4E58E2BBBA3 + unencrypted_suffix: _unencrypted + version: 3.13.1 diff --git a/tests/unit/template.bats b/tests/unit/template.bats index 8074b5fd..8aa45666 100755 --- a/tests/unit/template.bats +++ b/tests/unit/template.bats @@ -701,6 +701,54 @@ key2: value" 2>&1 assert_success } +@test "template: helm template w/ chart + secrets.trailing-newline.yaml + wrapper matches secrets protocol" { + if on_windows || ! is_backend "sops"; then + skip "Multiline does not work on windows" + fi + + VALUES="assets/values/${HELM_SECRETS_BACKEND}/secrets.trailing-newline.yaml" + VALUES_PATH="${TEST_TEMP_DIR}/${VALUES}" + + create_chart "${TEST_TEMP_DIR}" + + expected_finalnewline=$' finalnewline: \\|\n hello\n world\n value1: \\|\\+' + expected_value1=$' value1: \\|\\+\n multi\n line\n[[:blank:]]*\n value2: \\|\\+' + expected_value2=$' value2: \\|\\+\n multi\n line\n[[:blank:]]*\n[[:blank:]]*\n value3: \\|\\+' + expected_value3=$' value3: \\|\\+\n multi\n line\n[[:blank:]]*\n[[:blank:]]*\n[[:blank:]]*\n labels:' + + run "${HELM_BIN}" template "${TEST_TEMP_DIR}/chart" -f "secrets://${VALUES_PATH}" 2>&1 + protocol_output="${output}" + assert_success + assert_output -e "${expected_finalnewline}" + assert_output -e "${expected_value1}" + assert_output -e "${expected_value2}" + assert_output -e "${expected_value3}" + + run "${HELM_BIN}" secrets template "${TEST_TEMP_DIR}/chart" -f "secrets://${VALUES_PATH}" 2>&1 + assert_success + refute_output --partial "[helm-secrets] Decrypt: secrets://" + assert_output -e "${expected_finalnewline}" + assert_output -e "${expected_value1}" + assert_output -e "${expected_value2}" + assert_output -e "${expected_value3}" + + run "${HELM_BIN}" secrets -q template "${TEST_TEMP_DIR}/chart" -f "${VALUES_PATH}" 2>&1 + assert_success + assert_equal "${output}" "${protocol_output}" + assert_output -e "${expected_finalnewline}" + assert_output -e "${expected_value1}" + assert_output -e "${expected_value2}" + assert_output -e "${expected_value3}" + + run "${HELM_BIN}" secrets -q template "${TEST_TEMP_DIR}/chart" -f "secrets://${VALUES_PATH}" 2>&1 + assert_success + assert_equal "${output}" "${protocol_output}" + assert_output -e "${expected_finalnewline}" + assert_output -e "${expected_value1}" + assert_output -e "${expected_value2}" + assert_output -e "${expected_value3}" +} + @test "template: helm template w/ chart + --set-file service.port=secrets+literal://" { if ! is_backend "vals"; then skip