Skip to content

Configuration.md

Latest commit

 

History

History
161 lines (126 loc) · 13.9 KB

Configuration.md

File metadata and controls

161 lines (126 loc) · 13.9 KB
Raw

This article contains information about the team's agreed-upon conventions and workflows regarding all things configuration in dialogporten.

The team recognizes three types of configuration, which should all be handled differently:

  1. Normal configuration
  2. Autogenerated secrets
  3. External secrets

Table of Contents:

1. Discoverability

To aid in the discoverability of available configuration, the complete set of configuration keys, both secrets and non secrets, that is declared in the domain should be present in appsettings.Development.json. For normal configuration, use a sensible local development default. For secrets use "TODO: Add to local secrets" or some other invalid, yet human readable useful value. Example:

// appsettings.Development.json
{
  "Infrastructure": {
    // Secret that should not be exposed
    "DialogDbConnectionString": "TODO: Add to local secrets", 
    "Altinn": {
      // Non secret with a sensible value for local development
      "BaseUri": "https://platform.tt02.altinn.no/"
    }
  }
}

2. Normal configuration

By "normal configuration" we mean non secrets or configuration that does not pose a security risk when shared externally. This type of configuration can live directly in the git repository, which gives us benefits like version controlled configuration, and a single point of truth for each environments configuration.

Each runtime defines an appsettings.{Environment}.json for each target environment. This file should contain environment specific configuration. In addition to the specific environment configuration files there should also exist a appsettings.json. This file should contain cross cutting environment configuration. The appsettings.{Environment}.json configuration takes precedence over the appsettings.json configuration, meaning - if the configuration key foo exists in both files, the value from appsettings.{Environment}.json is used.

2.1. Word of caution

Due to the implicit and cross cutting nature of appsettings.json it should be used cautiously. It's often better to let a runtime fail hard and crash due to no configuration, than using the wrong configuration value silently. Say for example that a .well-known authentication service endpoint is configured in appsettings.json because the same endpoint is used in Development, Test and Stage. However, the developer forgets to add it to appsettings.Production.json and thereby not overriding it for the Production environment. Thus Production is now using the same .well-known endpoint as the rest of the environments and thereby open to non-production tokens. It should produce a big fat red critical error, but instead it continues on like everything is fine. Therefore the .well-known endpoint, and many other configurations, does not belong in appsettings.json. The absence of a configuration is often better than the wrong one silently working.

2.2. How to

Adding, removing or updating non secret configuration is as simple as modifying appsettings.json for cross cutting configuration or appsettings.[Environment].json for environment specific configuration. Use caution when adding configuration to appsettings.json, see 2.1. Word of caution. When adding a required configuration to appsettings.[Environment].json, remember to add the configuration for all the environment specific json files. GitHub actions will do the rest when merging to main.

2.3. CI/CD

When a pull request containing configuration changes is merged with main, one of two things happens:

  • If there are configuration changes and not changes to the src folder, a GitHub action workflow kicks in pushing only the configuration to Azure App Configuration and updating the sentinel value.
  • If there are changes to the src folder regardless of configuration changes, the normal build-deploy GitHub action workflow will kick in. It in turn will reference the above workflow pushing changes to Azure App Configuration and updating the sentinel value only when there are configuration changes. We can use the paths-filter to run the config action depending on if there are changes to the configuration.

Each runtime / presentation layer adds its own configuration keys to azure app configuration through appsettings.json and appsettings.[Environment].json with a runtime tag, indicating which runtime that owns that specific configuration value.

2.4. Local-only configuration

When developing, it is often useful to have configuration that is only used locally, eg. to override default LocalDevelopment configuration values. If a appsettings.local.json file is present, this file will take precedence over all other configuration files after merge. This file should not be checked in to the repository (it is in .gitignore).

2.5. SQL statement and parameter logging

SQL statement and parameter logging is controlled through app configuration. The settings live under the Infrastructure section and are available in all runtimes that use the shared infrastructure package.

Keys:

  • Infrastructure:EnableSqlStatementLogging (bool). Records successful SQL statement activities (db.statement) in OpenTelemetry traces.
  • Infrastructure:EnableSqlParametersLogging (bool). Includes SQL parameter values in traces and adds a consolidated db.query.parameters tag.

Example:

{
  "Infrastructure": {
    "EnableSqlStatementLogging": true,
    "EnableSqlParametersLogging": true
  }
}

Notes:

  • If you need parameter values for successful statements, enable both settings. When statement logging is disabled, successful SQL statement activities are filtered out and parameter values will only appear on failed statements. Note that certain failures (unique/foreign key violations) are still suppressed, so parameter values will not surface for those errors.
  • These settings can expose sensitive data. Enable them only for troubleshooting and disable them when done.

Propagation (Azure App Configuration):

  • Update the keys in Azure App Configuration for the target environment/runtime.
  • Bump the Sentinel key value (any change is sufficient, e.g. an incrementing number or timestamp) to trigger refresh. Apps poll for refresh on an interval of about 1 minute, so expect up to ~1 minute for changes to apply.

3. Autogenerated secrets

Autogenerated secrets are secrets that are generated and used by the IaC when creating resources in Azure. When a resource which requires a password is generated, the IaC bootstrap script will create a new key value pair with the name of the password as its key, and the actual password as its value. The IaC then checks if the environments key vault has a secret with the same name. If it does, the autogenerated password will be discarded and the "cached" password will be used. If not, it will add/"cache" the autogenerated password to the key vault and use it when creating the resource.

An example of a autogenerated secret could be PostgreSql admin password.

The auto generated secrets will not have to be handled or known by anyone other than the specific environments azure key vault. In addition, new environments does not require human intervention to create and store these kinds of secrets.

3.1. How to

TODO: Kan vi bruke miljøets KV direkte som cache?

Modifying autogenerated secrets is a multi step process.

To add and use the new secret:

  1. Generate the secret in gh action, like we do for the PostgreSQL password: 
    ...
  - name: Generate PostgreSQL password
    id: pwd-generator
    shell: pwsh
    run: |
      Import-module "./.github/tools/pwdGenerator.ps1" -Force
      $password = (GeneratePassword -length 30).Password
      echo "::add-mask::$password"
      echo "::set-output name=postgresqlPassword::$password"
...
    Pass the password as parameter to the deployment step. In the case of the PostgreSQL password, this password will be different every time the IaC runs which is why we don't want to use this value directly. The bicep module postgreSql will add postgresqlPassword to the specific environments key vault if it does not already exist.
  2. Use the secret by fetching it from the environment key vault envKeyVault.getSecret('postgresqlPassword').

Removing or modifying autogenerated secrets is not supported through IaC.

4. External secrets

External secrets are secrets that require human intervention to configure and must be known ahead of time, before the IaC runs. This could be a 3rd party api key or certificates.

To be able to support both external secrets and an IaC that can spin up a brand new environment on the fly, we've introduced a convention based "source key vault". The source key vault should exclusively contain external secrets and must exist before the IaC runs as it depends on it. The source key vault could be used by multiple environments.

4.1. Naming convention

When the IaC runs, it copies specific secrets from the source to the deployments target environment key vault based on the following naming convention:

Convention type Source key vault Environment key vault
Wildcard dialogporten--any--[SecretKey] [SecretKey]
Environment dialogporten--[Environment]--[SecretKey] [SecretKey]

There are two conventions, one that targets a specific environment, and one "any" that targets all environments that the source is connected to through IaC. The environment specific convention takes precedence over the "any" convention. For example, say we are running an IaC that targets the Stage environment with the following keys in the source key vault:

Source key vault Stage key vault IsCopied Description
dialogporten--any--foo false Environment specific "foo" convention takes precedence.
dialogporten--any--bar bar true No environment specific "bar" exists.
dialogporten--Stage--foo foo true Overrides the "any" convention for "foo".
dialogporten--Test--bar false Targets another environment.
dialogporten--Stage--foo--bar foo--bar true Supports hierarchical values with -- as separator.
iDoNotFollowTheConvention false Is ignored as it does not follow any convention.

4.2. How to

Instruct your friendly secrets manager to modify the desired key value pair in the source key vault before merging your pull request to the main branch.

4.3. Secret expiry and rotation

Expiring external secrets (Maskinporten JWKs, certificates, vendor keys) are monitored by the daily Check Key Vault secret expiry workflow (.github/workflows/check-keyvault-secret-expiry.yml). It reads attributes.expires on the source key vault secrets and alerts to Slack as expiry approaches (30 / 7 / 1 days out), plus a full overview digest on the 1st of each month.

Conventions:

  • Set attributes.expires on the source secret to the credential's actual expiry date (a Maskinporten key's valid_until from Samarbeidsportalen, or a certificate's notAfter). The workflow warns ahead of expiry on its own, so enter the real date rather than an earlier reminder date.
  • Set the date only on the source secret. copySecrets.bicep copies a secret's value into each environment key vault but not its attributes, so the environment-vault copies have no expiry date by design. Only the source secrets are monitored; leave the environment-vault copies as they are.
  • Watch-listed secrets must have an expiry. Any secret named in .github/keyvault-expiry-monitored-secrets.yml is required to have attributes.expires set; if one is missing it, the workflow flags it in Slack.

Maskinporten clients

There are two Maskinporten clients: one shared by test, yt01 and staging (via the non-prod source key vault), and one for prod. A client can hold several registered JWKs at once; the JWT kid header in the signed assertion tells Maskinporten which registered key to validate against. This is what makes a no-downtime rotation possible: register the new key alongside the old one, switch over, then remove the old one (see below).

Rotating a Maskinporten JWK

  1. Obtain the new key and register its JWK on the Maskinporten client. Keep the old key registered until the new one is verified (overlap window).
  2. Update the source secret value and set attributes.expires to the new key's valid_until.
  3. Deploy the affected environment(s) so copySecrets propagates the new value, then restart the apps so they reload the key (it is read at startup via an app-config Key Vault reference).
  4. Verify the apps authenticate with the new key: no invalid_grant, and a successful Maskinporten token call visible in Application Insights.
  5. Deregister the old key from the client once nothing uses it.

See the rotation runbook for step-by-step detail: https://digdir.atlassian.net/wiki/x/QIDDAgE