Improve secrets management

[debovema]

TL;DR : The current way of managing secrets should be improved.

Current situation

Currently, the app descriptor (flogo.json) is pre-processed during engine startup to replace all occurrences of "SECRET:xxx" by the decrypted value of xxx based on AES-256 with a secret decryption key provided by an environment variable FLOGO_DATA_SECRET_KEY.

See line

core/engine/secret/config.go

Line 11 in 98a1d79

re := regexp.MustCompile("SECRET:[^\\\\\"]*")

and following for reference.

Default value of FLOGO_DATA_SECRET_KEY encryption key is flogo. If this default value is not changed, all secret values can be decrypted easily, leaving the user in the illusion of security.

Moreover, where the secret values will be encoded ?

• If it's in the Flogo UI, it might use the default value flogo as encryption key. It could certainly be changed but that would complexify the configuration of Flogo UI.
• If it's manually, it would require tooling or tedious documentation.

Eventually, this pre-processing of app descriptor triggered by

core/engine/appconfig.go

Line 61 in 98a1d79

updated, err := secret.PreProcessConfig(jsonBytes)

is in conflict with properties finalization triggered by

core/engine/engineimpl.go

Line 103 in 98a1d79

option := app.FinalizeProperties(enableExternalPropResolution, secret.PropertyProcessor)

Especially, secret.PropertyProcessor function at

core/engine/secret/config.go

Line 33 in 98a1d79

func PropertyProcessor(properties map[string]interface{}) error {

will not replace anything since properties with values prefixed by "SECRET:" have already been replaced.
That's a pity because the property.PostProcessor approach seems the best.

Proposal

1. Simplify properties post-processing implementation

See PR #13.

With this simplification it will be easier to add new properties post-processors to manage secrets values.

2. Remove JSON pre-processing for "SECRET:*" values

See PR #14.

As said above, this pre-processing is conflicting with the actual property.PostProcessor post-processor.

3. Make the secrets "type-safe"

Considering the following properties array:

  "properties": [
    {
      "name": "backend.url",
      "type": "string",
      "value": "http://xxx"
    },
    {
      "name": "backend.username",
      "type": "string",
      "value": "user"
    },
    {
      "name": "backend.password",
      "type": "string",
      "value": "SECRET:xjOqSXAFanom8Y8NW5MCJg=="
    }
  ]

We could replace it by:

  "properties": [
    {
      "name": "backend.url",
      "type": "string",
      "value": "http://xxx"
    },
    {
      "name": "backend.username",
      "type": "string",
      "value": "user"
    },
    {
      "name": "backend.password",
      "type": "secret",
      "value": "xjOqSXAFanom8Y8NW5MCJg=="
    }
  ]

4. Create (external) secret resolvers

Following the model of external property resolvers, we could implement external secret resolvers.
Especially the two kinds of resolvers would match the properties to process based on their type defined in 3.