Reply to comments

Signed-off-by: Henrique Spanoudis Matulis <[email protected]>

Author: hsmatulis

proposals/2025-02-24_secret_providers.md

@@ -24,15 +24,15 @@
 
 ## Why
 
-The motivation behind this design document is to enhance the security and flexibility of secret management in Prometheus. Currently, Prometheus only supports reading secrets from the filesystem or directly from the configuration file, which can lead to security vulnerabilities and limitations when working with certain service providers.
+The motivation behind this design document is to enhance the flexibility of secret management in Prometheus. Currently, Prometheus only supports reading secrets from the filesystem or directly from the configuration file, which can be cumbersome when running it in certain enviroments or when frequent secret rotations are needed.
 
-This proposal introduces secret discovery, similar to service discovery, where different secret providers can contribute code to read secrets from their respective APIs. This would allow for more secure and dynamic secret retrieval, eliminating the need to store secrets in the filesystem and reducing the potential for unauthorized access.
+This proposal introduces secret discovery, similar to service discovery, where different secret providers can contribute code to read secrets from their respective APIs. This would allow for more dynamic secret retrieval, eliminating the need to store secrets in the filesystem and simplifying the user experience.
 
 ### Pitfalls of the current solution
 
-Storing secrets in the filesystem poses risks, especially in environments like Kubernetes, where any pod on a node can access files mounted on that node. This could expose secrets to attackers. Additionally, configuring secrets through the filesystem often requires extra setup steps in some environments, which can be cumbersome for users.
+In certain enviroments, configuring secrets through the filesystem often requires extra setup steps or it might not even be possible. This can be cumbersome for users.
 
-Storing secrets inline can also pose risks, as the configuration file may still be accessible through the filesystem. Additionally it can lead to configuration files becoming cluttered and difficult to manage.
+Storing secrets inline is always possible, but it can lead to configuration files becoming cluttered and difficult to manage. Additionally rotating inline secrets will be more troublesome.
 
 ## Goals
 
@@ -69,19 +69,20 @@ Wherever a `<secret>` type is present in the configuration files, we will allow
 
 ```
 secret_field:
-  provider:  <type of the provider>
-  <property1>: <value1>
-  ...
-  <propertyN>: <valueN>
+  <type of the provider>:
+    <property1>: <value1>
+    ...
+    <propertyN>: <valueN>
 ```
 
 For example when specifying a password fetched from the kubernetes provider with an id of `pass2` in namespace `ns1` for the HTTP passsword field it would look like this:
 
 ```
         password:
-          provider: kubernetes
-          namespace: ns1
-          secret_id: pass2
+          kubernetes:
+            namespace: <ns>
+            name: <secret name>
+            key: <data's key for secret name>
 ```
 
 ### Inline secrets
@@ -99,6 +100,8 @@ The first case is that we have not gotten any secret value since startup. In thi
 
 The second case is that we already have a secret value, but refreshing it has resulted in an error. In this case we should keep the component that uses this secret running with the potentially stale secret, and schedule a retry.
 
+If we are starting up prometheus and do not get any secret values, startup will continue and prometheus will continue to run and retry finding secrets. If there are components that are fully specified, they will run during this time. The idea is that it is better to send partial metrics than no metrics, and the emitted metrics for secrets can alert users if there is a problem with their service providers.
+
 
 ### Secret rotation
 
@@ -129,7 +132,7 @@ A state enum describing in which error condition the secret is in:
 ```
 # HELP prometheus_remote_secret_state Describes the current state of a remotely fetched secret.
 # TYPE prometheus_remote_secret_state gauge
-prometheus_remote_secret_state{provider="kubernetes", secret_id="pass1", state="none"} 0
+prometheus_remote_secret_state{provider="kubernetes", secret_id="pass1", state="success"} 0
 prometheus_remote_secret_state{provider="kubernetes", secret_id="auth_token", state="stale"} 1
 prometheus_remote_secret_state{id="myk8secrets", secret_id="pass2", state="error"} 2
 ```
@@ -141,19 +144,110 @@ Secret providers might require secrets to be configured themselves. We will allo
 ```
 ...
         password:
-          provider: bootstrapped
-          secret_id: pass1
-          auth_token:
-            provider: kubernetes
-            secret_id: auth_token
+          bootstrapped:
+            secret_id: pass1
+            auth_token:
+              kubernetes:
+                name: <secret name>
+                key: <data's key for secret name>
 ```
 
+Note that there is a 'chicken and egg' problem here, where you need to have credentials to access the secret provider itself. Normally this bootstrapping would be done through inline or filesystem secrets. For cloud enviroments, there is usually an identity associated with the machine in the enviroment that can be used. However, in both cases this type of 'bootstrapping' doesn't really increase security as you should already have access to the underlying secrets. Our goal here is just to decrease toil.
+
 However, an initial implementation might only allow inline secrets for secret providers. This might limit the usefulness of certain providers that require sensitive information for their own configuration.
 
 ### Where will code live
 
 Both the Alertmanager and Prometheus repos will be able to use secret providers. The code will eventually live in a separete repository specifically created for it.
 
+## Open questions
+
+* What is the process for creating a secret provider implementation?
+* How can we prevent too many dependencies from getting pulled in from different providers?
+
+## Secret provider interfaces in the wild 
+
+A summary of popular secret providers
+### [HashiCorp Vault](https://developer.hashicorp.com/hcp/docs/vault-secrets)
+
+```
+// Step 1: Authenticate with the Vault server
+// This typically involves providing credentials or using an authentication method (e.g., token, AppRole)
+authentication_response = authenticate_with_vault(authentication_method, credentials)
+auth_token = authentication_response.token
+
+// Step 2: Specify the path to the secret you want to retrieve
+secret_path = "secret/data/my_application/database_credentials"
+
+// Step 3: Read the secret data from the specified path using the auth token
+secret_data_response = read_vault_secret(secret_path, auth_token)
+```
+
+### [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)
+
+```
+// Step 1: Configure AWS credentials and region
+// This is typically done via environment variables, IAM roles, or config files
+configure_aws_sdk()
+
+// Step 2: Specify the name or ARN of the secret
+secret_name = "my/application/database_secret"
+
+// Step 3: Retrieve the secret value from AWS Secrets Manager
+// The secret value is returned as a string, often containing JSON
+secret_value_response = get_aws_secret_value(secret_name)
+secret_password = secret_value_response.secret_string
+```
+
+### [Azure Key Vault](https://learn.microsoft.com/en-us/azure/key-vault/general/overview)
+
+```
+// Step 1: Authenticate with Azure Active Directory
+// This is often done using Managed Identity or a Service Principal
+authentication_client = create_azure_identity_client()
+credential = authentication_client.get_credential()
+
+// Step 2: Create a Key Vault client
+key_vault_url = "https://my-key-vault-name.vault.azure.net/"
+key_vault_client = create_key_vault_secret_client(key_vault_url, credential)
+
+// Step 3: Specify the name of the secret
+secret_name = "DatabasePassword"
+
+// Step 4: Retrieve the secret
+secret = key_vault_client.get_secret(secret_name)
+```
+
+### [Google Secret Manager](https://cloud.google.com/secret-manager/docs)
+
+```
+// Step 1: Authenticate with Google Cloud
+// This is typically handled by the client library using environment variables or service account keys
+secret_manager_client = create_google_secret_manager_client()
+
+// Step 2: Specify the secret name and version
+// Format: projects/PROJECT_ID/secrets/SECRET_NAME/versions/VERSION_ID (use 'latest' for the current version)
+secret_version_name = "projects/my-gcp-project/secrets/my-database-secret/versions/latest"
+
+// Step 3: Access the specified secret version
+response = secret_manager_client.access_secret_version(secret_version_name)
+```
+
+### [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
+
+```
+// Step 1: Configure Kubernetes client
+// This typically involves loading the kubeconfig file or using in-cluster configuration
+kubernetes_client = configure_kubernetes_client()
+
+// Step 2: Specify the namespace and name of the secret
+secret_namespace = "my-application-namespace"
+secret_name = "my-database-secret"
+
+// Step 3: Retrieve the secret object from the Kubernetes API
+secret_object = get_kubernetes_secret(secret_namespace, secret_name, kubernetes_client)
+```
+
 ## Action Plan
 
 * [ ] Create action plan after doc is stable!