[Helm] Allow providing SSL settings to the Elastic Agent (standalone mode)

[eedugon]

Describe the enhancement:
This issue has the same nature as #6285, which is created for Fleet managed agents. I've created a different issue because the way to resolve this would be different than in the case of a managed agent.

When following the doc https://www.elastic.co/guide/en/fleet/current/example-kubernetes-standalone-agent-helm.html to install an standalone agent with something like:

helm upgrade --install std-demo1 ./deploy/helm/elastic-agent \
--set kubernetes.enabled=true \
--set outputs.default.type=ESPlainAuthAPI \
--set outputs.default.url=https://monitoring-es-http:9200 \
--set outputs.default.api_key="WmNrZTBKTUJ5ei1BZUJaR1IyazY6MWxqb1djeFdRTlNfcElKdDVjTngzZw=="

The generated agents are going to fail to contact Elasticsearch if the cluster certificate is signed with private / corporate CAs, giving errors like:

{"log.level":"error","@timestamp":"2024-12-16T15:40:54.134Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"network.transport":"tcp","log.logger":"esclientleg","log.origin":

The solution to this is to provide the CA certificate via ssl.certificate_authorities setting, per document https://www.elastic.co/guide/en/fleet/current/elastic-agent-ssl-configuration.html.

I would suggest to provide a way to provide ANY SSL setting supported by the Agent, for example ssl.verification_mode: none would also allow to solve this situation.

cc: @pkoutsovasilis / @nimarezainia

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pkoutsovasilis]

@eedugon you tagged the wrong Panos 😄

could you please try to run the following and tell me if it gets you the desired outcome

helm upgrade --install std-demo1 ./deploy/helm/elastic-agent \
--set kubernetes.enabled=true \
--set outputs.default.type=ESPlainAuthAPI \
--set outputs.default.url=https://monitoring-es-http:9200 \
--set outputs.default.ssl.certificate=<contents of the certificate>  \
--set outputs.default.ssl.certificate_authorities=<contents of the certificate>  \
--set outputs.default.<any_output_specific_key>=<...>  \
--set outputs.default.api_key="WmNrZTBKTUJ5ei1BZUJaR1IyazY6MWxqb1djeFdRTlNfcElKdDVjTngzZw=="

[eedugon]

very good point @pkoutsovasilis , I didn't realize the settings could also be configured at the output level, so it should definitely work without any effort more than the documentation.

I'll try that out and let you know the result. I will try out the certificate_authorities setting, and not the certificate or key, as those should be used when mutual TLS is needed.

btw, the certiticate_authorities setting can be provided with any of:

ssl.certificate_authorities: ["/path/to/root/ca.pem"] --> In this case we would need a secret to store and mount the file.

ssl.certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    CERTIFICATE CONTENT APPEARS HERE
    -----END CERTIFICATE-----

^^ This case is easy if we use a values file during helm install execution.... is it feasible to provide that in the command line directly through --set?

I'll get back to you with my test results.

[pkoutsovasilis]

ssl.certificate_authorities: ["/path/to/root/ca.pem"] --> In this case we would need a secret to store and mount the file.

sure thing you can create a k8s Secret utilise the extraVolumes and extraVolumeMounts in the agent preset and then there pass the path of the volumeMount and not the full certificate string 🙂

[eedugon]

@pkoutsovasilis , as mentioned in private, the CA certificate can be added to the installation in the following way:

1. create values file (in my case I called it values-es-ca.crt) with the following content:

outputs:
  default:
    ssl.certificate_authorities:
      - |
        -----BEGIN CERTIFICATE-----
        MIIDSjCCAjKgAwIBAgIRALfMeXFmYLUW4HaNXLzfP4cwDQYJKoZIhvcNAQELBQAw
        LzETMBEGA1UECxMKbW9uaXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRw
        MB4XDTI0MTIxMTEwMTMzNVoXDTI1MTIxMTEwMjMzNVowLzETMBEGA1UECxMKbW9u
        aXRvcmluZzEYMBYGA1UEAxMPbW9uaXRvcmluZy1odHRwMIIBIjANBgkqhkiG9w0B
        AQEFAAOCAQ8AMIIBCgKCAQEAsljXOJrCsvZGHr2SroKUGJOnJwtz8VTx2spQ96OO
        8Q+Tw8gX5C32bjplwAeQsnZ7i5YRRLneaG6NXJuaUEDefsKeG6jdN/bjce+Sz5xm
        U6guXe3TuIyk0+UoFtOzZ1lYUNk6lg9+60iOllRO3xI7SwxqKAaC4KKs7QL1jQCR
        Q14QedcPrS4v76OT+TJvYWrbTFLtYYvfJDGop5EE90v7iB5j0ehSLjfC2R4CD5Kr
        OSYJrGqnhnznbUUjulVqCkPKmgZdcvcIBn4NnZlN6oYzwhRHSSj6r3sy11j3A6SA
        7KeG+IlY+LmRtrj85tiRJ3pXz1FD2d/Mf6cNI6lBGRrNZwIDAQABo2EwXzAOBgNV
        HQ8BAf8EBAMCAoQwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA8GA1Ud
        EwEB/wQFMAMBAf8wHQYDVR0OBBYEFMgVU7RwXciOOz18FcQDTQZXy9gIMA0GCSqG
        SIb3DQEBCwUAA4IBAQCgOSe2s3Xc0QKR+86xmoAADpoe7SFT0Yyh3rMjL+0p02m3
        CqrILqCRNFu9az8gc47hUt9Crb1BXmTR0Sb23M1NvGmR2D2K7CLp/SvkAP6RlB4M
        dZ70UKw4ohq+VSSSiLOoHYdlH46xtunLL31GLYRwD+OgeKAc5pwqWgZkndzxrouB
        uNyoxB5NGvaVUqIouILQ9V2fvraCNf+RxuQ0AaPxdt/CNpFaXpbJBuXJCphlydu0
        KztVqRv5EZjuYpcXDfGP9BEvMy6o895H4iG0M2wb2e3WEDo6jH5pecZfc4yz8iae
        jLwbOPbWqOGRkxTMLOV6Q1dtr09zf2SuOQuxm7F2
        -----END CERTIFICATE-----

Note that if the previous file defines directly outputs.default.ssl.certificate_authorities this won't work (not sure why).

2. Then install with something like:

helm upgrade --install std-demo2 ./deploy/helm/elastic-agent \
-f values-es-ca.yaml \
--set kubernetes.enabled=true \
--set outputs.default.type=ESPlainAuthAPI \
--set outputs.default.url=https://monitoring-es-http:9200 \
--set outputs.default.api_key="FMoH1JMByz-AeBZGZwox:5tiD4ageTWW_gGSDKqPqfg"

I haven't checked by referencing a file and using the extraVolumes + extraVolumeMounts settings, but I think with the previous would be enough.

If there's a way to add the CA cert directly with a --set instead of using a values file let me know.

I will prepare changes in our documented example to cover this use case.

As a final comment (before determining if we close this issue or not), I'd like to mention that we are solving this by adding the CA directly at outputs.default configuration level, which is a valid approach.

Offering a way to configure the ssl settings at root config level might be interesting too, as those would be global for the agent.