Question on TLS and TLS cipher suites.

[jonsch318]

I have a certificate bundle (my intermediate ca and a root ca for openbao) set as the tls_cert_file and the private key at tls_key_file. Config looks like:

ui = true

listener "tcp" {
  address = "[::]:8200"
  cluster_address = "[::]:8201"
  tls_disable = "false"
  tls_client_ca_file = "/etc/trust-bundle/trust-bundle.pem"
  tls_cert_file = "/etc/certs/tls.crt"
  tls_key_file = "/etc/certs/tls.key"
}

# Storage is done through extraSecretVolume
storage "postgresql" {
  ha_enabled = "true"
  max_idle_connections = "5"
  max_parallel = "5"
}

telemetry {
  prometheus_retention_time = "30s"
  disable_hostname = "true"
}

service_registration "kubernetes" {}

I have 2 problems.

1. My Envoy reverse proxy cannot create a connection to openbao:8200 giving the following error:
   upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: TLS_error:|268435736:SSL routines:OPENSSL_internal:NO_SUPPORTED_VERSIONS_ENABLED:TLS_error_end

2. Maybe somewhat related when scaling up 2 multiple openbao instances.

2025-09-02T19:30:08.669Z [INFO]  core: Initializing version history cache for core
2025-09-02T19:30:39.594Z [INFO]  core.cluster-listener.tcp: starting listener: listener_address=[::]:8201
2025-09-02T19:30:39.595Z [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2025-09-02T19:30:39.595Z [INFO]  core: vault is unsealed
2025-09-02T19:30:39.596Z [INFO]  core: entering standby mode
2025-09-02T19:30:40.796Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]
2025-09-02T19:30:40.966Z [ERROR] core: error during forwarded RPC request: error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: remote error: tls: internal error\""
2025-09-02T19:30:40.966Z [ERROR] core: forward request error: error="error during forwarding RPC request"
2025-09-02T19:30:43.331Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=["req_fw_sb-act_v1"]

the api_addr and cluster_addr are correctly set & used, but again no tls cipher suite can be negotiated.

the active server has:

2025-09-02T19:40:19.999Z [INFO]  http: TLS handshake error from 10.244.6.133:40816: EOF
2025-09-02T19:49:24.421Z [INFO]  http: TLS handshake error from 10.244.6.133:44576: EOF
2025-09-02T19:53:10.883Z [INFO]  http: TLS handshake error from 10.244.6.133:60984: EOF
2025-09-02T19:53:10.884Z [INFO]  http: TLS handshake error from 10.244.6.133:60992: EOF
2025-09-02T19:53:10.884Z [INFO]  http: TLS handshake error from 10.244.6.133:60990: EOF
2025-09-02T19:57:43.678Z [INFO]  http: TLS handshake error from 10.244.6.133:38362: EOF
2025-09-02T19:57:43.892Z [INFO]  http: TLS handshake error from 10.244.6.133:38364: EOF
2025-09-02T20:07:44.844Z [INFO]  http: TLS handshake error from 10.244.3.192:39998: remote error: tls: bad certificate
2025-09-02T20:07:51.783Z [INFO]  http: TLS handshake error from 10.244.5.193:40072: remote error: tls: bad certificate

(the EOF are probably from envoy)

!!However
When using tls_passthrough or port-forwarding the browser can correctly negotiate and connect via tls. Just for reference my browser uses: TLS 1.3, X25519MLKEM768 and AES_128_GCM.
My complete certficate chain is a normal ECDSA-p256 with either SHA-512 or SHA-256 so nothing fancy.

[cipherboy]

@jonsch318 Can you share a packet capture of just the failed and working handshake? Wireshark should let you edit it to remove unrelated frames.

The other thing might be to run one of the OpenSSL based connections tools (openssl s_client connect), I seem to recall there was a tlsscan shell script wrapper or similar.

This is strange. :-)

[jonsch318]

Connecting to ::1
CONNECTED(00000003)
Can't use SSL_get_servername
depth=2 C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-root
verify return:1
depth=1 C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-int-man
verify return:1
depth=0 C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-openbao
verify return:1
---
Certificate chain
 0 s:C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-openbao
   i:C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-int-man
   a:PKEY: EC, (prime256v1); sigalg: ecdsa-with-SHA256
   v:NotBefore: Sep  2 19:10:14 2025 GMT; NotAfter: Mar  3 19:10:14 2026 GMT
 1 s:C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-int-man
   i:C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-root
   a:PKEY: EC, (prime256v1); sigalg: ecdsa-with-SHA256
   v:NotBefore: Feb 26 11:55:17 2025 GMT; NotAfter: Nov 23 11:55:17 2027 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDgTCCAyigAwIBAgIUfoLqEciKV2LP8LTxaUZhgPkLbG4wCgYIKoZIzj0EAwIw
gYcxCzAJBgNVBAYTAkRFMQwwCgYDVQQIDANOUlcxDzANBgNVBAcMBkFhY2hlbjES
MBAGA1UECwwJam9uc2NoLmV1MSwwKgYJKoZIhvcNAQkBFh1qb25hcy5tYXguc2No
bmVpZGVyQGdtYWlsLmNvbTEXMBUGA1UEAwwOam9uc2NoLWludC1tYW4wHhcNMjUw
OTAyMTkxMDE0WhcNMjYwMzAzMTkxMDE0WjCBhzELMAkGA1UEBhMCREUxDDAKBgNV
BAgMA05SVzEPMA0GA1UEBwwGQWFjaGVuMRIwEAYDVQQLDAlqb25zY2guZXUxLDAq
BgkqhkiG9w0BCQEWHWpvbmFzLm1heC5zY2huZWlkZXJAZ21haWwuY29tMRcwFQYD
VQQDDA5qb25zY2gtb3BlbmJhbzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABKNQ
WwY0mVP/PYE3FCJrja91afjrep1qqsvTs7ymihAEYGrUIcWQ16aGqp5Wr4c7WE/7
8X9RK0n0qVWp4fjz/LSjggFuMIIBajAdBgNVHQ4EFgQUekI/V7C6AVFyTTtqZD7z
tSqjzyEwDAYDVR0TAQH/BAIwADAOBgNVHQ8BAf8EBAMCA9gwIAYDVR0lAQH/BBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMIHnBgNVHREEgd8wgdyCEW9wZW5iYW8uam9u
c2NoLmV1giFvcGVuYmFvLnNlY3JldHMuc3ZjLmNsdXN0ZXIubG9jYWyCKG9wZW5i
YW8tYWN0aXZlLnNlY3JldHMuc3ZjLmNsdXN0ZXIubG9jYWyCKW9wZW5iYW8taW50
ZXJuYWwuc2VjcmV0cy5zdmMuY2x1c2VyLmxvY2FsgilvcGVuYmFvLXN0YW5kYnku
c2VjcmV0cy5zdmMuY2x1c3Rlci5sb2NhbIIkb3BlbmJhby11aS5zZWNyZXRzLnN2
Yy5jbHVzdGVyLmxvY2FsMB8GA1UdIwQYMBaAFMEl2Y7rei3PIYOCOhLQ8C5vSx3L
MAoGCCqGSM49BAMCA0cAMEQCIAhmnLs5KN2ygdIoUl/pmxHmzUqSBgGKSZe/N3cJ
UoIfAiBxrrnUl6X04roAFMoTawMiJh/Y6F2+b/lGSC6fWkCCuA==
-----END CERTIFICATE-----
subject=C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-openbao
issuer=C=DE, ST=NRW, L=MyCity, OU=MyOU, emailAddress=myemail@email.com, CN=jonsch-int-man
---
No client certificate CA names sent
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: ecdsa_secp256r1_sha256
Negotiated TLS1.3 group: X25519MLKEM768
---
SSL handshake has read 3218 bytes and written 1726 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: A064C22C6B7C7D823910E9CE0D84FA624AC08999EF1CF63ED47A9D1269BBD233
    Session-ID-ctx:
    Resumption PSK: B24F1D7EA4F1E7D560D668DB538F5A50F4AD38A4D80DC6608315F03392222C22
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - dd 0b b1 59 de 70 3f 7b-b8 0e c4 2c 05 18 ea 07   ...Y.p?{...,....
    0010 - 3f f5 55 d8 3a 73 b9 83-25 54 74 54 2f 59 03 54   ?.U.:s..%TtT/Y.T
    0020 - 31 0d 19 72 fa 59 7b 10-e6 41 cf c5 ff 79 98 e3   1..r.Y{..A...y..
    0030 - 2c 29 43 0f e9 2d 99 54-cc 2a 13 38 bd 64 38 f8   ,)C..-.T.*.8.d8.
    0040 - fc de 3b d5 3e 65 3f 21-14 48 5a 2b da 64 85 af   ..;.>e?!.HZ+.d..
    0050 - ef d2 34 4f 3d 5e c7 cb-d4 fa 9e f6 a6 dd 0d ba   ..4O=^..........
    0060 - 62 67 ce 4c 97 8b 81 a1-4f                        bg.L....O

    Start Time: 1757102730
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

This is the result of openssl s_client -connect localhost:8200 using kubernetes port-forwaring to the active pod.

I will try to get the wireshark by making a development environment with my certificates and test it locally. Could take a bit

[jonsch318]

@cipherboy Here are wireshark packets containing: Client Hello, Change Cipher Spec and Server Hello. In Between and Afterwards are just Encrypted App traffic but if you need more just ask. I am not good in Wireshark, and do not understand any specific values of the tls messages, but maybe this helps

https://gigamove.rwth-aachen.de/en/download/95c79cdef4d93a181f8991c1bd38e540

Password is my github username

[DrDaveD]

You seem to have created the certificate chain yourself and it may be that some of the options you chose are not recognized, although you must have got the CA passed into openssl s_client somehow.

Where your s_client output shows

Peer signature type: ecdsa_secp256r1_sha256
Negotiated TLS1.3 group: X25519MLKEM768
---
SSL handshake has read 3218 bytes and written 1726 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Protocol: TLSv1.3
Server public key is 256 bit
This TLS version forbids renegotiation.

mine (which works) shows

Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2945 bytes and written 383 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 4096 bit
Secure Renegotiation IS NOT supported

Maybe your reverse proxy can't handle the Peer signature type.