leaf tls cert does not work on rustls

[goenning]

Current and expected behavior

I have a k8s cluster using a Let's Encrypt cert, and when connecting to it via kube-rs it does not with the leaf tls cert.

kube-rs + fullchain cert = OK
kube-rs + intermediate cert = OK
kube-rs + leaf cert = NOK

Error: ServiceError: client error (Connect)

Caused by:
    0: client error (Connect)
    1: invalid peer certificate: UnknownIssuer

kubectl + fullchain cert = OK
kubectl + intermediate cert = OK
kubectl + leaf cert = OK

Possible solution

I'm not sure where the issue is, I'm opening this issue hoping someone will have an idea what could be the issue

Additional context

related aptakube/aptakube#445

Environment

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0

Configuration and features

I was able to replicate this using the HEAD + cargo run --example kubectl -- get nodes

Affected crates

kube-client

Would you like to work on fixing this bug?

yes

[goenning]

Adding a kubeconfig that uses a remote server to repro the bug:

apiVersion: v1
kind: Config
clusters:
- name: bug
  cluster:
    # Leaf cert, only works on kubectl, not kube-rs
    # `Unauthorized` is the expected error, but on kube-rs we get `invalid peer certificate: UnknownIssuer`
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURqRENDQXhLZ0F3SUJBZ0lTQnNQYWZPZ043UDlDYThjSjRiNUtoQk9iTUFvR0NDcUdTTTQ5QkFNRE1ESXgKQ3pBSkJnTlZCQVlUQWxWVE1SWXdGQVlEVlFRS0V3MU1aWFFuY3lCRmJtTnllWEIwTVFzd0NRWURWUVFERXdKRgpOVEFlRncweU5UQTJNRFV4TWpReU16ZGFGdzB5TlRBNU1ETXhNalF5TXpaYU1Cc3hHVEFYQmdOVkJBTVRFR3N6CmN5NXRhVzVwWjJWMGN5NWpiMjB3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVFqWmV6eWlRODUKZytEL0djL3FGY3JabE4wOG1uUTljOHJMNFFoNlNZazN6b2twWUx5M1Rvam9xY0JoWHY1OEJrUE9KSEdLd0dpSQp1QTFLSVh0MW1RRmZvNElDSFRDQ0Foa3dEZ1lEVlIwUEFRSC9CQVFEQWdlQU1CMEdBMVVkSlFRV01CUUdDQ3NHCkFRVUZCd01CQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUFNQjBHQTFVZERnUVdCQlNrNCtwY3VNL1IKRWNsd3czK241U09vOWlSdWpEQWZCZ05WSFNNRUdEQVdnQlNmSzEvUFBDRlBuUVMzN1Nzc3hNWndpOUxYRFRBeQpCZ2dyQmdFRkJRY0JBUVFtTUNRd0lnWUlLd1lCQlFVSE1BS0dGbWgwZEhBNkx5OWxOUzVwTG14bGJtTnlMbTl5Clp5OHdHd1lEVlIwUkJCUXdFb0lRYXpOekxtMXBibWxuWlhSekxtTnZiVEFUQmdOVkhTQUVEREFLTUFnR0JtZUIKREFFQ0FUQXRCZ05WSFI4RUpqQWtNQ0tnSUtBZWhoeG9kSFJ3T2k4dlpUVXVZeTVzWlc1amNpNXZjbWN2TWpndQpZM0pzTUlJQkF3WUtLd1lCQkFIV2VRSUVBZ1NCOUFTQjhRRHZBSFVBN1R4TDF1Z0d3cVNpQUZmYnl5VGlPQUhmClVTL3R4SWJGY0E4ZzNiYytQK0FBQUFHWFFGTDA3Z0FBQkFNQVJqQkVBaUFmL0c4UmlrdEs3YkZUMGZKYnlidTkKdGt0V3MrY2xyWnJZL3QwS2hCdWV4QUlnUndQS0dKYkZzWGFnTnMxcGtKdE1sWnlsMC9pblhTcTRJSG1HVjFyMwpOdHdBZGdBUzhVNDB2Vk55VElRR0djT1BQM29UK09lMVlvZUluRzB3QllUcjVZWW1PZ0FBQVpkQVV2ek1BQUFFCkF3QkhNRVVDSUYvTWp3cHlZUkpjai9MazZRUzJFR2xtNkZFS2FpUU8rNzNtZHBFUGlZWDRBaUVBcWFzSVdlY00KS0hDNHJxbit1N2tGaVJ2eEJZU1N2L1ZxZ3BNandOVGhDV013Q2dZSUtvWkl6ajBFQXdNRGFBQXdaUUl4QU1QYwpHWk4vMmFveGJ3ME8ydGZNaEVIcEppdjEzZm04d0pNZmZ6VjgwR0x1Y1lWU3pJTEh0bVZlVHRhb00zN2RhZ0l3Clc2NU1TSVJ1U3c3OGVoNlkyN3EyNjJodjJUdSt1UzhOQnRodmpueHN3eHRwbXU3RERqb0xoOU05TDlLM2syaGsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==

    # Intermediate cert, works on both
    # `Unauthorized` is the expected error
    # certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVWekNDQWorZ0F3SUJBZ0lSQUlPUGJHUE9zVG1NWWdaaWd4WEovZDR3RFFZSktvWklodmNOQVFFTEJRQXcKVHpFTE1Ba0dBMVVFQmhNQ1ZWTXhLVEFuQmdOVkJBb1RJRWx1ZEdWeWJtVjBJRk5sWTNWeWFYUjVJRkpsYzJWaApjbU5vSUVkeWIzVndNUlV3RXdZRFZRUURFd3hKVTFKSElGSnZiM1FnV0RFd0hoY05NalF3TXpFek1EQXdNREF3CldoY05NamN3TXpFeU1qTTFPVFU1V2pBeU1Rc3dDUVlEVlFRR0V3SlZVekVXTUJRR0ExVUVDaE1OVEdWMEozTWcKUlc1amNubHdkREVMTUFrR0ExVUVBeE1DUlRVd2RqQVFCZ2NxaGtqT1BRSUJCZ1VyZ1FRQUlnTmlBQVFOQ3pxSwphMkdPdHUvY1gxam54a0pGVkt0ajltWmhTQW91V1hXMGdRSTNVTGMvRm5uY21PeWhLSmR5SUJ3c3o5VjhVaUJPClZIaGJoQlJyd0pDdWhlekFVVUU4V29kL0JrM1UvbURSK213dDRYMlZFSWlpQ0ZRUG1ScE01dW9Lck5pamdmZ3cKZ2ZVd0RnWURWUjBQQVFIL0JBUURBZ0dHTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQ0JnZ3JCZ0VGQlFjRApBVEFTQmdOVkhSTUJBZjhFQ0RBR0FRSC9BZ0VBTUIwR0ExVWREZ1FXQkJTZksxL1BQQ0ZQblFTMzdTc3N4TVp3Cmk5TFhEVEFmQmdOVkhTTUVHREFXZ0JSNXRGbm1lN2JsNUFGemdBaUl5QnBZOXVtYmJqQXlCZ2dyQmdFRkJRY0IKQVFRbU1DUXdJZ1lJS3dZQkJRVUhNQUtHRm1oMGRIQTZMeTk0TVM1cExteGxibU55TG05eVp5OHdFd1lEVlIwZwpCQXd3Q2pBSUJnWm5nUXdCQWdFd0p3WURWUjBmQkNBd0hqQWNvQnFnR0lZV2FIUjBjRG92TDNneExtTXViR1Z1ClkzSXViM0puTHpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQWdFQUgzS2RORVZDUWRxazBMS3l1TkltVEtkUkpZMUMKMnV3MlNKYWp1aHFreUdQWThDK3p6c3VmWittZ25obnExQTJLVlFPU3lrT0VuVWJ4MWN5NjM3ckJBaWh4OTdyKwpiY3diWk02c1RESWFFcmlSL1BMazZMS3M5QmUwdW9WeGdPS0RjcEc5c3ZEMzNKK0c5TGNmdjFLOWx1RG1TVGdHCjZYTkZJTjV2Zkk1Z3MvbE1QeW9qRU1kSXpLOWJsY2wyLzF2S3hPOFdHQ2NqdnNRMW5KL1B3dDhMUVpCZk9GeVYKWFA4dWJBcC9hdTNkYzRFS1dHOU1PNXpjeDFxVDkrTlhSR2RWV3hHdm1CRlJBYWpjaU1mWE1FMVp1R21rMy9HTwprb0FNN1pralptbGV5b2tQMUxHem1mSmNVZDlzN2VldTEvOS9lZzVYbFhkLzU1R3RZakFNK0M0REc1aTdlYU5xCmNtMkYreXhZSVB0NmNiYnRZVk5KQ0dmSFdxSEVRNEZZU3RVeUZudjhzanlxVTh5cGdaYU5KOWFWY1dTSUNMT0kKRTEvUXYvN29Lc25aQ1dKOTI2d1U2UnFHMU9ZUEdPaTF6dUFCaEx3NjFjdVBWRFQyOG5RUy9lNno5NWNKWHEwZQpLMUJjYUo2Zkpac21ialJnRDVwM212RWY1dmRRTTdNQ0V2VTB0SGJzeDJJNW1ISEpvQUJIYjhLVkJnV3AvbGNYCkdXaVdhZU95QjdSUCtPZkR0dmkyT3NhcHhYaVY3dk5WczdmTWxyUmpZMWpvS2FxbW15Y25CdkFxMTRBRWJ0eUwKc1ZmT1M2NkI4YXBrZUZYMk5ZNFhQRVlWNFpTQ2U4VkhQcmRyRVJrMndJTEczVC9FR21TSWtDWVZVTVNuam1KZApWUUQ5RjZOYS8rem1YQ2M9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0=

    # Full chain, works on both
    # `Unauthorized` is the expected error
    # certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURqRENDQXhLZ0F3SUJBZ0lTQnNQYWZPZ043UDlDYThjSjRiNUtoQk9iTUFvR0NDcUdTTTQ5QkFNRE1ESXgKQ3pBSkJnTlZCQVlUQWxWVE1SWXdGQVlEVlFRS0V3MU1aWFFuY3lCRmJtTnllWEIwTVFzd0NRWURWUVFERXdKRgpOVEFlRncweU5UQTJNRFV4TWpReU16ZGFGdzB5TlRBNU1ETXhNalF5TXpaYU1Cc3hHVEFYQmdOVkJBTVRFR3N6CmN5NXRhVzVwWjJWMGN5NWpiMjB3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVFqWmV6eWlRODUKZytEL0djL3FGY3JabE4wOG1uUTljOHJMNFFoNlNZazN6b2twWUx5M1Rvam9xY0JoWHY1OEJrUE9KSEdLd0dpSQp1QTFLSVh0MW1RRmZvNElDSFRDQ0Foa3dEZ1lEVlIwUEFRSC9CQVFEQWdlQU1CMEdBMVVkSlFRV01CUUdDQ3NHCkFRVUZCd01CQmdnckJnRUZCUWNEQWpBTUJnTlZIUk1CQWY4RUFqQUFNQjBHQTFVZERnUVdCQlNrNCtwY3VNL1IKRWNsd3czK241U09vOWlSdWpEQWZCZ05WSFNNRUdEQVdnQlNmSzEvUFBDRlBuUVMzN1Nzc3hNWndpOUxYRFRBeQpCZ2dyQmdFRkJRY0JBUVFtTUNRd0lnWUlLd1lCQlFVSE1BS0dGbWgwZEhBNkx5OWxOUzVwTG14bGJtTnlMbTl5Clp5OHdHd1lEVlIwUkJCUXdFb0lRYXpOekxtMXBibWxuWlhSekxtTnZiVEFUQmdOVkhTQUVEREFLTUFnR0JtZUIKREFFQ0FUQXRCZ05WSFI4RUpqQWtNQ0tnSUtBZWhoeG9kSFJ3T2k4dlpUVXVZeTVzWlc1amNpNXZjbWN2TWpndQpZM0pzTUlJQkF3WUtLd1lCQkFIV2VRSUVBZ1NCOUFTQjhRRHZBSFVBN1R4TDF1Z0d3cVNpQUZmYnl5VGlPQUhmClVTL3R4SWJGY0E4ZzNiYytQK0FBQUFHWFFGTDA3Z0FBQkFNQVJqQkVBaUFmL0c4UmlrdEs3YkZUMGZKYnlidTkKdGt0V3MrY2xyWnJZL3QwS2hCdWV4QUlnUndQS0dKYkZzWGFnTnMxcGtKdE1sWnlsMC9pblhTcTRJSG1HVjFyMwpOdHdBZGdBUzhVNDB2Vk55VElRR0djT1BQM29UK09lMVlvZUluRzB3QllUcjVZWW1PZ0FBQVpkQVV2ek1BQUFFCkF3QkhNRVVDSUYvTWp3cHlZUkpjai9MazZRUzJFR2xtNkZFS2FpUU8rNzNtZHBFUGlZWDRBaUVBcWFzSVdlY00KS0hDNHJxbit1N2tGaVJ2eEJZU1N2L1ZxZ3BNandOVGhDV013Q2dZSUtvWkl6ajBFQXdNRGFBQXdaUUl4QU1QYwpHWk4vMmFveGJ3ME8ydGZNaEVIcEppdjEzZm04d0pNZmZ6VjgwR0x1Y1lWU3pJTEh0bVZlVHRhb00zN2RhZ0l3Clc2NU1TSVJ1U3c3OGVoNlkyN3EyNjJodjJUdSt1UzhOQnRodmpueHN3eHRwbXU3RERqb0xoOU05TDlLM2syaGsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRVZ6Q0NBaitnQXdJQkFnSVJBSU9QYkdQT3NUbU1ZZ1ppZ3hYSi9kNHdEUVlKS29aSWh2Y05BUUVMQlFBdwpUekVMTUFrR0ExVUVCaE1DVlZNeEtUQW5CZ05WQkFvVElFbHVkR1Z5Ym1WMElGTmxZM1Z5YVhSNUlGSmxjMlZoCmNtTm9JRWR5YjNWd01SVXdFd1lEVlFRREV3eEpVMUpISUZKdmIzUWdXREV3SGhjTk1qUXdNekV6TURBd01EQXcKV2hjTk1qY3dNekV5TWpNMU9UVTVXakF5TVFzd0NRWURWUVFHRXdKVlV6RVdNQlFHQTFVRUNoTU5UR1YwSjNNZwpSVzVqY25sd2RERUxNQWtHQTFVRUF4TUNSVFV3ZGpBUUJnY3Foa2pPUFFJQkJnVXJnUVFBSWdOaUFBUU5DenFLCmEyR090dS9jWDFqbnhrSkZWS3RqOW1aaFNBb3VXWFcwZ1FJM1VMYy9Gbm5jbU95aEtKZHlJQndzejlWOFVpQk8KVkhoYmhCUnJ3SkN1aGV6QVVVRThXb2QvQmszVS9tRFIrbXd0NFgyVkVJaWlDRlFQbVJwTTV1b0tyTmlqZ2ZndwpnZlV3RGdZRFZSMFBBUUgvQkFRREFnR0dNQjBHQTFVZEpRUVdNQlFHQ0NzR0FRVUZCd01DQmdnckJnRUZCUWNECkFUQVNCZ05WSFJNQkFmOEVDREFHQVFIL0FnRUFNQjBHQTFVZERnUVdCQlNmSzEvUFBDRlBuUVMzN1Nzc3hNWncKaTlMWERUQWZCZ05WSFNNRUdEQVdnQlI1dEZubWU3Ymw1QUZ6Z0FpSXlCcFk5dW1iYmpBeUJnZ3JCZ0VGQlFjQgpBUVFtTUNRd0lnWUlLd1lCQlFVSE1BS0dGbWgwZEhBNkx5OTRNUzVwTG14bGJtTnlMbTl5Wnk4d0V3WURWUjBnCkJBd3dDakFJQmdabmdRd0JBZ0V3SndZRFZSMGZCQ0F3SGpBY29CcWdHSVlXYUhSMGNEb3ZMM2d4TG1NdWJHVnUKWTNJdWIzSm5MekFOQmdrcWhraUc5dzBCQVFzRkFBT0NBZ0VBSDNLZE5FVkNRZHFrMExLeXVOSW1US2RSSlkxQwoydXcyU0phanVocWt5R1BZOEMrenpzdWZaK21nbmhucTFBMktWUU9TeWtPRW5VYngxY3k2MzdyQkFpaHg5N3IrCmJjd2JaTTZzVERJYUVyaVIvUExrNkxLczlCZTB1b1Z4Z09LRGNwRzlzdkQzM0orRzlMY2Z2MUs5bHVEbVNUZ0cKNlhORklONXZmSTVncy9sTVB5b2pFTWRJeks5YmxjbDIvMXZLeE84V0dDY2p2c1ExbkovUHd0OExRWkJmT0Z5VgpYUDh1YkFwL2F1M2RjNEVLV0c5TU81emN4MXFUOStOWFJHZFZXeEd2bUJGUkFhamNpTWZYTUUxWnVHbWszL0dPCmtvQU03WmtqWm1sZXlva1AxTEd6bWZKY1VkOXM3ZWV1MS85L2VnNVhsWGQvNTVHdFlqQU0rQzRERzVpN2VhTnEKY20yRit5eFlJUHQ2Y2JidFlWTkpDR2ZIV3FIRVE0RllTdFV5Rm52OHNqeXFVOHlwZ1phTko5YVZjV1NJQ0xPSQpFMS9Rdi83b0tzblpDV0o5MjZ3VTZScUcxT1lQR09pMXp1QUJoTHc2MWN1UFZEVDI4blFTL2U2ejk1Y0pYcTBlCksxQmNhSjZmSlpzbWJqUmdENXAzbXZFZjV2ZFFNN01DRXZVMHRIYnN4Mkk1bUhISm9BQkhiOEtWQmdXcC9sY1gKR1dpV2FlT3lCN1JQK09mRHR2aTJPc2FweFhpVjd2TlZzN2ZNbHJSalkxam9LYXFtbXljbkJ2QXExNEFFYnR5TApzVmZPUzY2QjhhcGtlRlgyTlk0WFBFWVY0WlNDZThWSFByZHJFUmsyd0lMRzNUL0VHbVNJa0NZVlVNU25qbUpkClZRRDlGNk5hLyt6bVhDYz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQ==
    server: https://k3s.minigets.com:6443
users:
- name: bug
  user:
    token: INVALID
contexts:
- name: bug
  context:
    cluster: bug
    user: bug
current-context: bug

[goenning]

seems like this the issue is here:

kube/kube-client/src/client/tls.rs

Line 56 in 8d5b223

ClientConfig::builder().with_root_certificates(root_store(certs)?)

where this function only accepts root/intermediate certs, It might be possible to do something like:

ClientConfig::builder().dangerous().with_custom_certificate_verifier(...) when the supplied cert is a leaf cert 🤔

[goenning]

some more experiments:

✅ openssl:

cargo run --example kubectl  --no-default-features --features=openssl-tls,latest,runtime -- get nodes

Error: ApiError: Unauthorized: Unauthorized (ErrorResponse { status: "Failure", message: "Unauthorized", reason: "Unauthorized", code: 401 })

Caused by:
    Unauthorized: Unauthorized

❌ rustls:

cargo run --example kubectl  --no-default-features --features=rustls-tls,latest,runtime -- get nodes

Error: ServiceError: client error (Connect)

Caused by:
    0: client error (Connect)
    1: invalid peer certificate: UnknownIssuer