tunnel_kubernetes doesnt honor the exec block

[the-mentor]

It seems like when I use the code to connect to kubernetes.

ephemeral "tunnel_kubernetes" "service" {
  service_name = "service"
  namespace    = "testns"
  target_port  = 3000
  kubernetes = {
    host                   = module.kubeconfig.cluster_endpoint
    cluster_ca_certificate = module.kubeconfig.b64_ca_data
    exec = {
      api_version = module.kubeconfig.eks_exec.api_version
      command     = module.kubeconfig.eks_exec.command
      args        = module.kubeconfig.eks_exec.args
    }
  }
}

here is an example of the args info

args = {
      + api_version = "client.authentication.k8s.io/v1"
      + args        = [
          + "eks",
          + "get-token",
          + "--cluster-name",
          + "eks-cluster",
          + "--output",
          + "json",
          + "--profile",
          + "testprofile",
          + "--region",
          + "us-east-1",
        ]
      + command     = "aws"
    }

When I use the config_path and config_context similar to this it works

  kubernetes = {
    config_path    = "~/.kube/config"
    config_context = "minikube"
  }

[the-mentor]

After some additional troubleshooting I think the issue is related to how the provider runs the k8s args section
When I get the token manually using an external data resource it works as expected

Here is the example code

data "external" "k8s_token" {
  program = [
    "aws",
    "eks", "get-token",
    "--cluster-name", "eks-cluster",
    "--profile", "testprofile",
    "--region", "us-east-1",
    "--output", "json",
    "--query", "status"
  ]
}

ephemeral "tunnel_kubernetes" "service" {
  service_name = "service"
  namespace    = "testns"
  target_port  = 3000
  kubernetes = {
    host                   = module.kubeconfig.cluster_endpoint
    cluster_ca_certificate = module.kubeconfig.b64_ca_data
    token                  = data.external.k8s_token.result.token
  }
}

[the-mentor]

hi @arnaud-dfns any idea what can cause this ?

[arnaud-dfns]

Hi @the-mentor, thanks for your report. Will look into it

[arnaud-dfns]

Can you provide the error output and the tunnel logs outputs? I have a working setup on my side calling exec with eks

[arnaud-dfns]

@the-mentor here is my testing setup

data "aws_instance" "bastion" {
  filter {
    name   = "tag:Name"
    values = ["bastion-instance"]
  }
}

data "aws_eks_cluster" "eks" {
  name = "eks-cluster"
}

ephemeral "tunnel_ssm" "eks" {
  target_host  = replace(data.aws_eks_cluster.eks.endpoint, "https://", "")
  target_port  = 443
  ssm_instance = data.aws_instance.bastion.id
}

ephemeral "tunnel_kubernetes" "coredns" {
  service_name = "kube-dns"
  namespace    = "kube-system"
  target_port  = 53
  kubernetes = {
    host                   = "https://${ephemeral.tunnel_ssm.eks.local_host}:${ephemeral.tunnel_ssm.eks.local_port}"
    tls_server_name        = lower(replace(data.aws_eks_cluster.eks.endpoint, "https://", ""))
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args = [
        "eks", "get-token",
        "--region", data.aws_eks_cluster.eks.region,
        "--cluster-name", data.aws_eks_cluster.eks.name
      ]
    }
  }
}

Provider logs /var/logs/k8s-tunnel-kube-system-kube-dns-53.log:

2025/12/17 12:19:03 starting tunnel: localhost:65301 -> kube-system/kube-dns:53
2025/12/17 12:19:04 forwarding to pod: coredns-84c8545b57-rphs6
Forwarding from 127.0.0.1:65301 -> 53
Forwarding from [::1]:65301 -> 53
2025/12/17 12:19:04 port forwarding is ready
2025/12/17 12:19:05 stopping tunnel: received interrupt signal

To be noted that I had an issue at first because my default kubeconfig was invalid

[the-mentor]

@arnaud-dfns make sure you don't have the cluster and context in your default kubeconfig file since iv seen it fall back to that in my testing.

Edit:I suggest you use a blank default kubeconfig for the test

Edit2: the log just say "Unauthorized" if I'm not mistaken

[arnaud-dfns]

Tried it again without any shell env and an empty kube config:

provider "aws" {
  region  = "eu-west-3"
  profile = "DevProfile"
}

data "aws_instance" "bastion" {
  filter {
    name   = "tag:Name"
    values = ["bastion-instance"]
  }
}

data "aws_eks_cluster" "eks" {
  name = "eks-cluster"
}

ephemeral "tunnel_ssm" "eks" {
  target_host  = replace(data.aws_eks_cluster.eks.endpoint, "https://", "")
  target_port  = 443
  ssm_instance = data.aws_instance.bastion.id
  ssm_profile  = "DevProfile"
  ssm_region   = data.aws_eks_cluster.eks.region
}

ephemeral "tunnel_kubernetes" "coredns" {
  service_name = "kube-dns"
  namespace    = "kube-system"
  target_port  = 53
  kubernetes = {
    host                   = "https://${ephemeral.tunnel_ssm.eks.local_host}:${ephemeral.tunnel_ssm.eks.local_port}"
    tls_server_name        = lower(replace(data.aws_eks_cluster.eks.endpoint, "https://", ""))
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
    exec = {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      args = [
        "eks", "get-token",
        "--profile", "DevProfile",
        "--region", data.aws_eks_cluster.eks.region,
        "--cluster-name", data.aws_eks_cluster.eks.name
      ]
    }
  }
}

$ k config view
apiVersion: v1
clusters: null
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null

$ terraform apply
data.aws_eks_cluster.eks: Reading...
data.aws_instance.bastion: Reading...
data.aws_eks_cluster.eks: Read complete after 0s [id=eks-cluster]
data.aws_instance.bastion: Read complete after 2s [id=i-0xxxxxxxxxxxxxxx]
ephemeral.tunnel_ssm.eks: Opening...
ephemeral.tunnel_ssm.eks: Opening complete after 5s
ephemeral.tunnel_kubernetes.coredns: Opening...
ephemeral.tunnel_kubernetes.coredns: Opening complete after 2s
ephemeral.tunnel_ssm.eks: Closing...
ephemeral.tunnel_kubernetes.coredns: Closing...
ephemeral.tunnel_ssm.eks: Closing complete after 0s
ephemeral.tunnel_kubernetes.coredns: Closing complete after 0s

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
ephemeral.tunnel_ssm.eks: Opening...
ephemeral.tunnel_ssm.eks: Opening complete after 5s
ephemeral.tunnel_kubernetes.coredns: Opening...
ephemeral.tunnel_kubernetes.coredns: Opening complete after 2s
ephemeral.tunnel_ssm.eks: Closing...
ephemeral.tunnel_kubernetes.coredns: Closing...
ephemeral.tunnel_ssm.eks: Closing complete after 0s
ephemeral.tunnel_kubernetes.coredns: Closing complete after 0s

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

[the-mentor]

@arnaud-dfns i see that you are using 'tls_server_name' I wonder if it could be related.

Is there a way to enable additional logging for the tunnel?

[the-mentor]

@arnaud-dfns i see that you are using 'tls_server_name' I wonder if it could be related.

Is there a way to enable additional logging for the tunnel?

[arnaud-dfns]

@the-mentor it does not look like it, klog which is used by k8s client does not have any environment configuration. You'd have to compile the provider to enable verbose logs

[the-mentor]

@arnaud-dfns thanks for looking into it I'll try to do more troubleshooting on my end.

It's really strange that using the same commands in the external data resource and token works but this doesn't 🤷‍♂️

[arnaud-dfns]

Maybe something around your shell environment. The aws command spawned by terraform may be missing important context vars

[the-mentor]

I made sure there are no environment variables or anything.
And I call the same command from the external data resource that is also invoked from terraform which is why I don't understand why it works in one and not the other.
I'll have to dig deeper and try to figure out if the issue is on my end.

Maybe something around your shell environment. The aws command spawned by terraform may be missing important context vars

[arnaud-dfns]

As a debugging step, instead of calling aws directly you can create a helper script (aws_debug for example) with some kind of built-in logging to a file