enable_tag_override has no effect on the sidecar-proxy service

[dani]

Nomad version

Nomad v1.9.7
BuildDate 2025-03-11T09:07:15Z
Revision f8695974efdbaa9f80ace06f0eec5bd4d2501035+CHANGES

Operating system and Environment details

AlmaLinux 9 with Nomad + Consul (latest versions for both as of today) installed manually from pre-built binaries

Issue

When a job has a service with enable_tag_override = true and is using Consul Connect, the main service is correctly registered in Consul Catalog with ServiceEnableTagOverride = true, but the sidecar-proxy service (the one with the same name as the main service with -sidecar-proxy suffix) doesn't have it

Reproduction steps

Here's a simple job to reproduce

job "test" {
  group "test" {
    network {
      mode = "bridge"
    }
    service {
      name = "test"
      enable_tag_override = true
      connect {
        sidecar_service {}
      }
    }
    task "test" {
      driver = "docker"
      config {
        image = "busybox"
        command = "sleep"
        args = ["infinity"]
      }
      resources {
        cpu = 10
        memory = 10
      }
    }
  }
}

Submit this job, then query Consul Catalog

curl --cacert ca.crt --cert cli.crt --key cli.key -H "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" https://consul.service.consul:8501/v1/catalog/service/test | jq '.[] | .ServiceEnableTagOverride'

This gives the expected output true
Now do the same for the sidecar-proxy service

curl --cacert ca.crt --cert cli.crt --key cli.key -H "X-Consul-Token: ${CONSUL_HTTP_TOKEN}" https://consul.service.consul:8501/v1/catalog/service/test-sidecar-proxy | jq '.[] | .ServiceEnableTagOverride'

This gives the unexpected output false

Expected Result

The sidecar-proxy should also be registered with ServiceEnableTagOverride = true

Actual Result

The sidecar-proxy is registered with ServiceEnableTagOverride = false

This is problematic when you want to create a service subset exposed in the mesh, based on tags. Eg, running a valkey repica, I want to connect only to instances the the primary tag. Randomly, connection from clients to the primary valkey is dropped (or not able to established) because the anti-entropy mecanism remove my custom tag from the sidecar-proxy service.

Updating the sidecar-proxy service manually setting ServiceEnableTagOverride = true is not a solution, as it's reverted to false automatically (I guess Nomad reconciliation is responsible)

[kochen]

Great sum-up @dani

I can confirm the issue also on older versions of Nomad (v1.9.3).

[jrasell]

Hi @dani and thanks for raising this issue. I have been able to reproduce it using the details you provided and will move this issue onto our backlog.