[QUESTION] V0/V1 configuration issue in Kubernetes

[indrif]

I've fully implemented use of hatchet locally with typescript and hatchet-lite (compose) and everything is working fine. I then tried to deploy the code to Kubernetes by following your documentation.

First of all, the documentation for Kubernetes deployment using Helm is very sparse, to say the least. It is missing a lot of information that I had to dig out from the source code to find out. Specifically regarding all configuration. The default behavior is that Hatchet wants to use Rabbitmq which I didn't find a way to turn off. After reading the source code I found the flag SERVER_TASKQUEUE_KIND=postgres that solved the issue. The actual flag (SERVER_TASKQUEUE_KIND) is documented but its different values and what they do is not. Also virtually all env variables are listed, with their default value, but without further explanation on what they do and when to use them. Most of them are self explanatory but far from all.

Anyway, after a lot of experimentation I finally got my working code to run along a deployed Hatchet setup in my cluster.

The application prints this log at startup:

🪓 1 | 01/28/26, 07:06:30 PM  [WARN/client-init] 🚨⚠️‼️ YOU ARE USING A V0 ENGINE WITH A V1 SDK, WHICH IS NOT SUPPORTED. PLEASE UPGRADE YOUR ENGINE TO V1.🚨⚠️‼️ undefined

I checked the source code in ts and found that it checks if the tenant is V1:

But I dont know exactly why or what information it is looking at. I checked the token it is using and that token points to a Tenant in the db which is V1=true.

But it's complaining about the engine, not the sdk. So I guess the engine maybe is missing some config or something? There is no logs whatsoever in the engine that I can see about v0/v1.

Any help would be appreciated!

These are the basic configuration that we have set:

grpcBroadcastAddress: "myrelease-engine.myns.svc.cluster.local:7070"
env:
      DATABASE_LOGGER_LEVEL: "info"
      SERVER_ADDITIONAL_LOGGERS_PGXSTATS_LEVEL: "info"
      SERVER_ADDITIONAL_LOGGERS_QUEUE_LEVEL: "info"
      SERVER_ALLOW_CHANGE_PASSWORD: "false"
      SERVER_ALLOW_CREATE_TENANT: "false"
      SERVER_ALLOW_INVITES: "false"
      SERVER_ALLOW_SIGNUP: "false"
      SERVER_DEFAULT_ENGINE_VERSION: "V1"
      SERVER_GRPC_BIND_ADDRESS: "0.0.0.0"
      SERVER_GRPC_INSECURE: "t"
      SERVER_LOGGER_LEVEL: "info"
      SERVER_TASKQUEUE_KIND: "postgres"
      # Located in .secrets and is replaced with real value
      DATABASE_URL: unset-database-url

[indrif]

This is the token:

{
  "aud": "...",
  "exp": 4923226779,
  "grpc_broadcast_address": "myrelease-engine.myns.svc.cluster.local:7070",
  "iat": 1769626779,
  "iss": "...",
  "server_url": "...",
  "sub": "707d0855-80ab-4e1f-a156-f1c4546cbf52",
  "token_id": "b691d00c-74d8-4e97-b825-070e672db0f8"
}

Looking at the Tenant in db with id=sub:

[indrif]

Any help here would be greatly appreciated! 🙏

[abelanger5]

Hi @indrif, thanks for the report! A few things to address:

it's complaining about the engine, not the sdk. So I guess the engine maybe is missing some config or something? There is no logs whatsoever in the engine that I can see about v0/v1.

This is indeed strange. The engine uses the version field stored on the Tenant model in the database when it starts up, unless you're running such an old version of the engine that it wouldn't support that value (which would be nearly 10 months old). Could you share the image version you're using for each of the Hatchet components you're running in the namespace? Something like:

kubectl get pods -n cloud -o custom-columns="POD:.metadata.name,IMAGE:.spec.containers[*].image"

First of all, the documentation for Kubernetes deployment using Helm is very sparse, to say the least. It is missing a lot of information that I had to dig out from the source code to find out

This is very much on our roadmap and we've started to document newer features and flags much more in depth. We'll definitely take a closer look here and try to get some PRs up over the next few weeks.

[indrif]

Here are the images used by the helm chart:

POD                                                    IMAGE
hatchet-development--ggyb0vhbdn-worker-token-75jpq     ghcr.io/hatchet-dev/hatchet/hatchet-admin:v0.71.0
hatchet-development--ggyb0vhbdn-worker-token-kp6x7     ghcr.io/hatchet-dev/hatchet/hatchet-admin:v0.71.0
hatchet-development--itbbysywpl-pq9ll                  ghcr.io/hatchet-dev/hatchet/hatchet-admin:v0.71.0
hatchet-development-elastx-api-6f8bcf47f7-69hrk        ghcr.io/hatchet-dev/hatchet/hatchet-api:v0.71.0
hatchet-development-elastx-api-6f8bcf47f7-bpqpf        ghcr.io/hatchet-dev/hatchet/hatchet-api:v0.71.0
hatchet-development-elastx-engine-688d556b5f-47vzf     ghcr.io/hatchet-dev/hatchet/hatchet-engine:v0.71.0
hatchet-development-elastx-frontend-5b76bb7b9c-xcdmb   ghcr.io/hatchet-dev/hatchet/hatchet-frontend:v0.71.0

[indrif]

First of all, the documentation for Kubernetes deployment using Helm is very sparse, to say the least. It is missing a lot of information that I had to dig out from the source code to find out

This is very much on our roadmap and we've started to document newer features and flags much more in depth. We'll definitely take a closer look here and try to get some PRs up over the next few weeks.

Thanks, appreciate it.

[indrif]

Looking at the released packages here on github the latest version seems to be 0.77 so this seems to be a bit behind, right?

[indrif]

We'll perform some testing today with the upgraded version 0.77.37 and see how that works out.

[indrif]

Tried again now with the latest version, still the same problem. Do I need to reset my database after an upgrade to 0.77?

[indrif]

To further explain the issue with the Hatchet helm chart:

This is how it works when we deploy the hatchet helm chart:

0. Hatchet engine and api is started but fails due to no hatchet-client-config yet created. They keep restarting over and over waiting for the secret.

1. A worker-token job is started. The job fails with the following:

setup-worker-token 2026-02-03T10:59:47.089Z INF primary database instance timezone verified: Etc/UTC service=database                                                                                                                           
setup-worker-token Error running [create-worker-token]:could not create session store: at least one cookie secret must be provided, and must provide an even number of secrets 

2. Job is automatically restarted, it now succeeds:

setup-worker-token 2026-02-03T10:59:58.799Z INF primary database instance timezone verified: Etc/UTC service=database                                                                                                                           
setup-worker-token 2026/02/03 10:59:58 cleaning up server config

3. Another job is started for migration and setup. It is completed with the following output (migrations already run):

migration-job 2026/02/03 10:59:47 goose: no migrations to run. current version: 20260129222535                                                                                                                                                 
seed-job Generating config files ./generated                                                                                                                                                                                                   
seed-job 2026-02-03T10:59:48.713Z INF primary database instance timezone verified: Etc/UTC service=database                                                                                                                                    
check-db-connection  ?column?                                                                                                                                                                                                                  
check-db-connection ----------                                                                                                                                                                                                                 
check-db-connection         1                                                                                                                                                                                                                  
check-db-connection (1 row)                                                                                                                                                                                                                    
check-db-connection 

After this the hatchet-client-config is created or updated. Hatchet api and engine finally starts.

Our other workload is then started. Sometimes I get an error (invalid auth) in our ts sdk. This happens every time we deploy when the secret already exists. I dont know exactly what happens in these cases but I guess it creates a new token and updates the secret. Sometimes it works, sometimes it doesnt.

In the case where it doesn't work I have to do the following:

• Restart engine and let it rollout again
• After engine is running, restart my other workloads to get them to fully connect to Hatchet

It seems that I am missing something here.

Regarding helm chart structure. Wouldn't it be advisable to run the jobs as helm hooks to force them to run before the deployments of engine and api? I think this would increase the resiliency in the deploy of hatchet and prevent stuff from starting before client config has been properly setup.