Refreshing app.toml and config.toml

[mcamou]

I need to make some changes to app.toml and config.toml. I make the changes to the manifest and apply it. That ends up in the appropriate ConfigMap. However, the pod does not see the changes. Even restarting the pod does not update the files. The only way I've found of refreshing it is deleting the PVC, which is bad since the node will have to resync (and if this is a full archive node and the rest of the nodes are pruned, you lose history). I'm wondering if I'm missing something.

We've also added a new key to our app.toml, optimistic-execution-enabled with a bool value. I've added it to nodes[0].config.override."app.toml" but it doesn't show up in the ConfigMap. Do I have to do something special for it to be applied?

[helder-moreira]

I need to make some changes to app.toml and config.toml. I make the changes to the manifest and apply it. That ends up in the appropriate ConfigMap. However, the pod does not see the changes. Even restarting the pod does not update the files. The only way I've found of refreshing it is deleting the PVC, which is bad since the node will have to resync (and if this is a full archive node and the rest of the nodes are pruned, you lose history). I'm wondering if I'm missing something.

We've also added a new key to our app.toml, optimistic-execution-enabled with a bool value. I've added it to nodes[0].config.override."app.toml" but it doesn't show up in the ConfigMap. Do I have to do something special for it to be applied?

@mcamou when you say you make changes to the manifest, you mean using .config.override in either ChainNode ou ChainNodeSet right?

What kind of fields are you changing? Changes to peers for example, do not cause restarts. Can you share logs from cosmopilot? This might be related to a bug in disruption check mechanism which I have found this week.

[mcamou]

@helder-moreira Yes, via config.overrides in a ChainNodeSet. I'm trying to add optimistic-execution-enabled = true to app.toml. It's in the ConfigMap but it's not showing up in /home/app/config/app.toml. For more detail, the nodes were already deployed with some overrides, I added an additional field to config.override in 2 nodes in a ChainSet, the field showed up in the ConfigMap but not in app.toml.

The logs from the last time I did that change have expired and we don't yet have Loki set up in that cluster, so I can't get them to you :-(. Earlier today I made a different change (added blockThreshold: 30s) and k8s had some issue restarting the pod, I deleted it manually and it got created again. Here's the event log:

error killing pod: [failed to "KillContainer" for "gopherd" with KillContainerError: "rpc error: code = DeadlineExceeded desc = an error occurs during waiting for container \"fd5132f020a94c15944e55eceacf4612ffe02f88d5eaa2533172457848b2b4a2\" to be killed: wait container \"fd5132f020a94c15944e55eceacf4612ffe02f88d5eaa2533172457848b2b4a2\": context deadline exceeded", failed to "KillPodSandbox" for "2fe8504f-0c85-424b-adb2-2664759c026c" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"]

Here's the Cosmopilot error log from when the error occurred:

{"level":"info","ts":"2025-12-18T10:41:10Z","msg":"starting reconcile","controller":"chainnode","controllerGroup":"apps.k8s.nibiru.org","controllerKind":"ChainNode","ChainNode":{"name":"gopher-testnet-public-rpc-0","namespace":"gopher-testnet"},"namespace":"gopher-testnet","name":"gopher-testnet-public-rpc-0","reconcileID":"78a7ac7f-7a99-44eb-9986-771c030e64eb"}
{"level":"info","ts":"2025-12-18T10:41:10Z","msg":"pre-checking pod status","controller":"chainnode","controllerGroup":"apps.k8s.nibiru.org","controllerKind":"ChainNode","ChainNode":{"name":"gopher-testnet-public-rpc-0","namespace":"gopher-testnet"},"namespace":"gopher-testnet","name":"gopher-testnet-public-rpc-0","reconcileID":"78a7ac7f-7a99-44eb-9986-771c030e64eb","pod-running":true}
{"level":"info","ts":"2025-12-18T10:41:10Z","msg":"loaded configs from cache","controller":"chainnode","controllerGroup":"apps.k8s.nibiru.org","controllerKind":"ChainNode","ChainNode":{"name":"gopher-testnet-public-rpc-0","namespace":"gopher-testnet"},"namespace":"gopher-testnet","name":"gopher-testnet-public-rpc-0","reconcileID":"78a7ac7f-7a99-44eb-9986-771c030e64eb","version":"main-20251217174448"}
{"level":"error","ts":"2025-12-18T10:41:10Z","msg":"Reconciler error","controller":"chainnode","controllerGroup":"apps.k8s.nibiru.org","controllerKind":"ChainNode","ChainNode":{"name":"gopher-testnet-public-rpc-0","namespace":"gopher-testnet"},"namespace":"gopher-testnet","name":"gopher-testnet-public-rpc-0","reconcileID":"78a7ac7f-7a99-44eb-9986-771c030e64eb","error":"failed to update latest height for gopher-testnet-public-rpc-0: Get \"http://gopher-testnet-public-rpc-0-internal.gopher-testnet.svc.cluster.local:8000/latest_height\": dial tcp 10.43.80.149:8000: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}