Client panics after many allocations are stopped

[msherman13]

Nomad version

Nomad v1.10.0
BuildDate 2025-04-09T16:40:54Z
Revision e26a2bd

Operating system and Environment details

Rocky 9.3

Issue

Client crashes after stopping many allocations. This started after upgrading to v1.10.0 from v1.8.4

Reproduction steps

Stop many allocations

Expected Result

Allocations are rescheduled

Actual Result

Client panics

Nomad Client logs (if appropriate)

-- VISUAL LINE --������������������������73��������1,1�����������Top
May 06 23:56:59 nomad[3795872]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x21d1d15]����������������������������������May 06 23:56:59 nomad[3795872]: goroutine 1022 [running]:
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*observer).stop(...)���������������������������������������May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:73�����������������������������������May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*checksHook).Update(0xc001262510, 0xc005967c40)
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:256 +0x275���������������������������May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).update(0xc00071d608, 0xc006c84e00)�����������May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner_hooks.go:243 +0x4d7
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdate(0xc00071d608, 0xc006c84e00)May 06 23:56:59 nomad[3795872]:���������github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1050 +0x2b5�������������������������May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdates(...)����������������������May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1031 �������������������������May 06 23:56:59 nomad[3795872]: created by github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run in goroutine 1186�������������May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:380 +0xdf �������������������������May 06 23:56:59 systemd[1]: nomad.service: Main process exited, code=exited, status=2/INVALIDARGUMENT�������������������������������������May 06 23:56:59 systemd[1]: nomad.service: Failed with result 'exit-code'.

[tgross]

Hi @msherman13! It's a little hard to tell what's going on here, as the log output you've presented has a lot of characters that GitHub is presenting as non-printing characters (�) and that don't show up in the plaintext of your post. It would be helpful if you could get me a clean output, and include the lines just before the panic too.

All that being said, I think the panic is coming from checks_hook.go#L73 but the address is non-nil. There's no "panic: runtime error: invalid memory address or nil pointer dereference" message at the top of your stack trace, so that follows. But the only way I see for a segfault here would be for a nil pointer, so that's mysterious. It might be a runtime bug but I don't see any other reports of this.

This is going to take a little bit of help from your end to figure out:

• Is this on amd64 or arm64?
• Does this happen repeatedly? Or was it a one-off event?
• If it does happen repeatedly, we could really use a core dump of the Nomad agent when it happens. You can send that to nomad-oss-debug@hashicorp.com, which is a write-only address that only the Nomad engineering team has access to.

[msherman13]

Hi @tgross ,
Apologies for the malformed logs here, I was admittedly copying this over from my iphone. Firstly, here is the original log output from the client without the formatting issues:

May 06 23:56:59 nomad[3795872]:     2025-05-06T23:56:59.168Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=584b1edb-b3d7-239c-cf88-83b15e5f07db task=<redacted> type="Task Setup" msg="Building Task Directory" failed=false
May 06 23:56:59 nomad[3795872]:  client.alloc_runner.task_runner: Task event: alloc_id=584b1edb-b3d7-239c-cf88-83b15e5f07db task=<redacted> type="Task Setup" msg="Building Task Directory" failed=false
May 06 23:56:59 nomad[3795872]:     2025-05-06T23:56:59.173Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=75a456f1-b455-68db-a9a3-bc2a0c0b3006 task=<redacted> type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
May 06 23:56:59 nomad[3795872]:  client.alloc_runner.task_runner: Task event: alloc_id=75a456f1-b455-68db-a9a3-bc2a0c0b3006 task=<redacted> type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
May 06 23:56:59 nomad[3795872]:     2025-05-06T23:56:59.244Z [INFO]  client.gc: marking allocation for GC: alloc_id=75a456f1-b455-68db-a9a3-bc2a0c0b3006
May 06 23:56:59 nomad[3795872]:  client.gc: marking allocation for GC: alloc_id=75a456f1-b455-68db-a9a3-bc2a0c0b3006
May 06 23:56:59 nomad[3795872]:     2025-05-06T23:56:59.262Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=85bc5ac0-d5b2-c600-61e6-ebab0ce1d0d3 task=<redacted> type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
May 06 23:56:59 nomad[3795872]:  client.alloc_runner.task_runner: Task event: alloc_id=85bc5ac0-d5b2-c600-61e6-ebab0ce1d0d3 task=<redacted> type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
May 06 23:56:59 nomad[3795872]:     2025-05-06T23:56:59.324Z [INFO]  client.gc: garbage collecting allocation: alloc_id=75a456f1-b455-68db-a9a3-bc2a0c0b3006 reason="number of allocations (98) is over the limit (50)"
May 06 23:56:59 nomad[3795872]:  client.gc: garbage collecting allocation: alloc_id=75a456f1-b455-68db-a9a3-bc2a0c0b3006 reason="number of allocations (98) is over the limit (50)"
May 06 23:56:59 nomad[3795872]:     2025-05-06T23:56:59.325Z [INFO]  client.gc: garbage collection skipped because no terminal allocations: reason="number of allocations (98) is over the limit (50)"
May 06 23:56:59 nomad[3795872]:  client.gc: garbage collection skipped because no terminal allocations: reason="number of allocations (98) is over the limit (50)"
May 06 23:56:59 nomad[3795872]: panic: runtime error: invalid memory address or nil pointer dereference 
May 06 23:56:59 nomad[3795872]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x21d1d15]
May 06 23:56:59 nomad[3795872]: goroutine 1022 [running]:
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*observer).stop(...)
May 06 23:56:59 nomad[3795872]:         github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:73
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*checksHook).Update(0xc001262510, 0xc005967c40)
May 06 23:56:59 nomad[3795872]:         github.com/hashicorp/nomad/client/allocrunner/checks_hook.go:256 +0x275
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).update(0xc00071d608, 0xc006c84e00)
May 06 23:56:59 nomad[3795872]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner_hooks.go:243 +0x4d7
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdate(0xc00071d608, 0xc006c84e00)
May 06 23:56:59 nomad[3795872]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1050 +0x2b5
May 06 23:56:59 nomad[3795872]: github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).handleAllocUpdates(...)
May 06 23:56:59 nomad[3795872]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:1031
May 06 23:56:59 nomad[3795872]: created by github.com/hashicorp/nomad/client/allocrunner.(*allocRunner).Run in goroutine 1186
May 06 23:56:59 nomad[3795872]:         github.com/hashicorp/nomad/client/allocrunner/alloc_runner.go:380 +0xdf

Secondly, I was able to work around this issue by clearing my nomad data directory on the client (I had to keep the secret-id so the client could stay in the cluster after restarting).

Below are responses to your specific questions:

Is this on amd64 or arm64? AMD64
Does this happen repeatedly? Or was it a one-off event? It would happen every time I tried to start the client
If it does happen repeatedly, we could really use a core dump of the Nomad agent when it happens. You can send that to nomad-oss-debug@hashicorp.com, which is a write-only address that only the Nomad engineering team has access to. Unfortunately, since I've wiped the data dir I cannot reproduce

[tgross]

Secondly, I was able to work around this issue by clearing my nomad data directory on the client (I had to keep the secret-id so the client could stay in the cluster after restarting).

Very interesting! There's a little boltdb database we keep in the client that we restore the client state from on restart. I'm going to mark this for further investigation.