Allow Agent Reload with API - /v1/agent/reload by benjamin-lykins · Pull Request #27106 · hashicorp/nomad

benjamin-lykins · 2025-11-14T19:19:49Z

Description

Provide an API endpoint to reload agent config without elevated permissions on the infrastructure running Nomad.

Provides ability to reload config remotely.
Leverage Nomad Token + ACL to reload.

Testing

Reloading log level without ACL

curl -k  -X PUT \
  https://localhost:4646/v1/agent/reload
{"message":"agent configuration reloaded"}

Attempting without token (ACL Enabled):

curl -k  -X PUT \  
https://localhost:4646/v1/agent/reload
Permission denied

Attempting with token (ACL Enabled; Agent = Write):

curl -k -X PUT \                                                    
  --header "X-Nomad-Token: 79d77f6a-e168-3bbb-8fa7-e668b2b09346" \
  https://localhost:4646/v1/agent/reload 
{"message":"agent configuration reloaded"}

Attempting with token (ACL Enabled; Agent = Read):

curl -k -X PUT \
  --header "X-Nomad-Token: 2cde0074-d0ac-53e1-a6dc-3f416a8e1737" \
  https://localhost:4646/v1/agent/reload
Permission denied

Invalid Config File:

curl -k -X PUT \
  --header "X-Nomad-Token: 79d77f6a-e168-3bbb-8fa7-e668b2b09346" \
  https://localhost:4646/v1/agent/reload
Error loading /Users/benjamin.lykins/git/nomad/nomad/bin/config/general.hcl: failed to decode HCL file /Users/benjamin.lykins/git/nomad/nomad/bin/config/general.hcl: At 3:16: literal not terminated

Links

Fixes: #24048

Contributor Checklist

Changelog Entry If this PR changes user-facing behavior, please generate and add a
changelog entry using the make cl command.
Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
ensure regressions will be caught.
Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
and job configuration, please update the Nomad website documentation to reflect this. Refer to
the website README for docs guidelines. Please also consider whether the
change requires notes within the upgrade guide.

Reviewer Checklist

Backport Labels Please add the correct backport labels as described by the internal
backporting document.
Commit Type Ensure the correct merge method is selected which should be "squash and merge"
in the majority of situations. The main exceptions are long-lived feature branches or merges where
history should be preserved.
Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
within the public repository.

If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

tgross

@benjamin-lykins there's a failing test not touched here but clearly related to this work. Let's get that resolved before we review:

=== FAIL: command/agent TestServer_Reload_TLS_Certificate (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x30fefa3]

goroutine 87962 [running]:
testing.tRunner.func1.2({0x3898300, 0x69a5e70})
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1872 +0x237
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1875 +0x35b
panic({0x3898300?, 0x69a5e70?})
	/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/panic.go:783 +0x132
github.com/hashicorp/nomad/command/agent.(*Agent).Reload(0xc00c83c700, 0xc00d476d08)
	/home/runner/work/nomad/nomad/command/agent/agent.go:1533 +0x63
github.com/hashicorp/nomad/command/agent.TestServer_Reload_TLS_Certificate(0xc00a9f8540)
	/home/runner/work/nomad/nomad/command/agent/agent_test.go:1197 +0x1d7
testing.tRunner(0xc00a9f8540, 0x42c8720)
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1997 +0x465

DONE 5542 tests, 12 skipped, 4 failures in 182.433s

benjamin-lykins · 2025-11-17T17:06:10Z

@benjamin-lykins there's a failing test not touched here but clearly related to this work. Let's get that resolved before we review:

=== FAIL: command/agent TestServer_Reload_TLS_Certificate (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x30fefa3]

goroutine 87962 [running]:
testing.tRunner.func1.2({0x3898300, 0x69a5e70})
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1872 +0x237
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1875 +0x35b
panic({0x3898300?, 0x69a5e70?})
	/opt/hostedtoolcache/go/1.25.3/x64/src/runtime/panic.go:783 +0x132
github.com/hashicorp/nomad/command/agent.(*Agent).Reload(0xc00c83c700, 0xc00d476d08)
	/home/runner/work/nomad/nomad/command/agent/agent.go:1533 +0x63
github.com/hashicorp/nomad/command/agent.TestServer_Reload_TLS_Certificate(0xc00a9f8540)
	/home/runner/work/nomad/nomad/command/agent/agent_test.go:1197 +0x1d7
testing.tRunner(0xc00a9f8540, 0x42c8720)
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1934 +0xea
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.25.3/x64/src/testing/testing.go:1997 +0x465

DONE 5542 tests, 12 skipped, 4 failures in 182.433s

I think I know what is causing this. I'll get it fixed.

…nil only

benjamin-lykins · 2025-11-17T18:17:07Z

@tgross fixed the test, which then uncovered another broken test after another I changed the return response. both of those were cleaned up and looks happier now here.

tgross

This is a good start @benjamin-lykins!

In addition to the comments I've left, make sure you've added a changelog entry file. You can do that via make cl. Once this is getting closer to being ready, we'll need a docs PR to go with this, which is now going to be over in the web-unified-docs repo.

tgross · 2025-11-21T14:34:24Z

+
+	newConf := DefaultConfig()
+
+	for _, path := range currConf.Files {


If the -config flag in the command line arguments for this agent points to a directory, this Files field will contain only the files that were in that directory at the time we started the agent. If you add a new file, it won't be present on reload. We may need to figure out a way to get the original -config flag attached to the config for this to work.

Thanks for catching this one, wasn't thinking of it or realized when I originally ran through it. Made some modifications to save the -config flag values and tested locally without issue. Tested if I passed over two -config as well and worked as expected. I was also able to reuse the logic in LoadConfig to check each path, which seems to work out well. Lastly, I tested adding files with bad configuration and got a decent response back, but I could also just send a canned message back

curl -k -X PUT \ https://localhost:4646/v1/agent/reload Error loading /Users/benjamin.lykins/git/nomad/nomad/bin/config1/badfile.hcl: failed to decode HCL file /Users/benjamin.lykins/git/nomad/nomad/bin/config1/badfile.hcl: At 1:6: key 'asdf' expected start of object ('{') or assignment ('=')

tgross · 2025-11-21T14:41:06Z

+
+	newConf.Files = append([]string(nil), currConf.Files...)
+
+	if err := s.agent.Reload(newConf); err != nil {


If you take a look at the signal handler for reloading, you'll see that this step only reloads the agent configuration and won't push any changes to the server, client, or HTTP server configuration. We should at least have parity with the signal handler here, which calls (*Command) handleReload. We may need to figure out a way to extract that function so that it's reachable here.

This still needs to be done, and I'll start looking ahead at how to get this going. I saw this when I was mucking around with TLS rotations with server/client, and it was only updating the more general agent config.

…uration options

… tests into one

…w config values/files

benjamin-lykins · 2025-12-07T20:34:35Z

@tgross

Testing : Made the updates with the testing, I realized I did see that in the contributor guides. I just realized I have some lines I need to clean up from when I was moving them from testify to test.
-config : Updated to store the config values which can be passed when starting up an agent. Tested on my end without issue, and made additional notes under your comment.

…hope so)

tgross

I think I see where you're going with this, but there's an architectural issue. See my comment for more details.

benjamin-lykins and others added 10 commits November 4, 2025 10:55

docs: update nomad.service to point to the current site

dc56e7b

Merge branch 'hashicorp:main' into main

b64afe4

Merge branch 'hashicorp:main' into main

244bcc0

Merge branch 'hashicorp:main' into main

36ea900

api: add reload endpoint to agent for configuration reload

6549d98

api: add simple returned message to reload endpoint

bb131b5

test: add tests for agent reload endpoint

a53b746

api: update notes

f560f36

api: update return message

295ecd2

api: add logging for agent reload process

5c17fa5

benjamin-lykins requested review from a team as code owners November 14, 2025 19:19

benjamin-lykins added 2 commits November 14, 2025 14:43

chore: cleaning up my comments and notes

a243a8c

api: cleanup

1cbf394

tgross self-requested a review November 17, 2025 15:09

tgross requested changes Nov 17, 2025

View reviewed changes

benjamin-lykins added 2 commits November 17, 2025 12:13

api: remove debug log from agent reload method, fix failing test

f345955

api: removing response validation in testing, check fo error and not …

8297287

…nil only

tgross requested changes Nov 21, 2025

View reviewed changes

benjamin-lykins added 5 commits December 4, 2025 16:38

api: update Reload method to accept AgentReloadOpts for future config…

906ca21

…uration options

api: update tests to use shoenig/test instead of testify, consolidate…

ddaa00a

… tests into one

api: testing clean up

ddc4a21

config: store -config paths on startup

41c6db1

api: fix reload api to use stored -config values when checking for ne…

e13400b

…w config values/files

tests: removed old commented code

dc19697

tgross added theme/api HTTP API and SDK issues type/enhancement labels Dec 11, 2025

api: move handleReload for parity between API and SIGHUP (at least i …

62de78a

…hope so)

benjamin-lykins commented Jan 9, 2026

View reviewed changes

Comment thread command/agent/config.go

benjamin-lykins commented Jan 9, 2026

View reviewed changes

Comment thread command/agent/agent.go Outdated

tgross requested changes Jan 15, 2026

View reviewed changes

Comment thread command/agent/command.go

benjamin-lykins and others added 6 commits March 10, 2026 10:18

api: revert functionality, add parity between api and SIGHUP

9d8d293

api: simplify comments in Agent struct and remove unnecessary whitespace

7111dbd

removing unused test

f3e7335

clean up

1beb513

Merge branch 'main' into feat/config-reload-api

10364cb

running gofmt

db5251c

mismithhisler reviewed Apr 29, 2026

View reviewed changes

Comment thread command/agent/agent.go

removing lock/unlock

9adfab8


		newConf := DefaultConfig()

		for _, path := range currConf.Files {


		newConf.Files = append([]string(nil), currConf.Files...)

		if err := s.agent.Reload(newConf); err != nil {

Conversation

benjamin-lykins commented Nov 14, 2025 • edited by tgross Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Links

Contributor Checklist

Reviewer Checklist

Changes to Security Controls

Uh oh!

tgross left a comment

Choose a reason for hiding this comment

Uh oh!

benjamin-lykins commented Nov 17, 2025

Uh oh!

benjamin-lykins commented Nov 17, 2025

Uh oh!

tgross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tgross Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

benjamin-lykins Dec 7, 2025

Choose a reason for hiding this comment

Uh oh!

tgross Nov 21, 2025

Choose a reason for hiding this comment

Uh oh!

benjamin-lykins Dec 7, 2025

Choose a reason for hiding this comment

Uh oh!

benjamin-lykins commented Dec 7, 2025

Uh oh!

Uh oh!

Uh oh!

tgross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

benjamin-lykins commented Nov 14, 2025 •

edited by tgross

Loading