feat(tls): automatically reload tls certs #3752

rchincha · 2026-01-31T07:29:47Z

Fixes issue #3747

Currently, zot requires a restart whenever tls certs change, which can occur whenever there are tls cert rotation etc.

This PR checks if the tls certs have be modified and if so reloads them without restarting zot.

What type of PR is this?

Which issue does this PR fix:

What does this PR do / Why do we need it:

If an issue # is not available please add repro steps and logs showing the issue:

Testing done on this change:

Automation added to e2e:

Will this break upgrades or downgrades?

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copilot

Pull request overview

This PR implements automatic TLS certificate reloading for zot without requiring server restarts, addressing issue #3747. The implementation uses Go's tls.Config.GetCertificate callback to check for certificate file changes on each TLS handshake and reload certificates when modifications are detected.

Changes:

Added a new CertReloader type that monitors TLS certificate files and reloads them when modified
Integrated certificate reloader into the HTTP server's TLS configuration
Added comprehensive unit tests and blackbox integration tests for certificate reload functionality

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
pkg/api/tlscert.go	New file implementing the CertReloader with automatic certificate monitoring and reloading
pkg/api/tlscert_test.go	Comprehensive unit tests for CertReloader covering reload, concurrency, and edge cases
pkg/api/controller.go	Integration of CertReloader into TLS server setup with GetCertificate callback
test/blackbox/tls_cert_reload.bats	End-to-end integration tests validating certificate reload in running server

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pkg/api/tlscert.go

codecov · 2026-01-31T08:59:24Z

Codecov Report

❌ Patch coverage is 74.41860% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.52%. Comparing base (b9aad15) to head (69bbdb9).
⚠️ Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/api/tlscert.go	75.42%	19 Missing and 10 partials ⚠️
pkg/api/controller.go	63.63%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3752      +/-   ##
==========================================
- Coverage   91.60%   91.52%   -0.09%     
==========================================
  Files         189      190       +1     
  Lines       26932    27059     +127     
==========================================
+ Hits        24672    24766      +94     
- Misses       1460     1482      +22     
- Partials      800      811      +11

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rchincha · 2026-02-01T07:21:14Z

@copilot open a new pull request to apply changes based on the comments in this thread

- Add logging when certificate reload fails using log.Logger - Implement fsnotify for efficient file system monitoring instead of polling on every handshake - Add reloadMu to prevent concurrent reload operations and race conditions - Add time-based cache (1 second) for fallback polling to reduce file system calls - Watch certificate directories for better reliability with atomic file updates - Graceful fallback to periodic checking if fsnotify fails to initialize Co-authored-by: rchincha <[email protected]>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 9 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pkg/api/tlscert.go

pkg/api/tlscert_test.go

pkg/api/tlscert.go

pkg/api/controller.go

pkg/api/tlscert.go

pkg/api/tlscert_test.go

test/blackbox/tls_cert_reload.bats

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pkg/api/tlscert.go

test/blackbox/tls_cert_reload.bats

pkg/api/tlscert_test.go

Fixes issue project-zot#3747 Currently, zot requires a restart whenever tls certs change, which can occur whenever there are tls cert rotation etc. This PR checks if the tls certs have be modified and if so reloads them without restarting zot. Signed-off-by: Ramkumar Chinchani <[email protected]>

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

andaaron · 2026-02-02T19:08:59Z

pkg/api/controller.go

+
+	// Close CertReloader to prevent resource leaks
+	if c.CertReloader != nil {
+		_ = c.CertReloader.Close()


If this is closed in StopBackgroundTasks() doesn't it need to start in StartBackgroundTasks() for simetry?

There are tests which call Run() and Shutdown() multiple times on the same object, I think I fixed the background tasks issues when I refactored the HTPasswdWatcher, but let's not risk it.

andaaron · 2026-02-02T19:13:58Z

pkg/api/tlscert.go

+// This method is safe to call multiple times.
+func (cr *CertReloader) Close() error {
+	var err error
+	cr.closeOnce.Do(func() {


Does this work? If I remember correctly, we have tests calling Run() multiple times on the same controller. That would mean starting the watcher multiple times. This code closes it just once.

rchincha requested a review from andaaron as a code owner January 31, 2026 07:29

rchincha force-pushed the issue-3747 branch from ab1b1ce to 3cf0c53 Compare January 31, 2026 08:26

rchincha requested a review from Copilot January 31, 2026 08:48

Copilot started reviewing on behalf of rchincha January 31, 2026 08:48 View session

rchincha mentioned this pull request Jan 31, 2026

[Feat]: Automatically reload TLS certificates #3747

Open

Copilot AI reviewed Jan 31, 2026

View reviewed changes

pkg/api/tlscert.go Show resolved Hide resolved

pkg/api/tlscert.go Outdated Show resolved Hide resolved

pkg/api/tlscert.go Show resolved Hide resolved

Copilot AI mentioned this pull request Feb 1, 2026

feat(tls): add logging, fsnotify, and race prevention to cert reloader #3758

Closed

rchincha force-pushed the issue-3747 branch 2 times, most recently from 805d0bf to b94f3ba Compare February 1, 2026 08:27

rchincha requested a review from Copilot February 1, 2026 10:00

Copilot started reviewing on behalf of rchincha February 1, 2026 10:00 View session

Copilot AI reviewed Feb 1, 2026

View reviewed changes

Copilot AI mentioned this pull request Feb 1, 2026

Fix resource leaks and race conditions in TLS certificate reloader #3759

Closed

rchincha force-pushed the issue-3747 branch from b94f3ba to 49b00a7 Compare February 1, 2026 21:03

rchincha requested a review from Copilot February 1, 2026 21:08

Copilot started reviewing on behalf of rchincha February 1, 2026 21:09 View session

Copilot AI reviewed Feb 1, 2026

View reviewed changes

pkg/api/tlscert.go Show resolved Hide resolved

test/blackbox/tls_cert_reload.bats Show resolved Hide resolved

pkg/api/tlscert_test.go Outdated Show resolved Hide resolved

rchincha force-pushed the issue-3747 branch from 49b00a7 to 7ead92b Compare February 1, 2026 22:48

Copilot AI mentioned this pull request Feb 1, 2026

Address review comments: fix test description and clarify reload failure behavior #3764

Closed

rchincha force-pushed the issue-3747 branch from 7ead92b to 69bbdb9 Compare February 2, 2026 06:56

rchincha requested a review from Copilot February 2, 2026 07:16

Copilot started reviewing on behalf of rchincha February 2, 2026 07:16 View session

Copilot AI reviewed Feb 2, 2026

View reviewed changes

andaaron reviewed Feb 2, 2026

View reviewed changes

rchincha added this to the v2.1.15 milestone Feb 5, 2026

feat(tls): automatically reload tls certs #3752

Are you sure you want to change the base?

feat(tls): automatically reload tls certs #3752

Uh oh!

Conversation

rchincha commented Jan 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

rchincha commented Feb 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

andaaron Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

andaaron Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Jan 31, 2026 •

edited

Loading