feat: Allow configuration of private keys for TLS management (#2854)

[pauloappbr]

Assistance Disclosure

I authored the code and performed the tests. I consulted Gemini (Google's AI) for implementation strategy guidance, architectural mapping within Caddy, and initial draft snippets. All code was manually reviewed, integrated, and validated locally.

Dependencies

⚠️ This PR depends on caddyserver/certmagic#371 and should not be merged before that PR is accepted.

Description

This PR adds support for the key_file directive in the TLS app and Caddyfile.

It allows users to provide a pre-existing private key for automated certificate management (ACME or Internal), enabling key continuity across deployments, infrastructure migrations, and environments where keys are pre-provisioned.

The default behavior remains unchanged. If key_file is not specified, Caddy continues generating keys automatically via CertMagic.

Relates to

Fixes #2854

Changes

• Caddyfile:

  • Added key_file subdirective to tls
  • Added key_file as a global option

• Automation:

  • Extended AutomationPolicy with a KeyFile field

• Provisioning:

  • Added logic in automation.go to instantiate certmagic.FileKeyGenerator when a custom key file is configured

• Policy Consolidation:

  • Updated consolidation logic in tlsapp.go to prevent merging policies that reference different custom keys

• Parsing:

  • Updated builtins.go and options.go to parse and validate the key_file token

Testing

Tested locally by:

1. Generating an EC P-384 private key:

   openssl ecparam -genkey -name secp384r1 -noout -out my_key.pem

2. Configuring a site block with:

   tls {
       key_file my_key.pem
       issuer internal
   }

3. Verifying via curl -v that the server presented a certificate using the specified P-384 key (confirmed public key type and size during the TLS handshake).

4. Verifying that global options correctly propagate the key_file setting across automation policies.

[pauloappbr]

The current CI failures (lint/tests) are expected.

This PR depends on the changes introduced in caddyserver/certmagic#371, which adds the FileKeyGenerator implementation. Since that PR has not yet been merged, the Caddy build cannot resolve the new type and related logic.

Once caddyserver/certmagic#371 is approved and merged, I will update the go.mod in this branch to reference the new CertMagic version, and the CI checks should pass.

I’ve converted this PR to Draft status in the meantime to avoid confusion.

[francislavoie]

I think we'd want to use storage loader modules instead of key_file for this. The private key might be in managed storage instead for example.

Also the FileKeyGenerator interface is confusing. It's not generating a key, it's just loading one from storage. Generating implies it's a brand new one randomly generated, but that's not the case.

[pauloappbr]

Thanks for the review and the feedback, @francislavoie and @mholt!

You are completely right. Using a local key_file and reading directly from the OS breaks the clustering capabilities for users relying on external storage backends (like Consul or Redis). Also, semantically hijacking the KeyGenerator interface to act as a "loader" was definitely a bit of a hack.

I really want to get this feature right since it solves a major pain point for mobile app pinning and infrastructure migrations (Issue #2854).

If we move away from KeyGenerator and shift towards a Storage module approach, where do you think is the most appropriate place in the certmagic provisioning lifecycle to intercept and inject this pre-existing private key? Should we introduce a new KeyLoader interface, or simply extend how the automation policy interacts with the Storage backend before falling back to the Generate() method?

Any quick pointer on the preferred architectural design would be awesome, and I can start working on the refactor right away.