Skip to content

feat: secure internal gRPC with JWT - #4938

Open
CooooolFrog wants to merge 1 commit into
dragonflyoss:mainfrom
CooooolFrog:feat/grpc-jwt-auth
Open

feat: secure internal gRPC with JWT#4938
CooooolFrog wants to merge 1 commit into
dragonflyoss:mainfrom
CooooolFrog:feat/grpc-jwt-auth

Conversation

@CooooolFrog

Copy link
Copy Markdown
Contributor

Description

Add optional JWT-based authentication for internal gRPC communication between Dragonfly components.

This change introduces:

  • Shared HMAC keyring loading from Base64-encoded secret files.
  • JWT generation and verification using component-specific audiences.
  • Unary and streaming gRPC client and server interceptors.
  • Three authentication modes:
    • disabled: preserves the existing unauthenticated behavior.
    • permissive: sends and validates JWTs while accepting requests without credentials during rolling upgrades.
    • required: rejects requests without a valid JWT.
  • Authentication for Manager and Scheduler business gRPC servers.
  • JWT credentials for Scheduler outbound gRPC calls.
  • Authentication metrics for requests and token generation.
  • Support for overlapping keys and kid-based key rotation.
  • A detailed protocol, deployment, rolling-upgrade, and key-rotation design document.

Authentication is disabled by default. When the configuration is omitted or set to disabled, no key files are required, clients do not attach JWTs, and servers do not authenticate requests.

Health checks and local Unix domain socket calls remain unauthenticated.

Related Issue

Related to #4417

Motivation and Context

Internal Dragonfly components currently communicate over gRPC without application-level authentication unless mTLS is enabled.

This change adds token-based authentication using short-lived JWTs signed with a shared secret. It provides an additional authentication option without requiring a PKI or an external token-issuing service.

The permissive mode supports a simple and backward-compatible first rollout:

  1. Upgrade the deployment with JWT-capable binaries and set the global mode to permissive.
  2. Old and new components can coexist during the rolling update.
  3. After all components and maintained external callers send JWTs, change the global mode to required.

After a deployment reaches required mode, subsequent compatible releases can use the normal one-step rolling upgrade while keeping authentication enabled.

The default disabled mode ensures that users who do not enable JWT authentication are unaffected.

Screenshots (if appropriate)

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation Update (if none of the other choices apply)

Checklist

  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 73.56688% with 83 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.77%. Comparing base (07bd92a) to head (93d82c3).

Files with missing lines Patch % Lines
pkg/rpc/auth/jwt/interceptor.go 75.70% 15 Missing and 11 partials ⚠️
pkg/rpc/auth/jwt/auth.go 77.02% 10 Missing and 7 partials ⚠️
scheduler/scheduler.go 0.00% 13 Missing ⚠️
scheduler/service/service_v2.go 9.09% 10 Missing ⚠️
pkg/rpc/auth/jwt/config.go 92.20% 3 Missing and 3 partials ⚠️
manager/manager.go 0.00% 5 Missing ⚠️
manager/rpcserver/rpcserver.go 0.00% 4 Missing ⚠️
pkg/rpc/manager/server/server.go 75.00% 1 Missing ⚠️
pkg/rpc/scheduler/server/server.go 75.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #4938      +/-   ##
==========================================
+ Coverage   28.86%   29.77%   +0.91%     
==========================================
  Files         226      229       +3     
  Lines       22565    22858     +293     
==========================================
+ Hits         6513     6807     +294     
+ Misses      15605    15583      -22     
- Partials      447      468      +21     
Flag Coverage Δ
unittests 29.77% <73.56%> (+0.91%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
manager/config/config.go 96.27% <100.00%> (+0.05%) ⬆️
scheduler/config/config.go 88.42% <100.00%> (+0.18%) ⬆️
scheduler/resource/standard/resource.go 52.94% <100.00%> (ø)
scheduler/rpcserver/rpcserver.go 100.00% <100.00%> (ø)
scheduler/rpcserver/scheduler_server_v2.go 1.02% <100.00%> (ø)
pkg/rpc/manager/server/server.go 94.73% <75.00%> (+94.73%) ⬆️
pkg/rpc/scheduler/server/server.go 95.00% <75.00%> (+95.00%) ⬆️
manager/rpcserver/rpcserver.go 0.00% <0.00%> (ø)
manager/manager.go 0.00% <0.00%> (ø)
pkg/rpc/auth/jwt/config.go 92.20% <92.20%> (ø)
... and 4 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CooooolFrog CooooolFrog added the enhancement New feature or request label Aug 13, 2026
Signed-off-by: CooooolFrog <zuliangwanghust@gmail.com>
@CooooolFrog CooooolFrog changed the title feat(auth): secure internal gRPC with JWT feat: secure internal gRPC with JWT Aug 13, 2026
@@ -0,0 +1,11 @@
{
"_comment": "TEST ONLY. This deterministic key must never be used in a deployment.",
"secretBase64": "MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this is a test, I suggest deleting and leaving these parts blank to avoid ambiguity

}

if _, ok := keys[config.JWT.ActiveKeyID]; !ok {
return nil, fmt.Errorf("grpc auth jwt active key id %q is not trusted", config.JWT.ActiveKeyID)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is recommended to uniformly use fmt. Errorf or errors. New to avoid misunderstanding.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants