Skip to content

feat(session): manage KAP mTLS agent credentials#1264

Merged
gyuho merged 15 commits into
mainfrom
cardyok/feat-kap-mtls-agent-credentials
Jul 13, 2026
Merged

feat(session): manage KAP mTLS agent credentials#1264
gyuho merged 15 commits into
mainfrom
cardyok/feat-kap-mtls-agent-credentials

Conversation

@cardyok

@cardyok cardyok commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • keep only the KAP mTLS status and credential-update session commands; kubeconfig is now fixed during Lepton bootstrap
  • validate each per-machine certificate, identity, endpoint, and CA fingerprint before writing an immutable credential generation
  • atomically switch the current symlink, restart the agent, and acknowledge the update only after /readyz succeeds
  • remove enable/disable state, kubeconfig mutation, pending markers, nested TLS preflight, and automatic rollback
  • keep certificate and private-key material redacted from session audit logs

Rollout

  • the companion Lepton MR dgxc-lepton/lepton!7510 installs the agent by default on new nodes and points kubelet at loopback from bootstrap
  • existing nodes are not backfilled; Lepton reconciles only Machines that report the kap-mtls-agent package
  • release this gpud change before or with the Lepton node-init rollout; failures leave new nodes blocked before kubelet registration

Verification

  • GOFLAGS=-mod=mod go test -count=1 ./pkg/kapmtls
  • GOFLAGS=-mod=mod go test -count=1 -gcflags='all=-N -l' ./pkg/session
  • GOFLAGS=-mod=mod go test -race ./pkg/kapmtls
  • GOFLAGS=-mod=mod go vet ./pkg/kapmtls ./pkg/session

The repository-wide default pkg/session test command requires Mockey's -gcflags='all=-N -l' on macOS/arm64; the changed package scope passes with that required flag.

Jira: LEP-5336

@codecov

codecov Bot commented Jul 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.30988% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.66%. Comparing base (c15cb49) to head (695d996).

Files with missing lines Patch % Lines
pkg/kapmtls/manager.go 94.73% 13 Missing and 11 partials ⚠️
pkg/login/login.go 33.33% 1 Missing and 1 partial ⚠️
pkg/server/server.go 86.66% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1264      +/-   ##
==========================================
+ Coverage   90.59%   90.66%   +0.06%     
==========================================
  Files         349      351       +2     
  Lines       26429    26997     +568     
==========================================
+ Hits        23944    24477     +533     
- Misses       1685     1708      +23     
- Partials      800      812      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gyuho gyuho self-assigned this Jul 9, 2026
Comment thread pkg/kapmtls/manager.go
Comment on lines +169 to +180
if err := m.swapCurrentSymlink(releaseID); err != nil {
return err
}
if _, err := m.runSystemctl(ctx, "enable", AgentService); err != nil {
return fmt.Errorf("enable KAP mTLS agent: %w", err)
}
if _, err := m.runSystemctl(ctx, "restart", AgentService); err != nil {
return fmt.Errorf("restart KAP mTLS agent: %w", err)
}
if !m.waitReady(ctx) {
return fmt.Errorf("KAP mTLS agent did not become ready")
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 — Failed activation is reported as successfully applied. UpdateCredentials switches current before enabling/restarting the service. If restart fails while the old agent remains alive, subsequent Status reports the new on-disk credentials together with AgentActive=true and AgentReady=true. The reconciler therefore considers the desired generation installed, although the running process still uses the old certificate/CA/endpoint. I reproduced this state. Track an applied generation only after readiness, roll back the symlink on activation failure, or verify the running certificate serial.

Comment thread pkg/kapmtls/manager.go Outdated

@eahydra eahydra left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

defer to @gyuho

cardyok and others added 13 commits July 12, 2026 19:38
Add crash-safe credential activation and kubelet cutover over the existing session channel for newly opted-in nodes.

Co-Authored-By: Codex <noreply@openai.com>
Prevent restart and rollback paths from advancing persisted state until the agent reports the expected loaded certificate.

Co-Authored-By: Codex <noreply@openai.com>
Report explicit disablement to the control plane and retain its durable marker until agent activation has fully succeeded.

Co-Authored-By: Codex <noreply@openai.com>
Cover malformed identity material, durable state corruption, rollback failures, and kubeconfig transition errors so the credential lifecycle remains diagnosable and retryable.

Co-Authored-By: Codex <noreply@openai.com>
Treat TLS verification as successful once the handshake completes because a peer may close before the best-effort close-notify write.

Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Serialize state-directory updates, roll back failed activation, and persist machine-bound session proof across restarts.\n\nCo-Authored-By: Codex <noreply@openai.com>
Exercise validation, atomic release, activation rollback, and readiness failure paths so the new manager behavior is protected by the PR coverage gate.\n\nCo-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
Co-Authored-By: Codex <noreply@openai.com>
@cardyok
cardyok force-pushed the cardyok/feat-kap-mtls-agent-credentials branch from 75954b9 to 5655918 Compare July 12, 2026 11:52
@gyuho
gyuho merged commit b6c6b6d into main Jul 13, 2026
16 checks passed
@gyuho
gyuho deleted the cardyok/feat-kap-mtls-agent-credentials branch July 13, 2026 03:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants