Skip to content

Add live Redis-backed admin key propagation across replicas#527

Open
Iweisc wants to merge 3 commits intolabring:mainfrom
Iweisc:live-redis-admin-key-propagation
Open

Add live Redis-backed admin key propagation across replicas#527
Iweisc wants to merge 3 commits intolabring:mainfrom
Iweisc:live-redis-admin-key-propagation

Conversation

@Iweisc
Copy link
Copy Markdown

@Iweisc Iweisc commented Apr 20, 2026

  • ADMIN_KEY is bootstrapped into a scoped Redis slot with SETNX
  • running instances poll that Redis slot every 500ms
  • when the value changes, the in-memory admin key is updated without restart
  • /api auth continues using in-memory state, so request handling does not hit Redis

@Iweisc
Copy link
Copy Markdown
Author

Iweisc commented Apr 20, 2026

I have some cool safety features in my mind that requires the ADMIN_KEY to hot-reload WITHOUT the deployments restarting.

@Iweisc
Copy link
Copy Markdown
Author

Iweisc commented Apr 20, 2026

One of those features could be like a fail-safe that triggers when certain actions are invoked, and you can rotate the ADMIN_KEY on the fly without any disruptions. The feature opens up a door to wide range of features. Oh and it has some nice performance boosts too.

@Iweisc
Copy link
Copy Markdown
Author

Iweisc commented Apr 20, 2026

You can even count admin/internal key matches with near-zero hot-path cost because the match logic is precomputed.

@cla-assistant
Copy link
Copy Markdown

cla-assistant Bot commented Apr 20, 2026

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ Iweisc
❌ sertdev


sertdev seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@Iweisc
Copy link
Copy Markdown
Author

Iweisc commented Apr 20, 2026

image doesn't seem to work with me.

@Iweisc
Copy link
Copy Markdown
Author

Iweisc commented Apr 20, 2026

Here are some cool use cases and features this opens up the door to, while i may not open follow up PRs with these features, i plan to follow up on some:

  • Zero-downtime ADMIN_KEY rotation across all replicas.
  • Instant cluster-wide revocation of a compromised admin credential.
  • Break-glass admin cutovers during incidents without restarts.
  • External control-plane driven admin-secret rollout.
  • Secret-manager or automation-driven key updates without pod recycling.
  • Auto-convergence of new or restarted replicas onto the active admin key.
  • Action-triggered fail-safe workflows that rotate privileged access immediately.
  • Kill-switch or maintenance workflows gated by a live-rotatable admin credential.
  • Temporary elevated admin-access windows with fast rollback.
  • Future dual-key or grace-period rotations with the same central auth state model.
  • Unified privileged auth behavior across /api, /v1, and /mcp.
  • Format-tolerant privileged auth for raw, Bearer, sk-, and Bearer sk- forms.
  • Easier future support for more privileged token formats without touching every middleware path.
  • Centralized privileged-auth policy instead of scattered per-handler parsing logic.
  • Hot-swappable in-memory auth state using atomic reads instead of restart-time env reloads.
  • Lower hot-path overhead for privileged auth because token variants are precomputed once.
  • Easier rollout of admin-backed runtime controls and operator APIs.
  • Easier implementation of scoped or derived privileged credentials later.
  • Easier auditing, metrics, and hooks around privileged-key changes.
  • Safer rolling deploys when privileged credentials change mid-rollout.

I believe, since we are pre-computing tokenVariants, adminKeyState, internalTokenState, etc etc. I believe with the speed enhancements, it'll allow for EVEN more features, that were previously impractical before because of the speed.

@zijiren233
Copy link
Copy Markdown
Member

Thank you for your contribution; please sign the CLA.

@Iweisc
Copy link
Copy Markdown
Author

Iweisc commented Apr 21, 2026

@zijiren233 doesn't seem to be working for me, but i'll try again.

@zijiren233
Copy link
Copy Markdown
Member

A commit has two contributors; sertdev has not signed the CLA, preventing CI execution.

@Iweisc
Copy link
Copy Markdown
Author

Iweisc commented Apr 21, 2026

A commit has two contributors; sertdev has not signed the CLA, preventing CI execution.

No. So, the thing is that "sertdev" is actually me, i am assuming the bot is detecting two contributors because i set my git config to the name sertdev instead of it matching the github name.....i can't really sign as sertdev.

@zijiren233
Copy link
Copy Markdown
Member

You may have used an incorrect commit email/username. Please sign the CLA correctly and resolve the conflicts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants