Collector supervisor enters infinite 401 restart loop when the server-side record has been pruned

[etgraylog]

Expected Behavior

When the Graylog server returns HTTP 401 to a collector whose certificate fingerprint is no longer recognised (for example because the server-side instance record was purged while the collector was unreachable), the supervisor could detect that its persisted credentials are no longer valid and re-enroll from scratch on the next start. A restarted collector process would then rejoin the fleet within one enrollment round-trip.

This is one reasonable policy rather than a specified behavior. Stock opamp-go deliberately delegates 401 handling to the application via OnConnectFailed, and "log-and-continue until an operator intervenes" is a valid alternative — notably where automatic re-enrollment would be undesirable (for example, deployments that want a human decision in the loop before an agent rotates identity, or environments where an enrollment-storm after a mass purge would be worse than silent-until-attended agents).

Current Behavior

This is an application-layer enhancement request — the OpAMP library (open-telemetry/opamp-go) is unchanged; it already delegates 401 handling to the application via OnConnectFailed.

Stock client has no 401 handler — OnConnectFailed just logs:

// superv/supervisor/supervisor.go — stock OnConnectFailed
OnConnectFailed: func(ctx context.Context, err error) {
    s.logger.Error("Failed to connect to OpAMP server", zap.Error(err))
},

Recovery from a server-side record purge is a missing behavior, not a broken one. On 401 the collector process stays running and keeps polling; every poll logs invalid response from server: 401 and returns; the collector is silent to the server with no automatic remediation. (OnConnectFailed shown above is the callback the supervisor would invoke, but stock opamp-go does not surface 401 through it — see the Companion change note in Tested in isolation below.)

Restarting the process doesn't help either, because IsEnrolled() just checks files on disk:

// superv/auth/manager.go
func (m *Manager) IsEnrolled() bool {
    return persistence.SigningKeyExists(m.keysDir) &&
        persistence.CertificateExists(m.keysDir)
}

And initAuth skips enrollment when IsEnrolled() returns true:

// superv/supervisor/supervisor.go
func (s *Supervisor) initAuth(ctx context.Context) error {
    if s.authManager.IsEnrolled() {
        s.logger.Debug("Loading existing credentials")
        if err := s.authManager.LoadCredentials(); err != nil {
            return fmt.Errorf("failed to load credentials: %w", err)
        }
        return nil
    }
    // ... otherwise prepare CSR and enroll
}

So any subsequent process start — automatic via a supervising process manager, or manual by an operator — re-reads the same cert, presents the same fingerprint, and gets the same 401. The IsEnrolled() invariant — cert on disk ↔ valid registration on server — no longer holds after the server-side purge, and stock has no mechanism to re-establish it.

Under any process-lifecycle model — log-and-continue, exit-without-clearing, supervised or not — invalid stored credentials must be removed before a healthy re-enrollment can occur, either by the collector itself (the patch below) or manually.

The typical trigger is PurgeExpiredCollectorInstancesPeriodical, a Graylog internal task that deletes records whose last_seen exceeds collector_expiration_threshold. A server-side upsert-on-reenroll would also address this, but the client fix below stands on its own: if the server has rejected our credentials, acquire new ones.

Possible Solution

Expose the recovery behavior as an opt-in configuration flag. With it enabled, the supervisor detects 401-while-enrolled in OnConnectFailed, clears persisted state (credentials + instance UID + own-logs settings), and exits via os.Exit(1); on the next process start IsEnrolled() returns false, initAuth runs the enrollment path, and the agent re-registers with a fresh UID and keypair. With it disabled (the default), behavior is identical to stock — the supervisor logs the 401 and continues polling.

// superv/config/types.go — AuthConfig additions
type AuthConfig struct {
    // ... existing fields ...

    // ResetOnAuthRejection controls whether the supervisor automatically
    // clears persisted credentials + instance UID + own-logs settings and
    // exits when the server rejects the agent's existing cert with HTTP 401.
    // When true, a subsequent process start triggers a fresh enrollment.
    // When false (the default), the supervisor logs the 401 and continues
    // polling; recovery requires manual removal of the stale credential
    // files. Intended for deployments where ephemeral auto-re-enrollment
    // is acceptable (e.g. disposable collector fleets); leave disabled
    // where a human-in-the-loop decision is required before an agent
    // rotates identity, or where a fleet-wide enrollment storm after a
    // mass server-side purge would be undesirable.
    ResetOnAuthRejection bool `koanf:"reset_on_auth_rejection"`
}

// superv/supervisor/supervisor.go — patched OnConnectFailed (gated on the config flag)
OnConnectFailed: func(ctx context.Context, err error) {
    s.logger.Error("Failed to connect to OpAMP server", zap.Error(err))
    if !strings.Contains(err.Error(), "401") && !strings.Contains(err.Error(), "unauthorized") {
        return
    }
    if !s.authManager.IsEnrolled() {
        s.logger.Warn("Authentication rejected (401) during enrollment, will retry")
        return
    }
    if !s.authCfg.ResetOnAuthRejection {
        // Opt-in recovery disabled — preserve stock log-and-continue semantics.
        s.logger.Warn("Authentication rejected (401) with existing credentials; auth.reset_on_auth_rejection is disabled, so recovery requires manual removal of persisted credentials")
        return
    }
    s.logger.Error("Authentication rejected (401) with existing credentials, clearing credentials, instance UID, and own_logs persistence and exiting to trigger re-enrollment")
    if clearErr := s.authManager.ClearCredentials(); clearErr != nil {
        s.logger.Error("Failed to clear credentials before exit", zap.Error(clearErr))
    }
    if clearErr := persistence.ClearInstanceUID(s.persistenceDir); clearErr != nil {
        s.logger.Error("Failed to clear instance UID before exit", zap.Error(clearErr))
    }
    if s.ownLogsPersistence != nil {
        if clearErr := s.ownLogsPersistence.Delete(); clearErr != nil {
            s.logger.Error("Failed to clear own_logs persistence before exit", zap.Error(clearErr))
        }
    }
    os.Exit(1)
},

Supporting helpers (new):

// persistence/keys.go
func ClearCredentials(keysDir string) error {
    for _, name := range []string{SigningKeyFile, SigningCertFile, encryptionKeyFile} {
        if err := os.Remove(filepath.Join(keysDir, name)); err != nil && !errors.Is(err, os.ErrNotExist) {
            return fmt.Errorf("removing %s: %w", name, err)
        }
    }
    return nil
}

// auth/manager.go
func (m *Manager) ClearCredentials() error {
    if err := persistence.ClearCredentials(m.keysDir); err != nil {
        return err
    }
    m.mu.Lock()
    m.signingKey = nil
    m.certificate = nil
    m.mu.Unlock()
    return nil
}

// persistence/instance.go
func ClearInstanceUID(dir string) error {
    if err := os.Remove(filepath.Join(dir, "identity.yaml")); err != nil && !errors.Is(err, os.ErrNotExist) {
        return fmt.Errorf("removing identity.yaml: %w", err)
    }
    return nil
}

Why each piece is necessary:

• ResetOnAuthRejection config flag, default false: makes the recovery behavior opt-in. Preserves stock log-and-continue semantics for deployments that prefer human-in-the-loop identity rotation, and makes the new behavior available to deployments where auto-re-enrollment is acceptable.
• Credentials (signing.key, signing.crt, encryption.key): primary gap addressed when the flag is enabled — without clearing these, the restarted process reads the same stale cert and hits the same 401.
• Instance UID (identity.yaml): server tracks agents by UID and rejects re-enrollment of a known UID (Rejecting enrollment: collector <UID> already enrolled). Without UID clearing, the 401 loop is replaced by an "already enrolled" rejection loop.
• Own-logs settings (own_logs-settings.yaml): references signing.key/signing.crt paths for the own-logs exporter; without clearing it, the next start emits a misleading load client certificate warning on what is otherwise a clean re-enrollment.

Missing files are treated as already-cleared; I/O errors are logged but don't block os.Exit.

Tested in isolation (single-pod A/B)

First validated 2026-04-22 in an observation environment (Kubernetes; ~52 s DELETE → Ready end-to-end with the recovery path unconditionally enabled). The isolation test below is methodologically stronger because it exercises both flag values directly on otherwise-identical pods.

Companion change: This patch's handleAuthRejection is wired to OnConnectFailed, which stock opamp-go does not invoke for HTTP 401 responses — 401 falls into the default: case of HTTPSender.attemptRequest with retry: false, and makeOneRequestRoundtrip logs and returns. A companion change in opamp-go that makes 401 retryable surfaces the error through OnConnectFailed and is a prerequisite for this patch to take effect. Filed as upstream-issue-opamp-go-401-retryable. The A/B below uses a build that has both patches in place; without the opamp-go companion, both Pod A (flag=true) and Pod B (flag=false) behave the same — both log invalid response from server: 401 every poll interval indefinitely, and neither restarts.

Two single-pod Deployments on the same cluster, both built from the image with both patches, identical except for the config flag — Pod A with GLC_SERVER__AUTH__RESET_ON_AUTH_REJECTION=true, Pod B with ...=false. Both pods enrolled cleanly; initial UIDs recorded from the Graylog API.

• t=0: DELETE /api/collectors/instances/<UID> issued against both pods' records (HTTP 204 each).
• t+28s: Pod A's next poll returns 401 — OnConnectFailed fires (via the companion opamp-go patch), handleAuthRejection runs, IsEnrolled() true + ResetOnAuthRejection=true → credentials/UID/own-logs cleared, os.Exit(1) called. Supervisor logs: Authentication rejected (401) with existing credentials; clearing credentials, instance UID, and own_logs persistence and exiting to trigger re-enrollment.
• t+29s: kubelet restarts the container. Restart count goes from 0 to 1.
• t+32s: pre-enrollment jitter (3.1 s) elapses, fresh enrollment completes. New UID assigned and visible to the server. Pod A is Ready.
• Pod B: same 401 observed via the same code path, but handleAuthRejection finds ResetOnAuthRejection=false and logs auth.reset_on_auth_rejection is disabled, so recovery requires manual removal of persisted credentials on every 401. No state is cleared, os.Exit is not called, restart count stays at 0. Pod B continues logging a 401 every ~30 s.

Result: Pod A re-enrolled with a fresh UID ~32 seconds after the DELETE (end-to-end: DELETE → Ready with new identity); Pod B stayed running and silent-to-server (confirms the config-gate works in both directions).

Steps to Reproduce

1. Have a collector process enrolled with a Graylog server (under any supervising process manager, or no supervisor at all).
2. Read the collector's instance_uid from its persistence directory:

   # read from /var/lib/graylog-collector/supervisor/identity.yaml
   awk '/^instance_uid/ {print $2}' \
     /var/lib/graylog-collector/supervisor/identity.yaml
   # (or via `kubectl exec ... -- cat ...` if running under Kubernetes)

3. Delete the record via the Graylog API:

   curl -s -u "<token>:token" -H 'X-Requested-By: cli' -X DELETE \
     "https://<graylog>/api/collectors/instances/<UID>"

4. The collector's next heartbeat returns 401.
5. Without the patch: the collector process stays running but permanently silent — every poll logs Failed to connect to OpAMP server and returns; the process never exits, so no supervising process manager sees a failure.
6. With the patch: single-restart recovery — the supervisor clears persisted state, exits with code 1; the process manager (if any) restarts it; initAuth sees IsEnrolled() == false and re-enrolls with a fresh UID. Measured ~32 s DELETE → Ready in the single-pod A/B (see Tested in isolation below).

Context

Encountered while scaling a single Graylog server past 1700 concurrent collector processes. A 30-second graylog-server restart that coincided with a subset of collectors being offline long enough for PurgeExpiredCollectorInstancesPeriodical to delete them left those collectors with persisted credentials the server no longer recognized. In the observation environment (Kubernetes) recovery required kubectl delete pod; on a SystemD host it would mean stopping the service, removing the stale credential files, and restarting it.

The purge scenario is one trigger; the gap is broader. Any condition that causes the server to reject a previously-valid cert — administrative purge via the API, a bulk record-loss event, a certificate-authority rotation, etc. — leaves the supervisor without an automatic path back to a valid enrollment. The patch addresses that gap regardless of how the 401 arises.

Your Environment

• Graylog Version: 7.1.0-beta.1
• Java Version: 21 (OpenJDK)
• OpenSearch Version: n/a (data-node not exercised by this path)
• MongoDB Version: 6.x
• Operating System: Linux (alpine:3.21 container)
• Browser version: n/a

Checklist

[x] This issue fix need to be backported — applies to all v2+ collector-sidecar builds that use the OpAMP supervisor.
[x] Does this issue have security implications? Auto-wiping persisted credentials is a security-relevant state transition — flagged so maintainers can confirm the opt-in flag's default (false) is the right posture, given that an attacker who can reliably provoke 401 (stale server cache, misconfig, MITM with rogue cert) could otherwise induce a fleet-wide credential-wipe + re-enroll storm on deployments that enable it.