fix: chores and reliability fixes (#123-#125, #128, #132, #137-#139) (#141)

* fix: chores and reliability fixes (#123-#125, #128, #132, #137-#139)

- #123: rename release-please.yml → changeset-version.yml; update RELEASING.md
- #124: pin alpine base image to digest in Dockerfile.goreleaser
- #125: add go-licenses NOTICE generation to release pipeline and goreleaser archives
- #128: audit events for secret.Store Create and Delete (Actor field, transactional EmitTx)
- #132: retry smtp.SendMail up to 3× in TestEmailReceive_FetchesUnseenMessages
- #137: numeric UID (10001) in Dockerfile and Helm podSecurityContext so runAsNonRoot resolves
- #138: remove worker/reaper liveness checkers from /readyz; probe checks DB only
- #139: revert unauthenticated /metrics; add comment explaining bearer-token Prometheus path;
        annotate serviceMonitor.bearerTokenSecret in values.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix: address CodeRabbit review feedback on PR #141

- Pin alpine:3.21 to correct digest (1f390db) in Dockerfile.goreleaser
- Pin go-licenses to @v2.0.1 instead of @latest in release-engine.yml
- Update stale comment in server.go: /readyz no longer uses worker/reaper
  liveness checkers; explain the intentional design decision

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Michael McNees <mmcnees@Michaels-MacBook-Pro.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

Author: 3 people

.github/workflows/release-please.yml .github/workflows/changeset-version.yml.github/workflows/release-please.yml renamed to .github/workflows/changeset-version.yml

@@ -44,6 +44,12 @@ jobs:
           # what triggered this workflow; goreleaser needs the plain vX.Y.Z form.
           git tag "${TAG}"
 
+      - name: Generate NOTICE
+        run: |
+          go install github.com/google/go-licenses@v2.0.1
+          go-licenses report ./... > NOTICE
+        working-directory: packages/engine
+
       - name: Run goreleaser
         uses: goreleaser/goreleaser-action@v6
         with:

.github/workflows/release-engine.yml

@@ -18,7 +18,7 @@ The automated CI path (Version PR → merge) requires no local tooling beyond gi
 ```
 1. changeset file added during dev
          ↓
-2. push to main → release-please.yml creates/updates "Version PR"
+2. push to main → changeset-version.yml creates/updates "Version PR"
          ↓
 3. merge Version PR
          ↓
@@ -28,8 +28,6 @@ The automated CI path (Version PR → merge) requires no local tooling beyond gi
 5b. release-helm.yml    → helm push OCI + GitHub Release
 ```
 
-> **Note on workflow naming:** `release-please.yml` is named after a tool we evaluated but didn't adopt — it actually runs the Changesets CLI. A rename to `changeset-version.yml` is tracked in [dvflw/mantle#123](https://github.com/dvflw/mantle/issues/123).
-
 ## Step 1: Add a Changeset (During Development)
 
 Every PR that changes user-visible behavior needs a changeset. Run this from the repo root:
@@ -56,7 +54,7 @@ If a PR touches only CI, tests, or tooling with no user impact, skip the changes
 
 ## Step 2: The Version PR
 
-After any push to `main` that contains pending changesets, the `release-please.yml` workflow automatically creates (or updates) a PR titled **"ci: version packages"**.
+After any push to `main` that contains pending changesets, the `changeset-version.yml` workflow automatically creates (or updates) a PR titled **"ci: version packages"**.
 
 This PR:
 - Bumps `version` in each affected `package.json` (taking the highest bump across all pending changesets)
@@ -65,7 +63,7 @@ This PR:
 
 Review the PR to confirm the version bumps and changelog entries are correct. If more changesets land on `main` before you merge, CI updates the PR automatically.
 
-> **Loop guard:** the workflow has an `if` condition that skips runs where the triggering commit message starts with `"ci: version packages"`. This prevents the workflow from retriggering on its own Version PR merge commit. If you ever rename the commit message in the workflow, update the guard condition in `release-please.yml` to match.
+> **Loop guard:** the workflow has an `if` condition that skips runs where the triggering commit message starts with `"ci: version packages"`. This prevents the workflow from retriggering on its own Version PR merge commit. If you ever rename the commit message in the workflow, update the guard condition in `changeset-version.yml` to match.
 
 ## Step 3: Merge the Version PR
 
@@ -107,7 +105,7 @@ The tag-triggered workflows are safe to retrigger — goreleaser will fail clean
 
 This applies when `package.json` is already at the target version but `.changeset/` contains no pending changeset files — for example, the first-ever release of a newly bootstrapped repo, or after manually editing `package.json` outside the changeset flow.
 
-In this state the `release-please.yml` workflow has nothing to process and will not create a Version PR. Push the tags directly:
+In this state the `changeset-version.yml` workflow has nothing to process and will not create a Version PR. Push the tags directly:
 
 ```bash
 git tag engine/v<version>

RELEASING.md

@@ -30,6 +30,9 @@ archives:
       # (packages/engine); LICENSE lives at the repo root.
       - src: "../../LICENSE"
         dst: "LICENSE"
+      # Third-party license notices generated by go-licenses during release CI.
+      - src: "NOTICE"
+        dst: "NOTICE"
 
 checksum:
   name_template: checksums.txt

packages/engine/.goreleaser.yaml

@@ -1,8 +1,8 @@
-FROM alpine:3.21
+FROM alpine:3.21@sha256:1f390db9d9746cc317e16a98a8be2854c599abcbf04d45b6320b040e72034e33
 
 RUN apk add --no-cache ca-certificates tzdata \
     && addgroup -S mantle \
-    && adduser -S -G mantle -h /home/mantle -s /bin/sh mantle
+    && adduser -S -u 10001 -G mantle -h /home/mantle -s /bin/sh mantle
 
 ARG TARGETPLATFORM
 COPY $TARGETPLATFORM/mantle /usr/local/bin/mantle
@@ -15,7 +15,7 @@ EXPOSE 8080
 HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
     CMD wget -qO- http://localhost:8080/healthz || exit 1
 
-USER mantle
+USER 10001:10001
 WORKDIR /home/mantle
 
 ENTRYPOINT ["mantle"]

packages/engine/Dockerfile.goreleaser

@@ -76,7 +76,9 @@ func AuthMiddleware(store *Store, oidcValidator TokenValidator, next http.Handle
 	}
 
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		// Skip auth for health checks only. /metrics requires authentication.
+		// Skip auth for health checks only. /metrics is intentionally protected —
+		// path labels expose workflow names. Configure Prometheus with a bearer
+		// token via serviceMonitor.bearerTokenSecret in the Helm values.
 		if r.URL.Path == "/healthz" || r.URL.Path == "/readyz" {
 			next.ServeHTTP(w, r)
 			return

packages/engine/internal/auth/middleware.go

@@ -137,8 +137,17 @@ func TestEmailReceive_FetchesUnseenMessages(t *testing.T) {
 		"\r\n" +
 		"This is the body of the test email.\r\n")
 
-	if err := smtp.SendMail(smtpAddr, nil, "sender@example.com", []string{username}, msg); err != nil {
-		t.Fatalf("failed to send test email: %v", err)
+	var smtpErr error
+	for attempt := 1; attempt <= 3; attempt++ {
+		smtpErr = smtp.SendMail(smtpAddr, nil, "sender@example.com", []string{username}, msg)
+		if smtpErr == nil {
+			break
+		}
+		t.Logf("smtp attempt %d: %v (retrying in 500ms)", attempt, smtpErr)
+		time.Sleep(500 * time.Millisecond)
+	}
+	if smtpErr != nil {
+		t.Fatalf("failed to send test email after retries: %v", smtpErr)
 	}
 
 	// Fetch via the connector — retry to allow GreenMail time to propagate

packages/engine/internal/connector/email_receive_test.go

@@ -4,16 +4,27 @@ import (
 	"context"
 	"database/sql"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"time"
 
+	"github.com/dvflw/mantle/internal/audit"
 	"github.com/dvflw/mantle/internal/auth"
 )
 
 // Store handles CRUD operations for encrypted credentials in Postgres.
+// Every state-changing method emits an audit event in the same transaction.
 type Store struct {
 	DB        *sql.DB
 	Encryptor *Encryptor
+	Actor     string // defaults to "cli" when empty
+}
+
+func (s *Store) actor() string {
+	if s.Actor == "" {
+		return "cli"
+	}
+	return s.Actor
 }
 
 // Create validates, encrypts, and stores a new credential.
@@ -62,6 +73,17 @@ func (s *Store) Create(ctx context.Context, name, typeName string, data map[stri
 		return nil, fmt.Errorf("storing credential: %w", err)
 	}
 
+	if err := audit.EmitTx(ctx, tx, audit.Event{
+		Timestamp: time.Now(),
+		Actor:     s.actor(),
+		Action:    audit.ActionCredentialCreated,
+		Resource:  audit.Resource{Type: "credential", ID: cred.ID},
+		TeamID:    teamID,
+		Metadata:  map[string]string{"name": name, "type": typeName},
+	}); err != nil {
+		return nil, fmt.Errorf("emitting audit event: %w", err)
+	}
+
 	if err := tx.Commit(); err != nil {
 		return nil, fmt.Errorf("committing credential: %w", err)
 	}
@@ -119,21 +141,42 @@ func (s *Store) List(ctx context.Context) ([]Credential, error) {
 	return creds, rows.Err()
 }
 
-// Delete removes a credential by name.
+// Delete removes a credential by name and emits a credential.deleted audit
+// event in the same transaction. Returns an error when no row matches.
 func (s *Store) Delete(ctx context.Context, name string) error {
 	teamID := auth.TeamIDFromContext(ctx)
-	result, err := s.DB.ExecContext(ctx,
-		`DELETE FROM credentials WHERE name = $1 AND team_id = $2`, name, teamID,
-	)
+
+	tx, err := s.DB.BeginTx(ctx, nil)
 	if err != nil {
-		return fmt.Errorf("deleting credential: %w", err)
+		return fmt.Errorf("starting transaction: %w", err)
+	}
+	defer tx.Rollback()
+
+	var deletedID string
+	err = tx.QueryRowContext(ctx,
+		`DELETE FROM credentials WHERE name = $1 AND team_id = $2 RETURNING id`,
+		name, teamID,
+	).Scan(&deletedID)
+	if errors.Is(err, sql.ErrNoRows) {
+		return fmt.Errorf("credential %q not found", name)
 	}
-	rows, err := result.RowsAffected()
 	if err != nil {
-		return err
+		return fmt.Errorf("deleting credential: %w", err)
 	}
-	if rows == 0 {
-		return fmt.Errorf("credential %q not found", name)
+
+	if err := audit.EmitTx(ctx, tx, audit.Event{
+		Timestamp: time.Now(),
+		Actor:     s.actor(),
+		Action:    audit.ActionCredentialDeleted,
+		Resource:  audit.Resource{Type: "credential", ID: deletedID},
+		TeamID:    teamID,
+		Metadata:  map[string]string{"name": name},
+	}); err != nil {
+		return fmt.Errorf("emitting audit event: %w", err)
+	}
+
+	if err := tx.Commit(); err != nil {
+		return fmt.Errorf("committing credential delete: %w", err)
 	}
 	return nil
 }

packages/engine/internal/secret/store.go

@@ -56,24 +56,6 @@ type Server struct {
 	running    map[string]context.CancelFunc // execution ID → cancel
 }
 
-// workerLivenessChecker adapts the Worker to the health.LivenessChecker interface.
-type workerLivenessChecker struct {
-	w         *engine.Worker
-	threshold time.Duration
-}
-
-func (c *workerLivenessChecker) IsAlive() bool { return c.w.IsAlive(c.threshold) }
-func (c *workerLivenessChecker) Name() string  { return "worker" }
-
-// reaperLivenessChecker adapts the Reaper to the health.LivenessChecker interface.
-type reaperLivenessChecker struct {
-	r         *engine.Reaper
-	threshold time.Duration
-}
-
-func (c *reaperLivenessChecker) IsAlive() bool { return c.r.IsAlive(c.threshold) }
-func (c *reaperLivenessChecker) Name() string  { return "reaper" }
-
 // New creates a Server with the given configuration.
 func New(db *sql.DB, eng *engine.Engine, address string) *Server {
 	logger := slog.Default()
@@ -128,9 +110,6 @@ func metricsMiddleware(next http.Handler) http.Handler {
 func (s *Server) Start(ctx context.Context) error {
 	mux := http.NewServeMux()
 
-	// Collect liveness checkers for readyz.
-	var livenessCheckers []health.LivenessChecker
-
 	// Prometheus metrics endpoint.
 	mux.Handle("/metrics", promhttp.Handler())
 
@@ -193,17 +172,6 @@ func (s *Server) Start(ctx context.Context) error {
 		go s.reaper.Run(ctx)
 		s.Logger.Info("reaper started", "interval", cfg.Engine.ReaperInterval)
 
-		// Register liveness checkers for readyz.
-		workerThreshold := 3 * cfg.Engine.WorkerPollInterval
-		if workerThreshold < time.Second {
-			workerThreshold = 3 * time.Second
-		}
-		reaperThreshold := 3 * cfg.Engine.ReaperInterval
-		livenessCheckers = append(livenessCheckers,
-			&workerLivenessChecker{w: s.worker, threshold: workerThreshold},
-			&reaperLivenessChecker{r: s.reaper, threshold: reaperThreshold},
-		)
-
 		// Start retention cleanup if configured.
 		if cfg.Retention.ExecutionDays > 0 || cfg.Retention.AuditDays > 0 {
 			go func() {
@@ -270,9 +238,10 @@ func (s *Server) Start(ctx context.Context) error {
 		go s.pollQueueDepth(ctx, cfg.Engine.ReaperInterval)
 	}
 
-	// Health endpoints (registered after engine components so checkers are populated).
+	// Health endpoints. /readyz checks DB connectivity only; worker/reaper
+	// liveness is no longer gated here to avoid flapping between poll cycles.
 	mux.Handle("/healthz", health.HealthzHandler())
-	mux.Handle("/readyz", health.ReadyzHandler(s.DB, livenessCheckers...))
+	mux.Handle("/readyz", health.ReadyzHandler(s.DB))
 
 	// Wrap with metrics middleware (innermost, runs for every request).
 	handler := metricsMiddleware(mux)

packages/engine/internal/server/server.go

@@ -37,6 +37,8 @@ resources:
 
 podSecurityContext:
   runAsNonRoot: true
+  runAsUser: 10001
+  runAsGroup: 10001
 
 securityContext:
   readOnlyRootFilesystem: true
@@ -85,4 +87,9 @@ serviceMonitor:
   enabled: false
   interval: 30s
   labels: {}
+  # bearerTokenSecret: populate with a Kubernetes Secret that holds a Mantle
+  # API key so Prometheus can authenticate against /metrics. Example:
+  #   bearerTokenSecret:
+  #     name: mantle-prometheus-token
+  #     key: token
   bearerTokenSecret: {}