Skip to content

Commit 1bd8cbf

Browse files
michaelmcneesclaudeMichael McNees
authored
feat: git sync — webhook, prune, manual flow (Plan C of issue #16) (#140)
* feat(db): add 020_git_sync_prune migration Adds disabled_at column to workflow_definitions and git_repo_workflows join table to support prune behavior for GitOps sync (issue #16). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(audit): add git.sync.pruned and git.push.received * feat(workflow): add Disable/Reenable for GitOps prune Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(repo/sync): add git_repo_workflows helpers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(repo/sync): wire prune into SyncRepo Add Pruned field to Report, call RecordSeen+Reenable per file, and drive Disable+audit+RemoveSeenRecords for stale workflows when Prune is set. Use DB clock for syncStart to avoid Go/Postgres time skew. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(repo/sync): sanitize URLs in audit metadata Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(server): add /hooks/git/<id> webhook receiver Registers POST /hooks/git/<repo-id>: HMAC-SHA256 verifies against webhook_secret, emits git.push.received audit event, fires SyncRepo in a goroutine for immediate 202 response. Adds Store.GetByID (no team_id filter) and wires RepoStore/GitDriver into Server. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(repo): clean up cloned working tree on Delete Adds a cloneBasePath parameter to Store.Delete so the CLI removes the on-disk clone directory (filepath.Join(cloneBasePath, repoID)) after the DB transaction commits. Filesystem failures are logged via slog.Warn but never fail the call — an orphan directory is preferable to an orphan row. The CLI resolves the path from cfg.Storage.Path (same fallback as serve). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(cli): add mantle repos update Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat: add mantle repos plan and apply for manual flow Adds PlanRepo (dry-run: pull + discover + classify without any DB writes or audit events), and exposes it via `mantle repos plan <name>`. Also adds `mantle repos apply <name>` as an explicit manual-sync alias for operators using auto_apply:false repos. Refactors newReposSyncCommand to share buildSyncDriver so all three commands use identical driver-selection logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs: document Plan C commands and webhook endpoint Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: address pre-push reviewer feedback for Plan C - Add --webhook-secret flag to repos add and repos update (blocker) - Remove stray fmt.Println from webhook test - Clarify plan/apply/update help text, mention commands in repos Long - Sanitize last_sync_error for pull/discover failure paths Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(server): use request context in webhook handler to resolve G118 gosec G118 (CWE-400) flags goroutines that use context.Background() when a request-scoped context is available. Two fixes: - emitGitAudit: called synchronously while the request is still in flight, so r.Context() is correct and preserves trace/deadline info. - background SyncRepo goroutine: context.WithoutCancel(r.Context()) propagates request values (e.g. trace IDs) without tying the goroutine's lifetime to the HTTP request, so it survives after the 202 response is sent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: compile errors and govulncheck exclusions on feat/git-sync-complete - Add WebhookSecret field to Repo struct (was dropped during rebase) - Restore reposCtx to return (context.Context, *config.Config); Plan C tests call store.Get with a real context — the 1-return form was wrong - Update govulncheck exclusions to include go-git transitive dep vulns (matching feat/git-sync-engine) and refactor with KNOWN variable Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(db): add migration 021 for webhook_secret column on git_repos Migration 019 intentionally deferred this column to Plan C. The column is required by store.Create and store.Update which pass webhook_secret as parameter $10 in the INSERT/UPDATE statements. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(repo): include webhook_secret in store.Get query store.Get selected 15 columns but did not include webhook_secret, so the field always read back as empty string. Added it to the SELECT and Scan alongside the other nullable fields, matching store.GetByID. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Michael McNees <mmcnees@Michaels-MacBook-Pro.local>
1 parent 7d76ee8 commit 1bd8cbf

21 files changed

Lines changed: 1333 additions & 101 deletions

CLAUDE.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,9 @@ mantle repos list # List registered repos with last-sync status
8585
mantle repos status <name> # Show detailed repo status
8686
mantle repos remove <name> -y # Unregister a repo (requires --yes)
8787
mantle repos sync <name> # Force an immediate sync of a registered repo
88+
mantle repos plan <name> # Dry-run: show what a sync would change
89+
mantle repos apply <name> # Manual sync (for auto_apply:false repos)
90+
mantle repos update <name> --credential <cred> # Update a repo's mutable fields
8891
mantle run <wf> --values f.yaml # Run with a values file (inputs + env overrides)
8992
mantle run <wf> --env <name> # Run against a stored named environment
9093
mantle plan <wf> --env <name> # Plan; appends resolved inputs/env with source
@@ -115,7 +118,7 @@ git_sync:
115118
116119
Credentials of type `git` accept `token` (for HTTPS), `ssh_key` (for SSH), and optional `username`. At least one of `token` or `ssh_key` is required.
117120

118-
> Plan A shipped the repo registry and CLI. Plan B ships the sync engine — when `mantle serve` runs, it reconciles this block into `git_repos` rows and then polls each enabled `auto_apply: true` repo every `poll_interval`, applying any changed workflow YAML. Force an immediate sync with `mantle repos sync <name>`.
121+
> **GitOps is fully shipped.** `mantle serve` reconciles this block into `git_repos` rows, then polls each enabled `auto_apply: true` repo every `poll_interval`. Workflows removed from the repo are disabled automatically when `prune: true`. Force an on-demand sync via `mantle repos sync <name>`, preview with `mantle repos plan <name>`, or manually apply via `mantle repos apply <name>`. Register push webhooks from GitHub / GitLab / Gitea pointing at `POST /hooks/git/<repo-id>` with `webhook_secret` set on the repo to trigger immediate syncs — Mantle verifies the HMAC signature via `X-Hub-Signature-256`.
119122

120123
## License
121124

packages/engine/internal/audit/audit.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,8 @@ const (
7272
ActionGitSyncFailed Action = "git.sync.failed"
7373
ActionGitSyncValidationFailed Action = "git.sync.validation_failed"
7474
ActionGitSyncApplyFailed Action = "git.sync.apply_failed"
75+
ActionGitSyncPruned Action = "git.sync.pruned"
76+
ActionGitPushReceived Action = "git.push.received"
7577
)
7678

7779
// Resource identifies the target of an audit event.

packages/engine/internal/cli/repos.go

Lines changed: 175 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -23,15 +23,20 @@ func newReposCommand() *cobra.Command {
2323
Short: "Manage GitOps workflow source repositories",
2424
Long: `Registers GitOps source repositories whose workflow YAML definitions are
2525
synced into this Mantle instance. This command manages the registry; use
26-
` + "`mantle repos sync`" + ` for on-demand syncs, or start ` + "`mantle serve`" + ` to let the
27-
background poller run. Auth material is stored in a "git" credential type
28-
(` + "`mantle secrets create --type git`" + `) and referenced here by name.`,
26+
` + "`mantle repos sync`" + ` for ad-hoc syncs, ` + "`mantle repos plan`" + ` to preview changes
27+
without writing, ` + "`mantle repos apply`" + ` for manual control over auto_apply:false
28+
repos, or start ` + "`mantle serve`" + ` to let the background poller run. Auth material
29+
is stored in a "git" credential type (` + "`mantle secrets create --type git`" + `) and
30+
referenced here by name.`,
2931
}
3032
cmd.AddCommand(newReposAddCommand())
33+
cmd.AddCommand(newReposUpdateCommand())
3134
cmd.AddCommand(newReposListCommand())
3235
cmd.AddCommand(newReposStatusCommand())
3336
cmd.AddCommand(newReposRemoveCommand())
3437
cmd.AddCommand(newReposSyncCommand())
38+
cmd.AddCommand(newReposPlanCommand())
39+
cmd.AddCommand(newReposApplyCommand())
3540
return cmd
3641
}
3742

@@ -51,7 +56,7 @@ func newRepoStore(cmd *cobra.Command) (*repo.Store, func(), error) {
5156
}
5257

5358
func newReposAddCommand() *cobra.Command {
54-
var url, branch, path, pollInterval, credential string
59+
var url, branch, path, pollInterval, credential, webhookSecret string
5560
var autoApply, prune bool
5661

5762
cmd := &cobra.Command{
@@ -70,14 +75,15 @@ credential must already exist and be of type "git".`,
7075
defer cleanup()
7176

7277
r, err := store.Create(cmd.Context(), repo.CreateParams{
73-
Name: args[0],
74-
URL: url,
75-
Branch: branch,
76-
Path: path,
77-
PollInterval: pollInterval,
78-
Credential: credential,
79-
AutoApply: autoApply,
80-
Prune: prune,
78+
Name: args[0],
79+
URL: url,
80+
Branch: branch,
81+
Path: path,
82+
PollInterval: pollInterval,
83+
Credential: credential,
84+
AutoApply: autoApply,
85+
Prune: prune,
86+
WebhookSecret: webhookSecret,
8187
})
8288
if err != nil {
8389
return err
@@ -94,6 +100,8 @@ credential must already exist and be of type "git".`,
94100
cmd.Flags().StringVar(&credential, "credential", "", "Git credential name (required)")
95101
cmd.Flags().BoolVar(&autoApply, "auto-apply", true, "Automatically apply changes (false = plan-only)")
96102
cmd.Flags().BoolVar(&prune, "prune", true, "When true, workflows deleted from the repo are disabled in Mantle")
103+
cmd.Flags().StringVar(&webhookSecret, "webhook-secret", "",
104+
"HMAC secret for push webhook verification (empty = no verification)")
97105
_ = cmd.MarkFlagRequired("url")
98106
_ = cmd.MarkFlagRequired("credential")
99107

@@ -168,7 +176,16 @@ command only stops future syncs. Requires --yes to confirm.`,
168176
}
169177
defer cleanup()
170178

171-
if err := store.Delete(cmd.Context(), args[0]); err != nil {
179+
cfg := config.FromContext(cmd.Context())
180+
cloneBase := ""
181+
if cfg != nil {
182+
artifactBase := cfg.Storage.Path
183+
if artifactBase == "" {
184+
artifactBase = filepath.Join(os.TempDir(), "mantle-artifacts")
185+
}
186+
cloneBase = filepath.Join(artifactBase, "git")
187+
}
188+
if err := store.Delete(cmd.Context(), args[0], cloneBase); err != nil {
172189
return err
173190
}
174191
fmt.Fprintf(cmd.OutOrStdout(), "Removed repo %q\n", args[0])
@@ -219,6 +236,51 @@ state, and timestamp of the last successful sync.`,
219236
}
220237
}
221238

239+
func newReposUpdateCommand() *cobra.Command {
240+
var branch, path, pollInterval, credential, webhookSecret string
241+
var autoApply, prune, enabled bool
242+
243+
cmd := &cobra.Command{
244+
Use: "update <name>",
245+
Short: "Update a registered repo's mutable fields",
246+
Long: `Replaces the branch, path, poll-interval, credential, auto-apply, prune,
247+
and enabled flags on an existing repo. URL and name are immutable — to
248+
change them, delete and recreate.
249+
All flags must be provided — update replaces the full set of mutable fields.
250+
Omitting a flag resets that field to its default value.`,
251+
Args: cobra.ExactArgs(1),
252+
RunE: func(cmd *cobra.Command, args []string) error {
253+
store, cleanup, err := newRepoStore(cmd)
254+
if err != nil {
255+
return err
256+
}
257+
defer cleanup()
258+
259+
r, err := store.Update(cmd.Context(), args[0], repo.UpdateParams{
260+
Branch: branch, Path: path, PollInterval: pollInterval,
261+
Credential: credential, AutoApply: autoApply, Prune: prune, Enabled: enabled,
262+
WebhookSecret: webhookSecret,
263+
})
264+
if err != nil {
265+
return err
266+
}
267+
fmt.Fprintf(cmd.OutOrStdout(), "Updated repo %q\n", r.Name)
268+
return nil
269+
},
270+
}
271+
cmd.Flags().StringVar(&branch, "branch", "main", "Branch to sync")
272+
cmd.Flags().StringVar(&path, "path", "/", "Subdirectory inside the repo to scan")
273+
cmd.Flags().StringVar(&pollInterval, "poll-interval", "60s", "Interval between syncs")
274+
cmd.Flags().StringVar(&credential, "credential", "", "Git credential name (required)")
275+
cmd.Flags().BoolVar(&autoApply, "auto-apply", true, "Automatically apply changes")
276+
cmd.Flags().BoolVar(&prune, "prune", true, "Disable workflows deleted from the repo")
277+
cmd.Flags().BoolVar(&enabled, "enabled", true, "Whether the repo is active")
278+
cmd.Flags().StringVar(&webhookSecret, "webhook-secret", "",
279+
"HMAC secret for push webhook verification (empty = no verification)")
280+
_ = cmd.MarkFlagRequired("credential")
281+
return cmd
282+
}
283+
222284
func newReposSyncCommand() *cobra.Command {
223285
var fromDir string
224286
cmd := &cobra.Command{
@@ -240,25 +302,7 @@ of cloning (useful for tests and CI-driven deployments).`,
240302
if err != nil {
241303
return err
242304
}
243-
var driver sync.Driver
244-
if fromDir != "" {
245-
driver = &sync.NoopDriver{BasePath: fromDir}
246-
} else {
247-
var secretResolver *secret.Resolver
248-
if cfg := config.FromContext(cmd.Context()); cfg != nil && cfg.Encryption.Key != "" {
249-
encryptor, encErr := secret.NewEncryptor(cfg.Encryption.Key)
250-
if encErr != nil {
251-
return fmt.Errorf("configuring encryption for git sync: %w", encErr)
252-
}
253-
secretResolver = &secret.Resolver{
254-
Store: &secret.Store{DB: store.DB, Encryptor: encryptor},
255-
}
256-
}
257-
driver = &sync.GoGitDriver{
258-
BasePath: filepath.Join(os.TempDir(), "mantle-repos"),
259-
Auth: sync.NewAuthResolver(cmd.Context(), secretResolver),
260-
}
261-
}
305+
driver := buildSyncDriver(cmd, store, fromDir)
262306
report, err := sync.SyncRepo(cmd.Context(), store.DB, store, r, driver)
263307
if err != nil {
264308
return fmt.Errorf("sync %q: %w", r.Name, err)
@@ -276,3 +320,102 @@ of cloning (useful for tests and CI-driven deployments).`,
276320
cmd.Flags().StringVar(&fromDir, "from-dir", "", "Path to an already-cloned repo directory (skips git pull; useful for CI pipelines or local testing)")
277321
return cmd
278322
}
323+
324+
func newReposPlanCommand() *cobra.Command {
325+
var fromDir string
326+
cmd := &cobra.Command{
327+
Use: "plan <name>",
328+
Short: "Dry-run: show what a sync would change",
329+
Long: `Plans a sync without writing anything: pulls the repo, discovers files,
330+
and classifies each as "would apply" (new or changed) or "unchanged". Use
331+
before mantle repos apply to preview pending changes before they land.`,
332+
Args: cobra.ExactArgs(1),
333+
RunE: func(cmd *cobra.Command, args []string) error {
334+
store, cleanup, err := newRepoStore(cmd)
335+
if err != nil {
336+
return err
337+
}
338+
defer cleanup()
339+
340+
r, err := store.Get(cmd.Context(), args[0])
341+
if err != nil {
342+
return err
343+
}
344+
driver := buildSyncDriver(cmd, store, fromDir)
345+
report, err := sync.PlanRepo(cmd.Context(), store.DB, r, driver)
346+
if err != nil {
347+
return fmt.Errorf("plan %q: %w", r.Name, err)
348+
}
349+
fmt.Fprintf(cmd.OutOrStdout(),
350+
"Plan for %s — would apply=%d unchanged=%d failures=%d (SHA %s)\n",
351+
r.Name, report.Applied, report.Unchanged, len(report.Failures), report.SHA)
352+
for _, f := range report.Failures {
353+
fmt.Fprintf(cmd.ErrOrStderr(), " FAIL %s: %s\n", f.RelPath, f.Err)
354+
}
355+
return nil
356+
},
357+
}
358+
cmd.Flags().StringVar(&fromDir, "from-dir", "", "Path to an already-cloned repo directory (skips git pull; useful for CI pipelines or local testing)")
359+
return cmd
360+
}
361+
362+
func newReposApplyCommand() *cobra.Command {
363+
var fromDir string
364+
cmd := &cobra.Command{
365+
Use: "apply <name>",
366+
Short: "Apply pending workflow changes from a repo",
367+
Long: `Runs one sync pass for the named repo. Useful when auto_apply is false and you
368+
want explicit control over when workflow YAML lands in Mantle. Also works for
369+
auto_apply:true repos when you want to apply immediately without waiting for
370+
the next poll.`,
371+
Args: cobra.ExactArgs(1),
372+
RunE: func(cmd *cobra.Command, args []string) error {
373+
store, cleanup, err := newRepoStore(cmd)
374+
if err != nil {
375+
return err
376+
}
377+
defer cleanup()
378+
379+
r, err := store.Get(cmd.Context(), args[0])
380+
if err != nil {
381+
return err
382+
}
383+
driver := buildSyncDriver(cmd, store, fromDir)
384+
report, err := sync.SyncRepo(cmd.Context(), store.DB, store, r, driver)
385+
if err != nil {
386+
return fmt.Errorf("apply %q: %w", r.Name, err)
387+
}
388+
fmt.Fprintf(cmd.OutOrStdout(),
389+
"Applied %s — applied=%d unchanged=%d failures=%d (SHA %s)\n",
390+
r.Name, report.Applied, report.Unchanged, len(report.Failures), report.SHA)
391+
for _, f := range report.Failures {
392+
fmt.Fprintf(cmd.ErrOrStderr(), " FAIL %s: %s\n", f.RelPath, f.Err)
393+
}
394+
return nil
395+
},
396+
}
397+
cmd.Flags().StringVar(&fromDir, "from-dir", "", "Path to an already-cloned repo directory (skips git pull; useful for CI pipelines or local testing)")
398+
return cmd
399+
}
400+
401+
// buildSyncDriver picks NoopDriver when --from-dir is set, otherwise a
402+
// GoGitDriver rooted at a temp path. Extracted from newReposSyncCommand so
403+
// plan and apply don't duplicate the branching logic.
404+
func buildSyncDriver(cmd *cobra.Command, store *repo.Store, fromDir string) sync.Driver {
405+
if fromDir != "" {
406+
return &sync.NoopDriver{BasePath: fromDir}
407+
}
408+
var secretResolver *secret.Resolver
409+
if cfg := config.FromContext(cmd.Context()); cfg != nil && cfg.Encryption.Key != "" {
410+
encryptor, encErr := secret.NewEncryptor(cfg.Encryption.Key)
411+
if encErr == nil {
412+
secretResolver = &secret.Resolver{
413+
Store: &secret.Store{DB: store.DB, Encryptor: encryptor},
414+
}
415+
}
416+
}
417+
return &sync.GoGitDriver{
418+
BasePath: filepath.Join(os.TempDir(), "mantle-repos"),
419+
Auth: sync.NewAuthResolver(cmd.Context(), secretResolver),
420+
}
421+
}

0 commit comments

Comments
 (0)