feat: support sparse + shallow git clones - #481
Conversation
|
Hi @GavinPJK. Thanks for your PR. I'm waiting for a jenkins-x-plugins member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
/ok-to-test |
There was a problem hiding this comment.
Pull request overview
This PR adds optional sparse-checkout and shallow-clone behavior to jx promote to reduce clone size/time, with special handling for file/kpt promotion rules and additional test coverage.
Changes:
- Add
--sparse-checkout/--sparse-checkout-patternflags and thread sparse-checkout settings into environment/dev-env cloning. - Implement default sparse-checkout patterns for environment repos, plus runtime expansion for file/kpt rule target paths.
- Add unit + integration tests validating sparse vs full clone promotion output and shallow-clone behavior.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/promote/promote.go | Adds CLI flags and enables dev-env sparse clone behavior. |
| pkg/promote/pr.go | Derives/expands sparse-checkout patterns for file/kpt rules during promotion. |
| pkg/promote/pr_internal_test.go | Unit tests for deriving sparse patterns from promote specs. |
| pkg/environments/types.go | Adds SparseCheckout option to environment PR options. |
| pkg/environments/gitops.go | Implements sparse clone w/ default patterns + shallow-clone rules + fallback to full clone. |
| pkg/envctx/env_context.go | Adds SparseCheckout toggle to dev environment context. |
| pkg/envctx/load.go | Switches dev-env clone to sparse clone of versionStream/ when enabled. |
| pkg/promote/promote_integration_test.go | Adds integration tests comparing sparse vs full promotion output and shallow behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| var cloneDir string | ||
| if e.SparseCheckout { | ||
| cloneDir, err = gitclient.SparseCloneToDir(gitter, gitCloneURL, "", true, "/versionStream/") | ||
| } else { | ||
| cloneDir, err = gitclient.CloneToDir(gitter, gitCloneURL, "") | ||
| } | ||
| if err != nil { | ||
| return fmt.Errorf("failed to clone URL %s: %w", gitCloneURL, err) | ||
| } |
| // git sparse-checkout add appends the pattern and re-applies it to the working | ||
| // tree, lazily fetching the now-included blobs from the (blobless) clone | ||
| if _, err := o.Gitter.Command(dir, "sparse-checkout", "add", pattern); err != nil { | ||
| return fmt.Errorf("failed to expand sparse checkout with file/kpt rule path %q in dir %s: %w", pattern, dir, err) | ||
| } | ||
| log.Logger().Infof("expanded sparse checkout to include file/kpt rule path %s", pattern) |
msvticket
left a comment
msvticket
left a comment
There was a problem hiding this comment.
Configuring sparse checkout exclusively with command line argument is potentially problematic since it means that the step needs to be overridden in the pipeline.
And if a user does that any changes we do jx3-pipeline-catalog wont go through to that user. I'd prefer if this was configured in .jx/settings.yaml.
|
/retest |
I see - I will look at adding something like I'll set this as a draft in the meantime. |
Changes
--sparse-checkoutand--sparse-checkout-patternflags topromote, applying to both cluster and application repositories:helmfile.yaml,helmfiles/,.jx/,env/) used when no explicit patterns are givenContext
Fully cloning the environment and application repositories is slow and wasteful when:
A sparse checkout fetches only the paths a promotion needs (default:
helmfile.yaml,helmfiles/,.jx/,env/), while a shallow clone ignores the git history from the fetched branch. Enabling these enables faster release steps, particularly for larger repos with large commit histories.There are exceptions to this that need to be handled with care, namely: