git: shallow clone via ls-remote for sanity, simplify mirror logic#27
git: shallow clone via ls-remote for sanity, simplify mirror logic#27gastmaier wants to merge 13 commits intoanalogdevicesinc:mainfrom
Conversation
The method is not used anywhere, remove it to simplify maintanence. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
It is a pre-requisite to have a functional git instance falling back on error, only on windows, is not useful for subsequent commands. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
The sha of the ls-remote command was being discarted, but is useful to match a sha to a branch,tag, and vice-versa. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Method to fetch a single reference (sha, branch, tag, ...) with a specified depth (usually 1). Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Remove the shallow methods, to refactor in a refactored clone_repo with shallow clone capability. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Add method to check if object, like commit shas, exists. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Determine the fetch refspec, update-ref target, and resolved SHA from ls-remote pairs. Returns (fetch_refspec, update_ref_name, sha), in order of precedence: - Tag v1.0 : (refs/tags/v1.0, refs/tags/v1.0, Some(sha)) - Branch foo : (refs/heads/foo, refs/heads/foo, Some(sha)) - Commit SHA : (sha, refs/heads/trunk, None) Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Test corner cases of the resolution method. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
The method was only called for the no-mirror or mirror not found condition, for the default with mirror, the original full clone happens without any feedback. Clean-up the method for now. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Now works uniformly for branches (refs/heads/*), tags (refs/tags/*), commit SHAs, and HEAD. Uses init -> remote add -> fetch -> checkout FETCH_HEAD to achieve this. Drops argument reference (not useful for same host with write access), and adds depth to set the depth to fetch the contents. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Use git ls-remote returned tuple for branch and tag detection, instead of relying in the local copy. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
Remove the methods that rely on a full clone of the repository: - is_branch_reference - get_latest_commit_for_branch - get_latest_commit_for_remote_branch Signed-off-by: Jorge Marques <jorge.marques@analog.com>
There was a fairly amount of duplication due to _no_mirror variants of the same methods. Merge them with a no_mirror boolean parameter instead. Signed-off-by: Jorge Marques <jorge.marques@analog.com>
| if !git_operations::cat_file(&mirror_repo_path, &target) { | ||
| let fetch_ok = | ||
| git_operations::fetch_ref(&mirror_repo_path, "origin", &fetch_refspec, 1) | ||
| .is_ok_and(|r| r.is_success()); | ||
| if fetch_ok { | ||
| let _ = git_operations::update_ref( | ||
| &mirror_repo_path, | ||
| &update_ref_name, | ||
| "FETCH_HEAD", | ||
| ); |
There was a problem hiding this comment.
This is the part that becomes unnecessary if we use worktrees, all mirror paths vanish.
|
Now I have spent a good amount of time on this PR and here is my summary. First, I wanted to compare time and size from different stages, with a script I did three things. Setup from scratch (empty mirrors), setup with mirrors primed and then run a "git remote update" in each gits. This is the "jupiter-sdr" setup that involves Linux, U-boot and TF-A. Here are the stats ( Some reflections:
Mirror In relation to this I've also been thinking about whether worktree's would work or not. Technically it would, but in practice I believe it will be problematic, perhaps not if you as a individual is the only user or the mirror, but if the mirror is stored as a shared mirror, then we would have different users contributing with different branches etc. Workspace PR continuation This patch https://github.com/adi-innersource/sdk-manager/commit/95f35cdcc1e9888fae38be23c01c8d65e268cf11 could be worth mention as well, which was a try to ensure that mirror finds and fetches updates from the upstream trees. We had an issue that branches that we're created wasn't found in the mirror and then the workspace setup failed. Do you think you can rework this PR, to a) skip things touching the mirror b) implement a |
|
I have taken most patches in this series and merged them to main. They mirror feature has been left out, i.e., the old mirror feature is as is. I took the liberty to change the git commit messages a bit. I think we can close this PR now. |
|
Acked, closing it! We definitely need a more broad approach than this pr touched for the shallow improvements. some ideas for the future: Noted on |
2 participants
Reworks the git operations layer to assume shallow clone.
Minor change:
(sha, refname)tuples instead ofrefname, allows to resolve a symbolic ref to its exact object without a local clone.New:
Main changes:
Simplification:
The _no_mirror variants of clone_repo_to_workspace, handle_existing_workspace_repo, update_workspace_repos, and update_workspace_repos_with_result are merged into their counterparts with a added no_mirror: bool / Option<&Path> parameter.
Notes:
Since the
mirroris a local copy, a better alternative is to use worktrees, and bring themirrorfeature in a future instance as a real, Ethernet or read-only storage, mirror.There is a new scoped thread that allows to drop many .clone() in the source code.