Try workaround for Bundler + git lfs issue (git-lfs/git-lfs#1889)#2
Merged
Conversation
Encountered an error when using `Gemfile` with `github: ` source to use soloist
gem from another project.
Error was:
Fetching https://github.com/LyraPhase/soloist.git
Downloading test/fixtures/cloud-init-seed.iso.qcow2 (459 KB)
Error downloading object: test/fixtures/cloud-init-seed.iso.qcow2 (a2643ef): Smudge error: Error downloading test/fixtures/cloud-init-seed.iso.qcow2 (a2643ef240ff6a0decafaeec8262c687c0d556000bfb6bfbbad559961b7e93be): error transferring "a2643ef240ff6a0decafaeec8262c687c0d556000bfb6bfbbad559961b7e93be": [0] remote missing object a2643ef240ff6a0decafaeec8262c687c0d556000bfb6bfbbad559961b7e93be
Errors logged to '/Users/exampleuser/src/pub/soloist/sprout-wrap/vendor/bundle/ruby/3.1.0/bundler/gems/soloist-537f1668364a/.git/lfs/logs/20250623T113632.054386.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: test/fixtures/cloud-init-seed.iso.qcow2: smudge filter lfs failed
Git error: command `git reset --hard 537f166` in directory /Users/exampleuser/src/pub/soloist/sprout-wrap/vendor/bundle/ruby/3.1.0/bundler/gems/soloist-537f1668364a has failed.
Revision 537f166 does not exist in the repository https://github.com/LyraPhase/soloist.git. Maybe you misspelled it?
If this error persists you could try removing the cache directory '/Users/exampleuser/src/pub/soloist/sprout-wrap/vendor/bundle/ruby/3.1.0/bundler/gems/soloist-537f1668364a'
Testing the `GIT_LFS_SKIP_SMUDGE=1 bundle install` command worked.
The repo using `git lfs` was being cloned by Bundler into a cache directory,
then re-cloned from that local filesystem location to the install location.
However, that local cached clone did not clone LFS files by default. Going into
the directory with `pushd $(bundle info --path soloist)` and showing the remote
with: `git remote -vv` shows what was going on.
$ pushd "$(bundle info --path soloist)"
$ pwd
/Users/exampleuser/src/pub/soloist/sprout-wrap/vendor/bundle/ruby/3.1.0/bundler/gems/soloist-537f1668364a
$ git remote -vv
git remote -vv
origin /Users/exampleuser/src/pub/soloist/sprout-wrap/vendor/bundle/ruby/3.1.0/cache/bundler/git/soloist-e4905c8fdc0d00e01f5e0e7c545bde55dee890e5 (fetch)
origin /Users/exampleuser/src/pub/soloist/sprout-wrap/vendor/bundle/ruby/3.1.0/cache/bundler/git/soloist-e4905c8fdc0d00e01f5e0e7c545bde55dee890e5 (push)
$ pushd /Users/exampleuser/src/pub/soloist/sprout-wrap/vendor/bundle/ruby/3.1.0/cache/bundler/git/soloist-e4905c8fdc0d00e01f5e0e7c545bde55dee890e5
$ git remote -vv
origin git@github.com:LyraPhase/soloist.git (fetch)
origin git@github.com:LyraPhase/soloist.git (push)
$ git branch -vv
develop 537f166 Merge pull request #1 from LyraPhase/upgrade-gems
upgrade-gems a369f81 Update README badges
* master 1adea10 Bump to 1.0.4
$ git checkout develop
fatal: this operation must be run in a work tree
$ ls
FETCH_HEAD HEAD config description hooks info lfs objects packed-refs refs
So, we can see the two repo paths where the first is cloned into Bundler's cache
from GitHub as a bare clone of `master` branch without a work tree. Any Git LFS
files were not pulled for the `develop` branch which bundler was trying to pull,
(`gem 'soloist', github: 'LyraPhase/soloist', ref: 'develop'` in `Gemfile`).
The second clone into the non-cache `vendor` directory is cloned from the cache,
using local filesystem as origin. However, when `develop` is checked out by
Bundler via `git reset --hard`, no LFS files are able to be cloned from the
cached clone directory because they were not pulled from GitHub.
That seems to be the crux of the issue. To workaround this, either we must
always use `GIT_LFS_SKIP_SMUDGE=1` when using `bundler install` with a
git source when that repo has LFS files, or try to force the upstream LFS
file location to be GitHub for all clones. That is what we will do here.
If that doesn't work, the only path left for convenient clones is to completely
rewrite history and remove usage of git LFS entirely from the repo. That'll be
our last resort. Otherwise, as `soloist` gem has been abandoned on RubyGems,
we will have to either rename & re-release on RubyGems or else always use a git
source in `Gemfile` + `GIT_LFS_SKIP_SMUDGE=1` for dependent projects. The only
file tracked by git LFS so far is the cloud-init seed ISO qcow2 image, and it's
only used during CI for the Vagrant VM so NFSv4 shares will work properly.
We may end up deleting this file and forcing regeneration during CI another way.
8f20df1 to
d02e2f2
Compare
Contributor
Author
|
It would seem that after testing this locally it should work. We will merge both to Another workaround found during testing would involve somehow hooking more commands during the initial $ git lfs fetch --all origin master develop
fetch: 2 objects found, done.Then the However, Bundler doesn't have a simple method to hook into this to integrate with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Encountered an error when using
Gemfilewithgithub:source to use soloistgem from another project.
Error was:
Testing the
GIT_LFS_SKIP_SMUDGE=1 bundle installcommand worked.The repo using
git lfswas being cloned by Bundler into a cache directory,then re-cloned from that local filesystem location to the install location.
However, that local cached clone did not clone LFS files by default. Going into
the directory with
pushd $(bundle info --path soloist)and showing the remotewith:
git remote -vvshows what was going on.So, we can see the two repo paths where the first is cloned into Bundler's cache
from GitHub as a bare clone of
masterbranch without a work tree. Any Git LFSfiles were not pulled for the
developbranch which bundler was trying to pull,(
gem 'soloist', github: 'LyraPhase/soloist', ref: 'develop'inGemfile).The second clone into the non-cache
vendordirectory is cloned from the cache,using local filesystem as origin. However, when
developis checked out byBundler via
git reset --hard, no LFS files are able to be cloned from thecached clone directory because they were not pulled from GitHub.
That seems to be the crux of the issue. To workaround this, either we must
always use
GIT_LFS_SKIP_SMUDGE=1when usingbundler installwith agit source when that repo has LFS files, or try to force the upstream LFS
file location to be GitHub for all clones. That is what we will do here.
If that doesn't work, the only path left for convenient clones is to completely
rewrite history and remove usage of git LFS entirely from the repo. That'll be
our last resort. Otherwise, as
soloistgem has been abandoned on RubyGems,we will have to either rename & re-release on RubyGems or else always use a git
source in
Gemfile+GIT_LFS_SKIP_SMUDGE=1for dependent projects. The onlyfile tracked by git LFS so far is the cloud-init seed ISO qcow2 image, and it's
only used during CI for the Vagrant VM so NFSv4 shares will work properly.
We may end up deleting this file and forcing regeneration during CI another way.