For now, if a job is using a docker image, the root source directory is copied under a specific mount point in a specific container. That mountpoint (ciProjectDir) is bound to a volume to save files for later.
That works great, but not when using LFS.
I have some huge files in LFS (that’s the point) and those files are:
- listed in
.dockerignore
- not checked out in CI (
GIT_LFS_SKIP_SMUDGE=1)
But for obvious reason, I have them checked out locally.
gitlab-ci-local uses the command docker cp to sync my local folder to ciProjectDir in the temporary container. That takes LOTS of time because docker first create a tar file of the current directory then send it back to the daemon, then the daemon read the tar and extract it to the final location.
When multiple jobs are run, it takes ages to run.
Unfortunately docker cp has no meaning to pass exclude files.
I was thinking of a more convoluted mean:
- bind mount the source to a specific folder in the container in read-only (that is quick)
- use other way to sync that folder to
ciProjectDir (rsync ?) by converting .dockerignore file to a format valid for the tool (rsync or another)
Any better option is also valid…
For now, if a job is using a docker image, the root source directory is copied under a specific mount point in a specific container. That mountpoint (
ciProjectDir) is bound to a volume to save files for later.That works great, but not when using LFS.
I have some huge files in LFS (that’s the point) and those files are:
.dockerignoreGIT_LFS_SKIP_SMUDGE=1)But for obvious reason, I have them checked out locally.
gitlab-ci-localuses the commanddocker cpto sync my local folder tociProjectDirin the temporary container. That takes LOTS of time because docker first create a tar file of the current directory then send it back to the daemon, then the daemon read the tar and extract it to the final location.When multiple jobs are run, it takes ages to run.
Unfortunately
docker cphas no meaning to pass exclude files.I was thinking of a more convoluted mean:
ciProjectDir(rsync?) by converting.dockerignorefile to a format valid for the tool (rsyncor another)Any better option is also valid…