Retrieving a git::http artifact fails on Modern distros

[ivanff]

Nomad version

nomad 1.9.5
nomad 1.8.4

Operating system and Environment details

Centos 10 Stream

Issue

Problems downloading artifacts from git repositories. The git client cannot access some directories when downloading artifacts.
This is evident on Linux operating systems with a 6 kernel on which the landlock kernel module is enabled. I was able to solve this problem by disabling it through the download menu.

Reproduction steps

Expected Result

Actual Result

Job file (if appropriate)

artifact { source = "git::https://nomad:***@***/infra/nomad-templates.git?ref=master" destination = "$${NOMAD_TASK_DIR}/templates" options { depth = 1 } }

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

Failed Artifact Download: failed to download artifact "git::https://nomad:@***/infra/nomad-templates.git": getter subprocess failed: exit status 1: failed to download artifact: error downloading 'https://nomad:redacted@***/infra/nomad-templates.git': /usr/bin/git exited with 128: Cloning into '/var/nomad/alloc/724e4df7-b08e-dc5e-8a21-6fa77b2299cf/nginx/local/templates'... warning: templates not found in /usr/share/git-core/templates error: unable to get random bytes for temporary file: Permission denied error: unable to get random bytes for temporary file: Permission denied fatal: Unable to create temporary file '/var/nomad/alloc/724e4df7-b08e-dc5e-8a21-6fa77b2299cf/nginx/local/templates/.git/objects/pack/tmp_pack_XXXXXX': Permission denied fatal: fetch-pack: invalid index-pack output

[ivanff]

I resolve this by edit boot kernel flags
/boot/loader/entries/* files
lsm=lockdown,capability,yama,selinux,bpf,ima,evm

if you are use GRUB_ENABLE_BLSCFG=true

[gulducat]

Hey there @ivanff, thanks for the report.

I'm not able to reproduce this on my fedora laptop with landlock enabled, so I wonder if something else might be at play.

Here are my reproduction steps ⬅️

# uname -r
6.16.12-200.fc42.x86_64

# dmesg | grep landlock
[    0.127575] LSM: initializing lsm=lockdown,capability,yama,selinux,bpf,landlock,ipe,ima,evm
[    0.127575] landlock: Up and running.

I'm running a basic dev agent like this

sudo nomad-1.9.5 agent -dev

and this job file using a public github repo:

job "art" {
  group "g" {
    task "t" {
      artifact {
        source      = "git::https://github.com/gulducat/gulducat.git?ref=main"
        destination = "${NOMAD_TASK_DIR}/templates"
      }
      driver = "docker"
      config {
        image   = "busybox:1"
        command = "sleep"
        args    = ["infinity"]
      }
    }
  }
}

Nomad agent logs show the artifact being downloaded

[DEBUG] client.alloc_runner.task_runner.task_hook.artifacts: downloading artifact: alloc_id=2860137a-b9c2-dca7-7c35-0edbe927d6fa task=t artifact=git::https://github.com/gulducat/gulducat.git?ref=main
[DEBUG] client.artifact: get: source=git::https://github.com/gulducat/gulducat.git?ref=main destination="${NOMAD_TASK_DIR}/templates" user=""
[DEBUG] client.artifact: sub-process: OUTPUT="artifact download was a success"

and the repo's in the target directory:

# ls -alh /tmp/NomadClient1976956180/2860137a-b9c2-dca7-7c35-0edbe927d6fa/t/local/templates/
total 4.0K
drwxr-xr-x. 3 root   root    80 Oct 21 13:24 .
drwxrwxrwx. 3 nobody nobody  60 Oct 21 13:24 ..
drwxr-xr-x. 5 root   root   180 Oct 21 13:24 .git
-rw-r--r--. 1 root   root   665 Oct 21 13:24 README.md

---

Among the git command errors in the output you provided

/usr/bin/git exited with 128: Cloning into '/var/nomad/alloc/724e4df7-b08e-dc5e-8a21-6fa77b2299cf/nginx/local/templates'... warning: templates not found in /usr/share/git-core/templates error: unable to get random bytes for temporary file: Permission denied error: unable to get random bytes for temporary file: Permission denied fatal: Unable to create temporary file '/var/nomad/alloc/724e4df7-b08e-dc5e-8a21-6fa77b2299cf/nginx/local/templates/.git/objects/pack/tmp_pack_XXXXXX': Permission denied fatal: fetch-pack: invalid index-pack output

I notice along with "permission denied", it also says unable to get random bytes for temporary file, which some sources online say can either be a faulty git install, or issues with git cache. So I wonder if something like that might be the cause, rather than landlock affecting git artifacts more broadly.

If that helps resolve the issue, or if I missed an important reproduction step or other evidence that could narrow it down to landlock, please let me know!

[ivanff]

Very interestingly, I used a clean Centos 10 Stream image provided by the provider, without any changes, with packages from the official repositories.
I tried to work locally from the user under whom nomad (root) is running and upload the artifact as a git clone and did not experience any problems, everything was uploaded normally. moreover, I installed go-getter, which also downloaded the url using exactly the same link as nomad.

now I will try to turn on the landlock module and try to download it again, I will unsubscribe what happens.

[ivanff]

I checked it again. And everything is the same.

root@edge-vps-Stage 2 merge ~]# cat/sys/kernel/security/lsm
lock, features, yama, selinux, bpf, fixed lock, ima, evm

Failed Artifact Download: failed to download artifact "git::https://nomad:***@**/infra/nomad-templates.git": getter subprocess failed: exit status 1: failed to download artifact: error downloading 'https://nomad:redacted@**/infra/nomad-templates.git?depth=1': /usr/bin/git exited with 128: Cloning into '/var/nomad/alloc/e5029b3e-3eee-b85e-9087-a42f05cf0f6c/nginx/local/templates'... warning: templates not found in /usr/share/git-core/templates error: unable to get random bytes for temporary file: Permission denied error: unable to get random bytes for temporary file: Permission denied fatal: Unable to create temporary file '/var/nomad/alloc/e5029b3e-3eee-b85e-9087-a42f05cf0f6c/nginx/local/templates/.git/objects/pack/tmp_pack_XXXXXX': Permission denied fatal: fetch-pack: invalid index-pack output

your's job

Failed Artifact Download: failed to download artifact "git::https://github.com/gulducat/gulducat.git?ref=main": getter subprocess failed: exit status 1: failed to download artifact: error downloading 'https://github.com/gulducat/gulducat.git?ref=main': /usr/bin/git exited with 128: Cloning into '/var/nomad/alloc/5dbbace5-6b82-5844-30db-0d9cc091c6b4/t/local/templates'... warning: templates not found in /usr/share/git-core/templates error: unable to get random bytes for temporary file: Permission denied error: unable to get random bytes for temporary file: Permission denied fatal: Unable to create temporary file '/var/nomad/alloc/5dbbace5-6b82-5844-30db-0d9cc091c6b4/t/local/templates/.git/objects/pack/tmp_pack_XXXXXX': Permission denied fatal: fetch-pack: invalid index-pack output

as soon as I put the boot in the kernel, everything is ok.

options root=UUID=c7322415-8b4f-47d2-b361-1368c4208a22 ro console=ttyS0,115200n8 no_timer_check crashkernel=2G-64G:256M,64G-:512M lsm=lockdown,capability,yama,selinux,bpf,ima,evmk

There is still a moment, perhaps worthy of a separate task. but that doesn't apply to the current problem.
for artifacts, you have to set passwords in the uri, which lights them up in the logs and is generally not secure.
it would be cool to be able to use the git-credentials store for the nomad user, who, again, does not allow this due to the landing

failed to download artifact "git::https://git.bit5.ru/infra/drm_404page.git": getter subprocess failed: exit status 1: failed to download artifact: error downloading 'https://git.bit5.ru/infra/drm_404page.git?depth=1': /usr/bin/git exited with 128: warning: unable to access '/root/.gitconfig': Permission denied warning: unable to access '/root/.gitconfig': Permission denied fatal: unknown error occurred while reading the configuration files