Misleading payu error when message when cloning a repo owned by someone else

[manodeep]

I was trying to clone @blimlim ESM1.5 experiment with payu, and while investigating, I found out that the error messages were different or did not occur at all, depending on the version of git used.

Error message when cloning with payu/dev

[~ @gadi-login-04]$ module use /g/data/vk83/prerelease/modules
[~ @gadi-login-04]$ module load payu/dev
Loading payu/dev-20250921T211851Z-d026d1e
  Loading requirement: singularity
[~ @gadi-login-04]$ payu clone  /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156 /g/data/tm70/ms2335/performance_optimisation_runs/classic/access-esm1p6/intel@19.0.3.199+target-x86-v4/sapphirerapids/ESM1p5-oneAPI-pr40-2-sr-atm240-ocn156-map-by-numa-checking-verbose
Traceback (most recent call last):
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/bin/payu", line 7, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/cli.py", line 49, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/subcommands/clone_cmd.py", line 53, in runcmd
    clone(repository=repository,
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/branch.py", line 317, in clone
    repo = git_clone(repository, control_path, branch)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/git_utils.py", line 229, in git_clone
    repo = git.Repo.clone_from(repository, to_path=directory)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/repo/base.py", line 1543, in clone_from
    return cls._clone(
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/repo/base.py", line 1414, in _clone
    finalize_process(proc, stderr=stderr)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/util.py", line 512, in finalize_process
    proc.wait(**kwargs)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/cmd.py", line 419, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v -- /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156 /g/data/tm70/ms2335/performance_optimisation_runs/classic/access-esm1p6/intel@19.0.3.199+target-x86-v4/sapphirerapids/ESM1p5-oneAPI-pr40-2-sr-atm240-ocn156-map-by-numa-checking-verbose
  stderr: 'fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
'
[~ @gadi-login-04]$ which git && git --version
/g/data/vk83/prerelease/apps/conda_scripts/payu-dev-20250921T211851Z-d026d1e.d/bin/git
git version 2.51.0

Error message with system git

~ @gadi-login-04]$ git clone -v -- /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156 /g/data/tm70/ms2335/performance_optimisation_runs/classic/access-esm1p6/intel@19.0.3.199+target-x86-v4/sapphirerapids/ESM1p5-oneAPI-pr40-2-sr-atm240-ocn156-map-by-numa-checking-verbose
Cloning into '/g/data/tm70/ms2335/performance_optimisation_runs/classic/access-esm1p6/intel@19.0.3.199+target-x86-v4/sapphirerapids/ESM1p5-oneAPI-pr40-2-sr-atm240-ocn156-map-by-numa-checking-verbose'...
fatal: detected dubious ownership in repository at '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156/.git'
To add an exception for this directory, call:

        git config --global --add safe.directory /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156/.git
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
[~ @gadi-login-04]$ which git
/bin/git
[~ @gadi-login-04]$ which git && git --version
/bin/git
git version 2.43.5

No error when using the latest git module

[~ @gadi-login-04]$ git clone -v -- /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156 /g/data/tm70/ms2335/performance_optimisation_runs/classic/access-esm1p6/intel@19.0.3.199+target-x86-v4/sapphirerapids/ESM1p5-oneAPI-pr40-2-sr-atm240-ocn156-map-by-numa-checking-verbose
Cloning into '/g/data/tm70/ms2335/performance_optimisation_runs/classic/access-esm1p6/intel@19.0.3.199+target-x86-v4/sapphirerapids/ESM1p5-oneAPI-pr40-2-sr-atm240-ocn156-map-by-numa-checking-verbose'...
done.
[~ @gadi-login-04]$ which git && git --version
/apps/git/2.39.2/bin/git
git version 2.39.2

Note: I tried to fool payu by loading payu/dev first and then module loading git/2.39.2 but payu continued to use the prerelease git.

In the end, I got around the problem by recursively copying the repo directory first, and then running payu clone but I figured I would flag the issue here, in case someone else runs into the same problem into the future.

If there is some payu clone flag I should be using, then I would also like to know that.

[aidanheerdegen]

Looks like a git behaviour change, but I'm having trouble tracking down when/why.

$ git --version
git version 2.51.0

$ git clone -l -v /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156
fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
$

There isn't much payu can do about this if git refuses to clone a repo.

Some more examples:

$ module load payu/1.1.3
$ git clone -l -v /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156
Cloning into 'oneapi-sr-atm240-ocn156'...
fatal: failed to create link 'oneapi-sr-atm240-ocn156/.git/objects/pack/pack-02bba63f102085cbcc403b0869512ae5db63c9f5.pack': Operation not permitted
$ git --version
git version 2.44.0
$ git clone --no-hardlinks -l -v /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156
Cloning into 'oneapi-sr-atm240-ocn156'...
done.
$ module unload payu
$ module load payu/1.1.7
Loading payu/1.1.7
  Loading requirement: singularity
$ rm -rf oneapi-sr-atm240-ocn156/
$ git clone --no-hardlinks -l -v /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156
fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
$ git --version
git version 2.49.0

$ module unload payu
Unloading payu/1.1.7
  Unloading useless requirement: singularity
$ module load payu/1.1.6
Loading payu/1.1.6
  Loading requirement: singularity
$ git --version
git version 2.47.1
$ git clone --no-hardlinks -l -v /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156
fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
$ module unload payu
Unloading payu/1.1.6
  Unloading useless requirement: singularity
$ module load payu/1.1.5
$ git --version
git version 2.46.0
$ git clone --no-hardlinks -l -v /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156
Cloning into 'oneapi-sr-atm240-ocn156'...
fatal: detected dubious ownership in repository at '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156/.git'
To add an exception for this directory, call:

	git config --global --add safe.directory /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156/.git
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Summary

git version	args	Clone	Error
2.44.0	-l	❌	fatal: failed to create link 'oneapi-sr-atm240-ocn156/.git/objects/pack/pack-02bba63f102085cbcc403b0869512ae5db63c9f5.pack': Operation not permitted
2.46.0	-l --no-hardlinks	✅	-
2.46.0	-l	❌	fatal: detected dubious ownership
2.46.0	add safe dir	❌	fatal: failed to create link 'oneapi-sr-atm240-ocn156/.git/objects/pack/pack-02bba63f102085cbcc403b0869512ae5db63c9f5.pack': Operation not permitted
2.46.0	-l --no-hardlinks	✅	-
2.47.1	-l	❌	fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
2.47.1	-l --no-hardlinks	❌	fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not
2.49.0	-l	❌	fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
2.49.0	-l --no-hardlinks	❌	fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
2.51.0	-l	❌	fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist
2.51.0	-l --no-hardlinks	❌	fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist

[manodeep]

@aidanheerdegen Cloning from the released ESM1.6 configs works fine:

[~ @gadi-login-02]$ module use /g/data/vk83/prerelease/modules
[~ @gadi-login-02]$ module load payu/dev
Loading payu/dev-20250921T211851Z-d026d1e
  Loading requirement: singularity
[~ @gadi-login-02]$ payu clone --branch "amip" https://github.com/ACCESS-NRI/access-esm1.5-configs ESM1p5-released-amip-config-xxxx
Cloned repository from https://github.com/ACCESS-NRI/access-esm1.5-configs to directory: /home/593/ms2335/ESM1p5-released-amip-config-xxxx
Checked out branch: amip
laboratory path:  /scratch/tm70/ms2335/access-esm
binary path:  /scratch/tm70/ms2335/access-esm/bin
input path:  /scratch/tm70/ms2335/access-esm/input
work path:  /scratch/tm70/ms2335/access-esm/work
archive path:  /scratch/tm70/ms2335/access-esm/archive
Updated metadata. Experiment UUID: fe879d19-dbae-4566-bf3b-c53e5d7c3264
Added archive symlink to /scratch/tm70/ms2335/access-esm/archive/ESM1p5-released-amip-config-xxxx-amip-fe879d19
To change directory to control directory run:
  cd ESM1p5-released-amip-config-xxxx

[~ @gadi-login-02]$ rm -Rf ESM1p5-released-amip-config-xxxx
[~ @gadi-login-02]$ payu clone --branch "dev-preindustrial+concentrations" https://github.com/ACCESS-NRI/access-esm1.6-configs access-esm1.6-PI-config-Sep1-pr126-4-with-oneapi2025-2-0-ompi508-416-cores-mpi-yield-when-idle-xxxx
Cloned repository from https://github.com/ACCESS-NRI/access-esm1.6-configs to directory: /home/593/ms2335/access-esm1.6-PI-config-Sep1-pr126-4-with-oneapi2025-2-0-ompi508-416-cores-mpi-yield-when-idle-xxxx
Checked out branch: dev-preindustrial+concentrations
laboratory path:  /scratch/tm70/ms2335/access-esm
binary path:  /scratch/tm70/ms2335/access-esm/bin
input path:  /scratch/tm70/ms2335/access-esm/input
work path:  /scratch/tm70/ms2335/access-esm/work
archive path:  /scratch/tm70/ms2335/access-esm/archive
Updated metadata. Experiment UUID: c12a366b-03e3-4491-93d2-17fb8bf9d236
Added archive symlink to /scratch/tm70/ms2335/access-esm/archive/access-esm1.6-PI-config-Sep1-pr126-4-with-oneapi2025-2-0-ompi508-416-cores-mpi-yield-when-idle-xxxx-dev-preindustrial+concentrations-c12a366b
To change directory to control directory run:
  cd access-esm1.6-PI-config-Sep1-pr126-4-with-oneapi2025-2-0-ompi508-416-cores-mpi-yield-when-idle-xxxx

[~ @gadi-login-02]$ ml
Currently Loaded Modulefiles:
 1) pbs   2) singularity   3) payu/dev-20250921T211851Z-d026d1e(dev)  
[~ @gadi-login-02]$ which git && git --version
/g/data/vk83/prerelease/apps/conda_scripts/payu-dev-20250921T211851Z-d026d1e.d/bin/git
git version 2.51.0

[JhanSrbinovsky]

For me git clone works for all avail git modules but not for system git. pay clone doesn't work for any of them.

branch I was cloning from was dev-preindustrial+concentrations

[aidanheerdegen]

Cloning from the released ESM1.6 configs works fine

Thanks. Good to know that the problem seems confined to local clones.

[aidanheerdegen]

or me git clone works for all avail git modules but not for system git

See @manodeep's original post above. System git was complaining with local clones about safety issues, but that shouldn't affect payu clone from a repo. @manodeep has no problem using payu clone from a repo.

Thanks for the report @JhanSrbinovsky, but we would need more detail about the payu environment and the commands you've run to be able to understand the behaviour you're seeing.

[JhanSrbinovsky]

here is just a CLI history:

gadi:Feat> git -v

git version 2.51.0

gadi:Feat> payu --version

payu 1.1.7+47.gd026d1e

gadi:Feat> payu clone git@github.com:ACCESS-NRI/access-esm1.6-configs.git -B dev-preindustrial+concentrations feat-transp_tile-3

Traceback (most recent call last):
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/bin/payu", line 7, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/cli.py", line 49, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/subcommands/clone_cmd.py", line 53, in runcmd
    clone(repository=repository,
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/branch.py", line 317, in clone
    repo = git_clone(repository, control_path, branch)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/git_utils.py", line 225, in git_clone
    repo = git.Repo.clone_from(repository,
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/repo/base.py", line 1543, in clone_from
    return cls._clone(
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/repo/base.py", line 1414, in _clone
    finalize_process(proc, stderr=stderr)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/util.py", line 512, in finalize_process
    proc.wait(**kwargs)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/cmd.py", line 419, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v --branch=dev-preindustrial+concentrations -- git@github.com:ACCESS-NRI/access-esm1.6-configs.git /g/data/p66/jxs599/ESM16/PAYU/Feat/feat-transp_tile-3
  stderr: 'Cloning into '/g/data/p66/jxs599/ESM16/PAYU/Feat/feat-transp_tile-3'...
OpenSSL version mismatch. Built against 30500000, you have 3050003f
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
'

gadi:Feat> git clone git@github.com:ACCESS-NRI/access-esm1.6-configs.git -b dev-preindustrial+concentrations feat-transp_tile-3

Cloning into 'feat-transp_tile-3'...
OpenSSL version mismatch. Built against 30500000, you have 3050003f
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

gadi:Feat> module load git/2.39.2

gadi:Feat> payu clone git@github.com:ACCESS-NRI/access-esm1.6-configs.git -B dev-preindustrial+concentrations feat-transp_tile-3

Traceback (most recent call last):
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/bin/payu", line 7, in <module>
    sys.exit(parse())
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/cli.py", line 49, in parse
    run_cmd(**args)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/subcommands/clone_cmd.py", line 53, in runcmd
    clone(repository=repository,
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/branch.py", line 317, in clone
    repo = git_clone(repository, control_path, branch)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/payu/git_utils.py", line 225, in git_clone
    repo = git.Repo.clone_from(repository,
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/repo/base.py", line 1543, in clone_from
    return cls._clone(
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/repo/base.py", line 1414, in _clone
    finalize_process(proc, stderr=stderr)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/util.py", line 512, in finalize_process
    proc.wait(**kwargs)
  File "/g/data/vk83/prerelease/apps/base_conda/envs/payu-dev-20250921T211851Z-d026d1e/lib/python3.10/site-packages/git/cmd.py", line 419, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v --branch=dev-preindustrial+concentrations -- git@github.com:ACCESS-NRI/access-esm1.6-configs.git /g/data/p66/jxs599/ESM16/PAYU/Feat/feat-transp_tile-3
  stderr: 'Cloning into '/g/data/p66/jxs599/ESM16/PAYU/Feat/feat-transp_tile-3'...
OpenSSL version mismatch. Built against 30500000, you have 3050003f
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
'

gadi:Feat> git clone git@github.com:ACCESS-NRI/access-esm1.6-configs.git -b dev-preindustrial+concentrations feat-transp_tile-3

Cloning into 'feat-transp_tile-3'...
remote: Enumerating objects: 3876, done.
remote: Counting objects: 100% (647/647), done.
remote: Compressing objects: 100% (98/98), done.
remote: Total 3876 (delta 603), reused 550 (delta 549), pack-reused 3229 (from 3)
Receiving objects: 100% (3876/3876), 1.67 MiB | 2.04 MiB/s, done.
Resolving deltas: 100% (2988/2988), done.

[aidanheerdegen]

@JhanSrbinovsky your error is related to an OpenSSL clash:

OpenSSL version mismatch. Built against 30500000, you have 3050003f

I assume you have a setting that is loading an OpenSSL version, as I've not seen others report this error.

[jo-basevi]

I was able to reproduce Jhan's error. There was a new openssl version that got released on 16th (OpenSSL 3.5.3 16 Sep 2025 (Library: OpenSSL 3.5.3 16 Sep 2025). Older versions of payu/dev pre 16 September seem to work ok. So maybe we need to also downgrade openssl in payu/dev?

[jo-basevi]

The latest payu/dev should now work with git cloning with ssh again (e.g. payu clone git@github.com:ACCESS-NRI/access-esm1.6-configs.git -B dev-preindustrial+concentrations feat-transp_tile-5 or git clone git@github.com:ACCESS-NRI/access-esm1.6-configs.git).

We have also figured out what is causing the fatal: repository '/home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156' does not exist errors with later git versions (from payu/1.1.6 onwards). The issue is that singularity container by default only mounts the user's home directory. For example, because my home directory is /home/189/..., inside the container I am only able to access files in /home/189/. Both git and payu commands run inside of the container so they wouldn't be able to find /home/565/sw6175/esm1.5/oneapi/oneapi-sr-atm240-ocn156. We could test mounting the entire home directory in the singularity container (see comment), but in the meantime it would be worth using system git or git modules to clone directories not in the same home directory to /g/data or /scratch first.

[aidanheerdegen]

For the moment we're going to have to park this as a known issue that requires using git outside of the payu conda environment on gadi when cloning from a home directory on a different home filesystem.