Invalid cross-device link

[adi-]

In recent version there is a problem generating requirements.txt from pip-compile. Problem stack:

File "/opt/venvs/test/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/opt/venvs/test/lib/python3.12/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/test/lib/python3.12/site-packages/click/core.py", line 1373, in main
    ctx.exit()
  File "/opt/venvs/test/lib/python3.12/site-packages/click/core.py", line 717, in exit
    self.close()
  File "/opt/venvs/test/lib/python3.12/site-packages/click/core.py", line 617, in close
    self._exit_stack.close()
  File "/usr/lib/python3.12/contextlib.py", line 618, in close
    self.__exit__(None, None, None)
  File "/usr/lib/python3.12/contextlib.py", line 610, in __exit__
    raise exc_details[1]
  File "/usr/lib/python3.12/contextlib.py", line 595, in __exit__
    if cb(*exc_details):
       ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 478, in _exit_wrapper
    callback(*args, **kwds)
  File "/opt/venvs/test/lib/python3.12/site-packages/click/utils.py", line 179, in close_intelligently
    self.close()
  File "/opt/venvs/test/lib/python3.12/site-packages/click/utils.py", line 172, in close
    self._f.close()
  File "/opt/venvs/test/lib/python3.12/site-packages/click/_compat.py", line 467, in close
    os.replace(self._tmp_filename, self._real_filename)
OSError: [Errno 18] Invalid cross-device link: '.__atomic-write88e304fc' -> '/srv/test/requirements.txt'

Requirements are generated in Vagrant.

[sirosen]

There isn't enough information here for us to investigate or troubleshoot.

Can you confirm that this works with an older pip-tools version (which may require an older pip version)?

Vagrant can mean a VM or a container. I think it matters what you're using underneath it. Can you share what the setup here actually is?
Are you running the command within a container but pointing it at files outside of the container? ¹

Footnotes

1. I suspect so, but really do need more info. ↩

[adi-]

Vagrant as an VM. The original requirements.in file is shared to VM by Vagrant’s default synced_folder. I assume that pip-compile creates a temporary file on the local disk and, at the end, tries to move (rename) it to requirements.txt. This is where the problem occurs.

I guess the root of the problem: https://stackoverflow.com/questions/42392600/oserror-errno-18-invalid-cross-device-link

[webknjaz]

I assume that pip-compile

There's no piptools in the traceback. The disk operation is coming from click which would be out of our control: https://github.com/pallets/click/blob/6965a6a/src/click/_compat.py#L467.

Without a repro, I'd be inclined to just declare this a click behavior: https://gist.github.com/Potherca/790fc75c5b9ff64efc398bfa6a093f73

[sirosen]

I think the absence of piptools in the trace is an artifact of click's implementation of "atomic write". It does, as @adi- correctly guessed, create a file and then move it into place.
I agree that we need a reproducer. I'd like to understand what's happening in case it impacts other containerized usage.

The original requirements.in file is shared to VM by Vagrant’s default synced_folder.

I'm not familiar with Vagrant -- haven't used it in many years -- does this mean that /srv/test/ from your traceback is a mount on the container?

I would expect that to work fine. From the click docs:

Files can also be opened atomically in which case all writes go into a separate file in the same folder and upon completion the file will be moved over to the original location. This is useful if a file regularly read by other users is modified.

So this should be writing to /srv/test/.__atomic-write$hash and then moving it to /srv/test/requirements.txt. I'm surprised that it's failing for you.

[webknjaz]

So this should be writing to /srv/test/.__atomic-write$hash and then moving it to /srv/test/requirements.txt. I'm surprised that it's failing for you.

I'm wondering if the host system is macOS. IIRC vagrant does sinister things when mounting file systems from host through Virtualbox into VMs... Especially because the file systems are different in operating systems (HFS+ vs. ext4 and so on). I think they might be using NFS for that.

[adi-]

It wasn't osx, but Linux (Ubuntu)

[sirosen]

@adi- can you give us any more information about the filesystem situation in which you encounter this? Is /srv/test/ a normal directory inside of your Vagrant VM? Or is it a mounted directory from your host filesystem? Is the host also Linux?

Right now, for lack of information, this is in the category of "could not reproduce" (CNR) because we just don't know how to recreate the issue. Without some info about how to make this happen, we can't do very much. This was already on my radar as "if we can't find out more from the user, this has to be closed as CNR."