Writing over xrootd to some location that requires an x509 proxy fails with: `[ERROR] Local error: read-only file system`

[ikrommyd]

To reproduce from your machine (use your own cernbox eos or any other eos location that requires an x509 proxy to write to) using https://hackmd.io/@andrzejnovak/SJxscCZvD as a guide to get x509 working locally:

import uproot
uproot.recreate("root://eosuser.cern.ch//eos/user/i/ikrommyd/dummy.root")

fails with OSError: Failed to submit <bound method File.close of <XRootD.client.file.File object at 0x109b50ec0>> request: [ERROR] Local error: read-only file system
At the same time, writing parquet is fine:

import awkward as ak
ak.to_parquet(ak.Array([1, 2, 3]), "root://eosuser.cern.ch//eos/user/i/ikrommyd/dummy.parquet")

cc @maxgalli @nsmith- (it may be an fsspec-xrootd issue directly)

[ikrommyd]

Opening the file with plain xrootd and then doing uproot.recreate fixes it.

In [1]: import uproot

In [2]: path = "root://eoscms.cern.ch//store/group/phys_egamma/tnpTuples/ikrommyd/dummy.root"

In [3]: uproot.recreate(path)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[3], line 1
----> 1 uproot.recreate(path)

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/uproot/writing/writable.py:110, in recreate(file_path, **options)
    106 file_path = uproot._util.regularize_path(file_path)
    107 storage_options = {
    108     key: value for key, value in options.items() if key not in recreate.defaults
    109 }
--> 110 sink = uproot.sink.file.FileSink(file_path, **storage_options)
    111 compression = options.pop("compression", create.defaults["compression"])
    113 initial_directory_bytes = options.pop(
    114     "initial_directory_bytes", create.defaults["initial_directory_bytes"]
    115 )

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/uproot/sink/file.py:53, in FileSink.__init__(self, urlpath_or_file_like, **storage_options)
     51 else:
     52     if not self._file_exists(urlpath_or_file_like, **storage_options):
---> 53         self._truncate_file(urlpath_or_file_like, **storage_options)
     55     self._open_file = fsspec.open(
     56         urlpath_or_file_like, mode="r+b", **storage_options
     57     )

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/uproot/sink/file.py:81, in FileSink._truncate_file(cls, urlpath, **storage_options)
     79 parent_directory = fs.sep.join(local_path.split(fs.sep)[:-1])
     80 fs.mkdirs(parent_directory, exist_ok=True)
---> 81 fs.touch(local_path, truncate=True)

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec/asyn.py:118, in sync_wrapper.<locals>.wrapper(*args, **kwargs)
    115 @functools.wraps(func)
    116 def wrapper(*args, **kwargs):
    117     self = obj or args[0]
--> 118     return sync(self.loop, func, *args, **kwargs)

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec/asyn.py:103, in sync(loop, func, timeout, *args, **kwargs)
    101     raise FSTimeoutError from return_result
    102 elif isinstance(return_result, BaseException):
--> 103     raise return_result
    104 else:
    105     return return_result

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec/asyn.py:56, in _runner(event, coro, result, timeout)
     54     coro = asyncio.wait_for(coro, timeout=timeout)
     55 try:
---> 56     result[0] = await coro
     57 except Exception as ex:
     58     result[0] = ex

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec_xrootd/xrootd.py:378, in XRootDFileSystem._touch(self, path, truncate, **kwargs)
    376 f = client.File()
    377 status, _ = await _async_wrap(f.open)(path, OpenFlags.DELETE)
--> 378 await _async_wrap(f.close)()
    379 if not status.ok:
    380     raise OSError(f"File not touched properly: {status.message}")

File ~/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec_xrootd/xrootd.py:66, in _async_wrap.<locals>.wrapped(*args, **kwargs)
     64 submit_status: XRootDStatus = func(*args, **kwargs, callback=callback)
     65 if not submit_status.ok:
---> 66     raise OSError(
     67         f"Failed to submit {func!r} request: {submit_status.message.strip()}"
     68     )
     69 return await future

OSError: Failed to submit <bound method File.close of <XRootD.client.file.File object at 0x107bccb30>> request: [ERROR] Local error: read-only file system

In [4]: from XRootD.client import File, flags

In [5]: f = File()

In [6]: status, _ = f.open(path, flags.OpenFlags.DELETE)

In [7]: status
Out[7]: <status: 0, code: 0, errno: 0, message: '[SUCCESS] ', shellcode: 0, error: False, fatal: False, ok: True>

In [8]: f.close()
Out[8]:
(<status: 0, code: 0, errno: 0, message: '[SUCCESS] ', shellcode: 0, error: False, fatal: False, ok: True>,
 None)

In [9]: uproot.recreate(path)
Out[9]: <WritableDirectory '/' at 0x000112a6ee70>

[ikrommyd]

Manually going to the target location and deleting dummy.root and trying uproot.recreate again brings back the error.

[maxgalli]

I'm not able to reproduce this when running on a Tier3 that requires an x509 proxy

$ python
Python 3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:10:22) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import uproot
>>> path = "root://eoshome-g.cern.ch//eos/user/g/gallim/dummy3.root"
>>> uproot.recreate(path)
<WritableDirectory '/' at 0x7fc66a424980>
>>>

Packages:

$ python
Python 3.13.2 | packaged by conda-forge | (main, Feb 17 2025, 14:10:22) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import uproot
>>> uproot.__version__
'5.6.0'
>>> import fsspec_xrootd
>>> fsspec_xrootd.__version__
'0.5.0'

$ xrootd -v
v5.7.3

[ikrommyd]

I reproduced this one lxplus with mamba create -n ak-uproot python=3.12 uproot xrootd fsspec-xrootd and mamba activate ak-uproot.

(ak-uproot) ➜  ~ python 
Python 3.12.9 | packaged by conda-forge | (main, Mar  4 2025, 22:48:41) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import uproot
>>> uproot.recreate("root://eosuser.cern.ch//eos/user/i/ikrommyd/dummy.root")
<WritableDirectory '/' at 0x7f7adf836420>
>>> exit()
(ak-uproot) ➜  ~ python                                                                                                                                                                        
Python 3.12.9 | packaged by conda-forge | (main, Mar  4 2025, 22:48:41) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import uproot
>>> uproot.recreate("root://cmseos.fnal.gov//store/user/ikrommyd/test/dummy.root")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/uproot/writing/writable.py", line 110, in recreate
    sink = uproot.sink.file.FileSink(file_path, **storage_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/uproot/sink/file.py", line 53, in __init__
    self._truncate_file(urlpath_or_file_like, **storage_options)
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/uproot/sink/file.py", line 81, in _truncate_file
    fs.touch(local_path, truncate=True)
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec/asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec/asyn.py", line 103, in sync
    raise return_result
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec/asyn.py", line 56, in _runner
    result[0] = await coro
                ^^^^^^^^^^
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec_xrootd/xrootd.py", line 378, in _touch
    await _async_wrap(f.close)()
  File "/afs/cern.ch/work/i/ikrommyd/miniforge3/envs/ak-uproot/lib/python3.12/site-packages/fsspec_xrootd/xrootd.py", line 66, in wrapped
    raise OSError(
OSError: Failed to submit <bound method File.close of <XRootD.client.file.File object at 0x7f5377fd6c00>> request: [ERROR] Local error: permission denied
>>> exit()
(ak-uproot) ➜  ~ voms-proxy-info --all                         
subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ikrommyd/CN=863399/CN=Iason Krommydas/CN=1762877112
issuer    : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ikrommyd/CN=863399/CN=Iason Krommydas
identity  : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ikrommyd/CN=863399/CN=Iason Krommydas
type      : RFC3820 compliant impersonation proxy
strength  : 2048
path      : /tmp/x509up_u154691
timeleft  : 191:57:58
key usage : Digital Signature, Key Encipherment
=== VO cms extension information ===
VO        : cms
subject   : /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ikrommyd/CN=863399/CN=Iason Krommydas
issuer    : /DC=ch/DC=cern/OU=computers/CN=cms-auth.cern.ch
attribute : /cms/Role=NULL/Capability=NULL
attribute : /cms/uscms/Role=NULL/Capability=NULL
timeleft  : 191:57:57
uri       : voms-cms-auth.cern.ch:15000

The first one succeeds because I'm writing to cernbox. The second one fails because I'm writing to FNAL eos which needs a certificate.

After mamba install pyarrow, parquet is totally fine.

(ak-uproot) ➜  ~ python
Python 3.12.9 | packaged by conda-forge | (main, Mar  4 2025, 22:48:41) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import awkward as ak
>>> ak.to_parquet(ak.Array([1,2,3]), "root://cmseos.fnal.gov//store/user/ikrommyd/test/dummy2.parquet")
<pyarrow._parquet.FileMetaData object at 0x7f860409ca40>
  created_by: parquet-cpp-arrow version 19.0.1
  num_columns: 1
  num_rows: 3
  num_row_groups: 1
  format_version: 2.6
  serialized_size: 0

[ikrommyd]

The pattern appears to be (at least to me) that if the file is there, it knows how to open it (similarly how uproot.open can open an existing file). If the file isn't there, uproot.recreate does not figure out it actually has permissions to create the file. Plain xrootd and root do.

[maxgalli]

I tried it on lxplus using the global redirector cms-xrd-global.cern.ch in order to get into a similar situation. The reproducer above gives a different error OSError: XRootD error: [ERROR] Server responded with an error: [3011] No servers have the file, while writing to parquet and plain XRootD give the following

>>> from XRootD.client import File, flags
>>> f = File()
>>> status, _ = f.open("root://cms-xrd-global.cern.ch//eos/user/g/gallim/dummy4.root"", flags.OpenFlags.DELETE)
  File "<python-input-6>", line 1
    status, _ = f.open("root://cms-xrd-global.cern.ch//eos/user/g/gallim/dummy4.root"", flags.OpenFlags.DELETE)
                                                                                     ^
SyntaxError: unterminated string literal (detected at line 1)
>>> status, _ = f.open("root://cms-xrd-global.cern.ch//eos/user/g/gallim/dummy4.root", flags.OpenFlags.DELETE)
>>> f.close(
... )
(<status: 1, code: 400, errno: 3011, message: '[ERROR] Server responded with an error: [3011] Unable to access file; eligible servers shunned.\n', shellcode: 54, error: True, fatal: False, ok: False>, None)

I'm not sure about their meaning, but they seem to be more related to the redirector itself (?).
Writing to root://eoshome-g.cern.ch//eos/user/g/gallim/dummy.root works correctly, as you said.
So at the moment I'm running out of ideas of how to reproduce the [ERROR] Local error: read-only file system error.

[ikrommyd]

I think this makes sense though. You're trying to access a file that is not there as if it was actual data (with cms-xrd-global.cern.ch). So the redirector is trying to find a site that contains the file /eos/user/g/gallim/dummy4.root and doesn't find any.

[ikrommyd]

By the way, it's not just me seeing this. I was testing this because a user was seeing it through uproot.dask_write (which does uproot.recreate internally) yesterday and reported it cause they couldn't use it and they were getting the same error. Can you try writing from lxplus to another site you have access to (your T3 maybe?)?. I only saw this on lxplus when I was trying to write to my FNAL eos area which you shouldn't have access to.

[maxgalli]

I was able to reproduce it after following the procedure suggested by @ikrommyd.
The problem is that this line in fsspec-xrootd uses a path that has been stripped of e.g. root://eoshome-g.cern.ch. If provided with the complete path, the example works as expected.
The question now is: where is it better to implement this change? Should we try to access the full path in fsspec-xrootd or implement a try/except here in uproot as follows?

        try:
            fs.touch(local_path, truncate=True)
        except OSError:
            fs.touch(urlpath, truncate=True)

[ianna]

I was able to reproduce it after following the procedure suggested by @ikrommyd. The problem is that this line in fsspec-xrootd uses a path that has been stripped of e.g. root://eoshome-g.cern.ch. If provided with the complete path, the example works as expected. The question now is: where is it better to implement this change? Should we try to access the full path in fsspec-xrootd or implement a try/except here in uproot as follows?

        try:
            fs.touch(local_path, truncate=True)
        except OSError:
            fs.touch(urlpath, truncate=True)

I think, it would be nice not to pin the version of fsspec, so I'd add some protection in uproot for now.

[ikrommyd]

Fixed by scikit-hep/fsspec-xrootd#80. It may be good to add such a try-except block in uproot as well if you don't wanna pin fsspec-xrootd as soon as a release is out. I guess you could also just not pin it and let it fail and if the users want this functionality, they can just update their fsspec-xrootd version. All seem "logical" to me.

(ak-uproot) ➜  ~ pip install git+https://github.com/scikit-hep/fsspec-xrootd.git                                            
Collecting git+https://github.com/scikit-hep/fsspec-xrootd.git
  Cloning https://github.com/scikit-hep/fsspec-xrootd.git to /private/var/folders/2c/94kt72fs0_q_9nzr2y5jphb00000gn/T/pip-req-build-gs0ebioa
  Running command git clone --filter=blob:none --quiet https://github.com/scikit-hep/fsspec-xrootd.git /private/var/folders/2c/94kt72fs0_q_9nzr2y5jphb00000gn/T/pip-req-build-gs0ebioa
  Resolved https://github.com/scikit-hep/fsspec-xrootd.git to commit a2ea12378aae1b37b585ba576f5d7ec5cca5b367
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: fsspec in ./miniforge3/envs/ak-uproot/lib/python3.12/site-packages (from fsspec_xrootd==0.5.1.dev1+ga2ea123) (2025.3.0)
Building wheels for collected packages: fsspec_xrootd
  Building wheel for fsspec_xrootd (pyproject.toml) ... done
  Created wheel for fsspec_xrootd: filename=fsspec_xrootd-0.5.1.dev1+ga2ea123-py3-none-any.whl size=14203 sha256=f486101a35040bdc1a8e0fc8a14b81624e5256bb4b985f7755b04959b1be2bcd
  Stored in directory: /private/var/folders/2c/94kt72fs0_q_9nzr2y5jphb00000gn/T/pip-ephem-wheel-cache-20got99n/wheels/bd/35/85/580d13611f1411c201ab812549fe821fbb5ab001fefa4a3d9e
Successfully built fsspec_xrootd
Installing collected packages: fsspec_xrootd
  Attempting uninstall: fsspec_xrootd
    Found existing installation: fsspec_xrootd 0.5.0
    Uninstalling fsspec_xrootd-0.5.0:
      Successfully uninstalled fsspec_xrootd-0.5.0
Successfully installed fsspec_xrootd-0.5.1.dev1+ga2ea123
(ak-uproot) ➜  ~ ipython 
Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 15:57:01) [Clang 17.0.6 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.29.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: exit()
(ak-uproot) ➜  ~ proxy                                                          
Enter GRID pass phrase:
Your identity: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=ikrommyd/CN=863399/CN=Iason Krommydas
Cannot open fileFilename=/Users/iason/.rnd
Function: 
Creating temporary proxy ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Done
Contacting  voms-cms-auth.cern.ch:443 [/DC=ch/DC=cern/OU=computers/CN=cms-auth.cern.ch] "cms" Done
Creating proxy ........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... Done

Your proxy is valid until Thu Mar 20 10:46:22 2025
(ak-uproot) ➜  ~ ipython 
Python 3.12.7 | packaged by conda-forge | (main, Oct  4 2024, 15:57:01) [Clang 17.0.6 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.29.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import uproot

In [2]: uproot.recreate("root://eosuser.cern.ch//eos/user/i/ikrommyd/dummy123.root")
Out[2]: <WritableDirectory '/' at 0x00010395aba0>

[nsmith-]

FYI https://github.com/scikit-hep/fsspec-xrootd/releases/tag/v0.5.1