create datalad-deatverse sibling on MAC does not work

[RalfKunkel]

After installing the datalad-dataverse interface on my new Mac (Apple M3, MacOS 15.0.1), using pip install datalad-dataverse,I'm running into an error, when trying to create a datalad sibling on Dataverse. I used the command:
datalad add-sibling-dataverse https://data-beta.fz-juelich.de doi:XXXXXXX --mode filetree and received the error message:

CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false annex initremote dataverse-storage type=external externaltype=dataverse url=https://data-beta.fz-juelich.de doi=doi:XXXXXX exporttree=yes encryption=none -c annex.dotfiles=true' failed with exitcode 1 under YYYYYY initremote dataverse-storage

failed
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/bin/git-annex-remote-dataverse", line 5, in <module>
    from datalad_dataverse.remote import main
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datalad_dataverse/remote.py", line 13, in <module>
    from .baseremote import DataverseRemote as BaseDataverseRemote
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datalad_dataverse/baseremote.py", line 23, in <module>
    from .dataset import OnlineDataverseDataset
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/datalad_dataverse/dataset.py", line 14, in <module>
    from pyDataverse.models import Datafile
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyDataverse/models.py", line 8, in <module>
    from pyDataverse.utils import validate_data
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pyDataverse/utils.py", line 8, in <module>
    from jsonschema import validate
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/jsonschema/__init__.py", line 13, in <module>
    from jsonschema._format import FormatChecker
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/jsonschema/_format.py", line 11, in <module>
    from jsonschema.exceptions import FormatError
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/jsonschema/exceptions.py", line 15, in <module>
    from referencing.exceptions import Unresolvable as _Unresolvable
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/referencing/__init__.py", line 5, in <module>
    from referencing._core import Anchor, Registry, Resource, Specification
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/referencing/_core.py", line 9, in <module>
    from rpds import HashTrieMap, HashTrieSet, List
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/rpds/__init__.py", line 1, in <module>
    from .rpds import *
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so, 0x0002): tried: '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so' (no such file), '/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/rpds/rpds.cpython-311-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
  external special remote protocol error, unexpectedly received "<EOF>" (unable to parse command)
git-annex: unable to use special remote due to protocol error
initremote: 1 failed

Any idea, what can be done to bring this to work? Did I made a mistake?

[welcome]

Hi! 👋 We are happy that you opened your first issue here! 😄 If you haven't done so already, please make sure you check out our Code of Conduct.

[mslw]

Uh-oh. The traceback suggests that the error comes from rpds-py, which is an indirect dependency of pyDataverse - here's part of pipdeptree output:

❱ pipdeptree
datalad_dataverse==1.0.2
...
└── pyDataverse [required: >=0.3.4, installed: 0.3.4]
    ...
    └── jsonschema [required: >=4.21.1,<5.0.0, installed: 4.23.0]
        ├── attrs [required: >=22.2.0, installed: 24.2.0]
        ├── jsonschema-specifications [required: >=2023.03.6, installed: 2024.10.1]
        │   └── referencing [required: >=0.31.0, installed: 0.35.1]
        │       ├── attrs [required: >=22.2.0, installed: 24.2.0]
        │       └── rpds-py [required: >=0.7.0, installed: 0.20.0]
        ├── referencing [required: >=0.28.4, installed: 0.35.1]
        │   ├── attrs [required: >=22.2.0, installed: 24.2.0]
        │   └── rpds-py [required: >=0.7.0, installed: 0.20.0]
        └── rpds-py [required: >=0.7.1, installed: 0.20.0]

The ImportError mentions architecture: (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'). You say you are on an M3 Mac. I don't use that, but AFAIK it is an arm-based system which has several ways of running x86-64 programs, depending on your configuration.

Looking at the rpds-py PyPI files page, it seems to have both ARM64 and x86-64 MacOS wheels. People in this StackOverflow thread (see also this Apple StackExchange thread) seem to have had success in similar situations by simply reinstalling the offending package (pip uninstall followed by pip install --no-cache). I would try that first.

As a side note, it seems you are installing the Python packages system-wide. For playing with the dependencies, I would consider using virtual environments (but it's not a necessity).

[RalfKunkel]

Thanks for your reply. I reinstalled both rdps-py and (after testing) also the pyDataverse as a whole, but still got the same error. Obviously, the libraries with arm64 architecture have been installed, but something seems to need amd64 specifically.

Any idea what I can do about this?

[mslw]

I'm afraid I don't have a good idea, just a few guesses and questions; I don't know what, other than python itself, could need amd64 specifically.

• What happens if you import rpds in python interpreter?
• Are you sure your python runs natively, and not with amd64 compatibility? What's the output of platform.machine() / platform.processor / platform.platform() in python iterpreter (must import platform first)?
• When reinstalling packages, did you use pip install --no-cache? From what I read, if the package with the wrong architecture is still cached by pip, you'll get it back if you run pip without the flag.
• Are you using Python which came with your system, or one which you installed? If so, how? Any virtual environments?

Other than that, it would be quick to try from scratch in a fresh virtual environment, and try there just to be sure... something like:

python3 -m venv ~/.venvs/datalad-dataverse
source ~/.venvs/datalad-dataverse/bin/activate
pip install --no-cache datalad-dataverse