Open
Description
We get doc build errors like this fairly frequently nowadays:
Traceback
Unexpected failing examples (1):
../examples/preprocessing/otp.py failed leaving traceback:
Traceback (most recent call last):
File "/home/circleci/project/examples/preprocessing/otp.py", line 81, in <module>
bias = compute_bias(raw)
^^^^^^^^^^^^^^^^^
...
File "/home/circleci/project/mne/parallel.py", line 95, in parallel_func
cache_dir = get_config("MNE_CACHE_DIR", None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/project/mne/utils/config.py", line 309, in get_config
config = _load_config(config_path)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/circleci/project/mne/utils/config.py", line 234, in _load_config
warn(msg)
File "/home/circleci/project/mne/utils/_logging.py", line 416, in warn
warnings.warn_explicit(
RuntimeWarning: The MNE-Python config file (/home/circleci/.mne/mne-python.json) is not a valid JSON file and might be corrupted
This started happening once we enabled parallel building in docs. I suspect it could be because the JSON file is being written by one example as it's trying to be read by another (or maybe more likely, written by one Python thread while another tries to read it). There are a few potential solutions:
- Ignore the corrupt-JSON warning in doc builds. A bit ugly and makes it so we'll miss if something more serious goes wrong with our config I/O. But I'll probably open a PR to do this for now just so that the build stops failing so often.
- Stop building docs in parallel. Has the severe disadvantage of doubling our build time.
- Use some file-lock mechanism to prevent reading while writing (or writing while writing).
It seems like (3) is probably the best solution. We could use filelock
(it's used for example by virtualenv
and maintained by tox-dev
so should be well supported; license is public domain) but it would require adding another hard dependency. We could also use msvcrt and fnctl but we'll be reinventing the wheel. Another option would be to vendor some or all of their code.