Skip to content

BUG: Parallel build JSON file issues #13139

Open
@larsoner

Description

@larsoner

We get doc build errors like this fairly frequently nowadays:

https://app.circleci.com/pipelines/github/mne-tools/mne-python/26824/workflows/2f0db808-f2fd-4931-841c-ac7c7d98da4f/jobs/71898

Traceback
    Unexpected failing examples (1):
    
        ../examples/preprocessing/otp.py failed leaving traceback:
    
        Traceback (most recent call last):
          File "/home/circleci/project/examples/preprocessing/otp.py", line 81, in <module>
            bias = compute_bias(raw)
                   ^^^^^^^^^^^^^^^^^
...
          File "/home/circleci/project/mne/parallel.py", line 95, in parallel_func
            cache_dir = get_config("MNE_CACHE_DIR", None)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/circleci/project/mne/utils/config.py", line 309, in get_config
            config = _load_config(config_path)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/circleci/project/mne/utils/config.py", line 234, in _load_config
            warn(msg)
          File "/home/circleci/project/mne/utils/_logging.py", line 416, in warn
            warnings.warn_explicit(
        RuntimeWarning: The MNE-Python config file (/home/circleci/.mne/mne-python.json) is not a valid JSON file and might be corrupted

This started happening once we enabled parallel building in docs. I suspect it could be because the JSON file is being written by one example as it's trying to be read by another (or maybe more likely, written by one Python thread while another tries to read it). There are a few potential solutions:

  1. Ignore the corrupt-JSON warning in doc builds. A bit ugly and makes it so we'll miss if something more serious goes wrong with our config I/O. But I'll probably open a PR to do this for now just so that the build stops failing so often.
  2. Stop building docs in parallel. Has the severe disadvantage of doubling our build time.
  3. Use some file-lock mechanism to prevent reading while writing (or writing while writing).

It seems like (3) is probably the best solution. We could use filelock (it's used for example by virtualenv and maintained by tox-dev so should be well supported; license is public domain) but it would require adding another hard dependency. We could also use msvcrt and fnctl but we'll be reinventing the wheel. Another option would be to vendor some or all of their code.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions