Asyncio and multiprocess damage the cache

[pombredanne]

This error is showing up in test runs https://dev.azure.com/nexB/python-inspector/_build/results?buildId=15849&view=logs&jobId=127eb2c5-1c4a-54f5-86f9-4c9024fa5f36&j=127eb2c5-1c4a-54f5-86f9-4c9024fa5f36&t=eb44e1a8-f951-5ab7-e23d-607c85fad059

This is because the cache is being written to concurrently and we have not designed that cache to be thread safe.
Having a global cache with settings in #214 or #192 is also compounding the issue.

self = <zipfile.ZipFile [closed]>

    def _RealGetContents(self):
        """Read in the table of contents for the ZIP file."""
        fp = self.fp
        try:
            endrec = _EndRecData(fp)
        except OSError:
            raise BadZipFile("File is not a zip file")
        if not endrec:
>           raise BadZipFile("File is not a zip file")
E           zipfile.BadZipFile: File is not a zip file

The solution is likely to use process-safe file locks like used in ScanCode when writing a wheels to avoid any corruption of cached files. Plus also some cache checks, pip style.

[pombredanne]

@netomi FYI.

[sschuberth]

Is someone actively working on this? @netomi, would have time to fix this up so we can finally get a new release?

[kerstin-bosch]

@pombredanne, @netomi: That would be really great and would also help us a lot if there is a new release soon. Is it possible to fix this soon? Thanks!

[pombredanne]

@kerstin-bosch the fix is likely to adopt the same interprocess locking approach as in scancode at:

• https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/scancode/lockfile.py
• https://github.com/aboutcode-org/scancode-toolkit/blob/c2d52ff855094d9c756a8e252c5799037692bf35/src/licensedcode/cache.py#L135

... I am technically off this week :] , but I will try to look into this by tomorrow.

[netomi]

is there a testcase illustrating the cache poisoning?
I did quite some number of tests for the async work and never seen anything like that. I am not aware of multiprocessing work that has been added, can you point me to that?

If multiple processes access the same cache, thing like that happen not sure how this is related to async though.