trackpy.batch only works with processes = 1

[ggomes-altos]

Python 3.12.4
trackpy 0.6.4

The following runs without error:
f = tp.batch(frames, 11, invert=False, minmass=500, processes=1)

whereas processes = 2 gives a long error message ultimately ending in TiffFileError: corrupted tag list at offset 254920375

and processes = 'auto' gives a warning ( ValueError: need at most 63 handles, got a sequence of length 66 ) which disappears and is replaced with the Frame number 1 and number of features, but the computation never ends.

Happy to include full error messages if necessary.

[nkeim]

Thanks for this report. Do you know what version of pims you have, and which package is installed for reading tiffs? (usually libtiff or tifffile) On Aug 8, 2024, at 7:01 PM, ggomes-altos ***@***.***> wrote: Python 3.12.4 trackpy 0.6.4 The following runs without error: f = tp.batch(frames, 11, invert=False, minmass=500, processes=1) whereas processes = 2 gives a long error message ultimately ending in TiffFileError: corrupted tag list at offset 254920375 and processes = 'auto' gives a warning ( ValueError: need at most 63 handles, got a sequence of length 66 ) which disappears and is replaced with the Frame number 1 and number of features, but the computation never ends. Happy to include full error messages if necessary. — Reply to this email directly, view it on GitHub<#771>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAWY23NWV4XON3CF3IULEZTZQP2DRAVCNFSM6AAAAABMHMNIRKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2TMNZYGE4DIOA>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[ggomes-altos]

Hi sorry for the delayed response.

I have pims 0.7
libtiff 4.6.0
tifffile 2023.4.12

I think tiffffile is doing the tiff reading since in the lowest level of the error message it seems to be using the tifffile package:

---

ValueError Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_18640\387384096.py in ?()
----> 1 f = tp.batch(frames, 11, invert=False, minmass=500, processes=2);

A_path\Lib\site-packages\trackpy\feature.py in ?(frames, diameter, output, meta, processes, after_locate, **kwargs)
577 output.put(features)
578 finally:
579 if pool:
580 # Ensure correct termination of Pool
--> 581 pool.terminate()
582
583 if output is None:
584 if len(all_features) > 0:

A_path\Lib\site-packages\slicerator_init_.py in ?(self, i)
183 def getitem(self, i):
184 """Getitem supports repeated slicing via Slicerator objects."""
185 indices, new_length = key_to_indices(i, len(self))
186 if new_length is None:
--> 187 return self._get(indices)
188 else:
189 return cls(self, indices, new_length, propagate_attrs)

A_path\Lib\site-packages\pims\base_frames.py in ?(self, key)
97 def getitem(self, key):
98 """getitem is handled by Slicerator. In all pims readers, the data
99 returning function is get_frame."""
--> 100 return self.get_frame(key)

A_path\Lib\site-packages\pims\tiff_stack.py in ?(self, j)
129 def get_frame(self, j):
130 t = self._tiff[j]
--> 131 data = t.asarray()
132 return Frame(data, frame_no=j, metadata=self._read_metadata(t))

A_path\Lib\site-packages\tifffile\tifffile.py in ?(self, out, squeeze, lock, maxworkers)
8835 f'{self!r} reading array from closed file', UserWarning
8836 )
8837 fh.open()
8838 fh.seek(self.dataoffsets[0])
-> 8839 result = fh.read_array(
8840 keyframe.parent.byteorder + keyframe._dtype.char,
8841 product(keyframe.shaped),
8842 out=out,

A_apth\Lib\site-packages\tifffile\tifffile.py in ?(self, dtype, count, offset, out)
14214 )
14215 n = nbytes
14216
14217 if n != nbytes:

14218 raise ValueError(f'failed to read {nbytes} bytes, got {n}')
14219
14220 if not result.dtype.isnative:
14221 if not dtype.isnative:

[nkeim]

This could be the same issue as #700 . It's still unclear why this happens, and tifffile itself appears to be entirely thread safe anyway. Perhaps the workarounds to check, which may also shed light on the problem, are: (1) insist on the fork model for multiprocessing, or (2) assuming there's enough RAM, give batch a list of image arrays, instead of passing a pims reader.

[jacopoabramo]

I opened issue #700 at the time. It would be better if OP could provide a minimal reproducible example. For what it's worth, I tried replicating the error with the trackpy-example repository.

Python: 3.10 / 3.12 (tested on two different venvs)
trackpy: 0.7.0
pims: 0.7.0
OS: Windows 11

import pims
import trackpy as tp

frames = pims.ImageSequence('sample_data/foam/*.tif')

tp.quiet()  # Disabling progress reports makes this a fairer comparison

tp.batch(frames[:100], 13, invert=True, processes=3)

When launching the script like this, as stated by OP there's a bunch of neverending errors which boil down to always the same message:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

Now, if you run the following script instead:

import pims
import trackpy as tp

if __name__ == "__main__":
    frames = pims.ImageSequence('sample_data/foam/*.tif')
    tp.quiet()  # Disabling progress reports makes this a fairer comparison
    features = tp.batch(frames[:100], 13, invert=True, processes=3)
    print(f"Found {len(features)} features in the first 100 frames.")

The problem goes away. I think that pims is automatically switching to using tifffile to read the images so in the end the problem is that when using multiprocessing.Pool (which is called in tp.batch), unless you specify the execution within an if __name__ == "__main__" condition, it'll never work.

It is fork's fault that makes using the idiom required.

EDIT: this problem occurs only on scripts; when using jupyter notebooks the code works fine even without __main__
EDIT: updated to notify that i also tested on python 3.12 to have consistency

[jacopoabramo]

I think that for the time being just documenting the behavior specific for Windows in the documentation should count as a fix.