Action items:

• Make sure separate logs print without mixed-together printing to the command line.
• Get all 6 tests passing.
• Produce a complete test_images_summary.md

Rebased after merging #751

I've tested Perlmutter and Chrysalis on compute nodes.

I don't have recent results from using a dev environment at the moment, but I appear able to activate a dev environment and run the unit tests on a Compute node.

Now that I've tested using these commits on Chrysalis, Perlmutter, and Compy and using compute nodes, I think this should be good to merge.

I remembered seeing a deadlock warning on Compy, but I don't recall seeing that on the other machines. In any case, Claude suggested the fixes in 9e2776d for that. I will test on Compy.

The spawn option works, but was 2.4x slower. Below, from Claude:

Let me break down how 'forkserver' works:

The Three Multiprocessing Methods

'fork' (default on Linux/Unix)

Parent Process (with threads) 
    |
    fork() --> Child Process (inherits threads - DANGEROUS!)

• Problem: The child gets a copy of the parent's memory, including thread states
• Threads in the parent were mid-operation, but in the child they're frozen/dead
• This can cause deadlocks if child tries to use locks those threads held

'spawn' (default on Windows)

Parent Process
    |
    spawn() --> Brand New Python Process
                    |
                    Import everything from scratch
                    Initialize all modules
                    Load your code
                    Finally run your function

• Safe: Child has no threads because it starts completely fresh
• Slow: Must re-import all modules, re-initialize everything (your 15 min runtime!)

'forkserver' (hybrid approach)

Step 1 (happens once at startup):
Parent Process --> Start Clean Server Process (no threads yet!)

Step 2 (for each worker):
Clean Server Process
    |
    fork() --> Worker Process (inherits clean state, no threads!)

Key Insight

The trick is timing:

1. Before your test code runs and creates threads, the forkserver starts a pristine Python process
2. That server process has all the basic Python machinery but no application threads
3. When you need workers, they fork from the clean server, not from your thread-heavy parent process

Visual Timeline

Time -->

'fork':
[Parent imports libs, threads start] --> [fork - COPIES THREADS] --> Danger!

'spawn':  
[Parent imports libs, threads start] --> [new process] --> [re-import everything] --> Slow!

'forkserver':
[Start clean server] --> [Parent imports libs, threads start] --> [server forks] --> Fast & Safe!

Why It's Faster Than 'spawn'

The server process can fork (copy-on-write memory) because it has no threads to worry about. So workers get:

• All the imported modules (numpy, pandas, etc.) via copy-on-write ✓
• No thread baggage ✓
• Much faster startup than re-importing everything ✓

Trade-off

'forkserver' has a small one-time startup cost (starting the server), but then each worker creation is much faster than 'spawn' because it uses fork's efficient copy-on-write mechanism.

For your use case with 6 workers, 'forkserver' should be significantly faster than 'spawn' while remaining safe.

Yes, forkserver gives us comparable speeds to before using spawn, without the deadlock concern. 69a6487 switches to that.

I've tested the new commits on Compy, so now I think this should be good to merge.