Memory issues with Dask and Mac OS Sequoia >=15.4.1

[rickecon]

We have encountered memory issues using Dask and with Mac OS Sequoia >= 15.4.1. In particular, we are getting distributed error outs in OG-USA with the TMD microdata file as well as black screens. Google Gemini search answered the search prompt "dask issues with new macos sequoia" with the following information.

Yes, macOS Sequoia (version 15.0) has been reported to cause issues with Dask and other network-based tools. The primary problem stems from changes to the built-in Application Firewall that now incorrectly blocks incoming internet connections, even when explicitly allowed by the user. This behavior can lead to inconsistent connectivity or complete connection drops for applications like Dask, which rely on network communication. (ref 1, ref 2)

Here's a more detailed breakdown:

Issue:

Firewall Changes:
macOS Sequoia's new Application Firewall has been modified, particularly how it handles application settings in property files. (ref 1)

Blocking Incoming Connections:
This change causes the firewall to automatically block incoming internet connections for certain applications, including Dask, even if the user has explicitly granted permission. (ref 1, ref 2)

Persistent Settings:
The firewall settings are not persistent across reboots, meaning the blocked connection behavior can reappear after a restart. (ref 1)

Impact on Dask:
Dask relies on network communication for various tasks, including distributed computing. The blocked connections interfere with Dask's ability to function correctly. (ref 2)

Solutions and Workarounds:

1. Update Dask: Ensure you are using the latest version of Dask, as developers may have released patches to address these issues. (ref 2, ref 3)
2. Firewall Settings: Carefully review and adjust your firewall settings to ensure that Dask and related applications are allowed to receive incoming connections. Remember that these settings may revert after a reboot. (ref 1)
3. Disable Firewall: As a temporary workaround, you can disable the Application Firewall entirely. However, this will compromise your system's security, so use this option with caution.
4. Monitor for Updates: Apple may release future updates to address this issue. Keep your macOS up-to-date to benefit from any bug fixes.

cc: @jdebacker

[jdebacker]

I am noticing this only when estimating tax functions of the HSV form with a large microdataset. It would be worth running a memory profiler on smaller data to see where bottlenecks are. A particular place to look would be the matrix inversion when estimating the HSV functions with OLS.

[jdebacker]

@rickecon Are you still noticing these issues or can we close this?