[Rockstar] Suggestion: Decouple 'num_blocks' from 'num_readers' in the halo finder interface

[Sergarmor]

Hello,

I am using the rockstar-galaxies halo finder through yt_astro_analysis and have encountered a limitation related to how parallel execution is configured.

The Problem

The current Rockstar interface in yt assumes that the number of file blocks for a snapshot (NUM_BLOCKS in Rockstar's config) is always equal to the number of reader processes (num_readers). This becomes an issue when analyzing simulations where the number of files per snapshot is greater than the number of available CPU cores on a machine.

Standalone Rockstar allows setting NUM_BLOCKS and NUM_READERS independently, which provides the flexibility to process a large number of files with a smaller number of cores.

My Use Case

• My simulation snapshot is split into 32 HDF5 files. Therefore, Rockstar requires NUM_BLOCKS = 32.
• My local machine has 22 physical cores.
• Ideally, I would like to run the analysis with a configuration like num_readers = 16 and num_writers = 4, which is well within my hardware limits.

Current Behavior and Error

Because the yt interface ties num_blocks to num_readers, I am forced to set num_readers = 32 to ensure all files are processed. When attempting to run this, the script fails shortly after starting, throwing a parallel HDF5 access error across multiple MPI ranks.

Here is the key error message:

KeyError: 'Unable to synchronously open object (unable to determine object type)'

This seems to be caused by contention issues when too many processes try to access the HDF5 files simultaneously on an oversubscribed system.

Proposed Solution

It would be extremely helpful to decouple these two parameters. A great solution would be to add a num_blocks keyword argument to the finder_kwargs passed to Rockstar with HaloCatalog. For example:

hc = HaloCatalog(
    data_ds = tsAREPO,
    finder_method = "rockstar",
    finder_kwargs = {
                     "num_readers":16,
                     "num_writers":4,
                     "num_blocks":32,
                     "particle_type":"PartType1",
                     }
)

To maintain backward compatibility, num_blocks could simply default to the value of num_readers if it is not explicitly provided.

Question

In the meantime, is there a recommended workaround for this scenario, other than manually patching the local yt_astro_analysis installation?

Thanks for developing and maintaining this great tool! Any guidance would be much appreciated.

[brittonsmith]

@Sergarmor, I don't see anything wrong with your proposed solution and would be happy to have this behavior in the main codebase. Would you be willing to issue a pull request for this?

[Sergarmor]

@brittonsmith I tried patching the code myself but on executing the command, it got stuck after reading the files (with a 0-1% usage of CPU), so my guess is that this problem could be related with how the code is parallelized, which I am unfamiliar with.

As a workaround with the current codebase, if someone comes around this, is assigning num_readers = number of files per snapshot, num_writers = num_readers + 1 and running the script with as many cores assigned as needed, even if you don't have as many. In my particular case, I assigned 66 cores even though my machine only has 22 and it ran without complaint.

[brittonsmith]

@Sergarmor after reading this again more closely, I might have previously misunderstood the problem. I frequently run rockstar with fewer readers than blocks (for Enzo simulation data) without a problem. I actually wonder if the issue is that you can't have more readers than writers. Does the code really not work if you set num_readers=4, num_writers=4 and run with a total of 9 processes?

[Sergarmor]

Sorry for the late response, I was away for a week and couldn't test this.

Apparently it does run with 4-4 -> 9 processes. Now, my guess is that the source of the error was starting from the first snapshot of the simulation (z=127) instead of somewhere with z~20, like I ended up doing.

Either way, the Rockstar log shows "Reading 4 blocks for snapshot n...". I don't know if this is just a log issue while the snapshot reading process is handled by yt or if rockstar is truly only reading 4 files out of 32.

[brittonsmith]

No problem. I think this is just misleading wording in the log message. Until I read your issue, I always assumed that any number of readers would still always read the entire dataset. Indeed, I often run with just one reader.