Expected procedure for adding new Rockstar revisions

[mtryan83]

(This is more of a question than a bug, so feel free to close)

The procedure for adding new Rockstar revisions seems relatively straightforward for versions that are a pure/"official" update:

1. Add revision to list of known revisions:

   - KNOWN_REVISIONS: list[int] = [0, 1, 2]
   + KNOWN_REVISIONS: list[int] = [0, 1, 2, 3]

2. Add/modify fields to match

   halo_dt: list[HaloDataType] = [
       ("particle_identifier", np.int64),
       ("particle_position_x", np.float32),
       ...
   - ("particle_corevel_z", np.float32, (1, 100)),  #Example modification/removal
   + ("particle_corevel_z", np.float32, (1, 2)), 
      ...
   + ("new_halo_field", np.float32, (3, 100)), #Example addition
   ]

What is the procedure, however, if the revision is effectively an alternate of one of the current revisions? E.g. a Rockstar dataset with all of the fields from revision 1, none of the fields from revision 2, and additional fields? Just call it revision 4 and follow the procedure above?

For context/example, I primarily use a fork of Rockstar from Andrew Wetzel (here), which follows revision 2, but has two additional fields (m_hires and m_lowres). So I'd follow steps 1 & 2 above:

KNOWN_REVISIONS: list[int] = [0, 1, 2, 3] # Added version 3
...
halo_dt: list[HaloDataType] = [
       ("particle_identifier", np.int64),
       ...
       ("av_density", np.float32, (2, 100)),
       ("m_hires", np.float32, (3, 100)), # New
       ("m_lowres", np.float32, (3, 100)), # New
]

But Andrew also has another fork (alt fork) which is based off of revision 1, with an additional field, halfmass_radius, between m_pe_d and num_p. So my options seem to be

• I could add halfmass_radii with the version gate (1, 1) or similar, but then there'd be problems if I use, say, Behroozi's original version for something.
• I could call it version 4, version gate all the fields introduced in revision 2 and add the new field, but that doesn't seem very resilient to future changes
• I could call it version 1.x (1.1, 1.5, etc) and add ("halfmass_radii", np.float32, (1.x, 1.x)),. This seems like the best option, but what happens if a later revision adds the field as well?
• Other?

Which one is preferred?

Corollary, would PR's for those changes be desired, even if they potentially conflict (e.g. like the alt fork above)?

I originally asked this question in #4997, but I've realized that's the wrong venue and am hoping to move the discussion here.

[mtryan83]

Quoting from @cphyc's reply in #4997:

So long as your alternative format has a different number of bytes in the header or in the members of the halos, what you wrote above should work (assuming m_hires and m_lowres are indeed located at the end of the halo definition).

If this works, would you mind opening a PR (on top of this one)?

As a side note, it would make the code much more readable if the syntax was:

halo_dt: list[HaloDataType] = [
    ("particle_identifier", np.int64),
    ("particle_position_x", np.float32),
    ("particle_position_y", np.float32),
    ("particle_position_z", np.float32),
    ("particle_mposition_x", np.float32, {0}),
    ("particle_mposition_y", np.float32, {0}),
    ("particle_mposition_z", np.float32, {0}),
    [...]
    ("m_pe_d", np.float32, {1, 2}),
    [...]
]

At the moment, the third (optional) argument gives a lower and upper boundary for the format revisions that should include the corresponding entry; using a set would be more explicit IMO.

[mtryan83]

I think @cphyc's suggestion of using set's for the versioning is a good solution, and I'm happy to PR both switching to using sets and to add both of Andrew's revisions, just want to verify first.

The only major issue I see is if somehow there's an explosion of Rockstar revisions that have wildly different formats, but I assume that's unlikely.

[cphyc]

At the moment révisions are uniquely identified with `ont`. It would he more readable to identify them with `int | str`, so that we can have a `wetzel_1` revision, or even point to the repo. That would pair nicely with using sets rather than bracketing versions. I see your concern regarding a potential increase in versions, so maybe another option would be to allow the descriptor to be registered, so that users can add their own version if needed. That way, we would support the two main revisions natively, and allow support for others? Either way, I think a PR would be welcome! 30 Aug 2025 04:32:42 mtryan83 ***@***.***>:

…

[Image]*mtryan83* left a comment (yt-project/yt#5278)[#5278 (comment)] I think @cphyc[https://github.com/cphyc]'s suggestion of using set's for the versioning is a good solution, and I'm happy to PR both switching to using sets and to add both of Andrew's revisions, just want to verify first. The only major issue I see is if somehow there's an explosion of Rockstar revisions that have wildly different formats, but I assume that's unlikely. — Reply to this email directly, view it on GitHub[#5278 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/ABJJII2NMEEQJMCFTGIU77L3QEEMRAVCNFSM6AAAAACFGAQNB6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMZYHA3TKMBUHE]. You are receiving this because you were mentioned. [Tracking image][https://github.com/notifications/beacon/ABJJII434DFDN6MIZGGZX2L3QEEMRA5CNFSM6AAAAACFGAQNB6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTWBBVH2S.gif]