Skip to content

load_book: support for fixed tones, smurfgaps flags, no_headers #1219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 4, 2025

Conversation

mhasself
Copy link
Member

  • When "special_channel=True" is passed to get_obs, the "untracked" channels are loaded and populate a .tones sub-AxisMan.
  • The smurfgaps flags are loaded from the per-stream frame data and put into .flags.smurfgaps_{stream_id}.
  • When user passes "no_headers=True", the smurf header fields will not be unpacked (so there is no .primary).

mhasself added 2 commits May 13, 2025 03:01
- When "special_channel=True" is passed to get_obs, the "untracked"
  channels are loaded and populate a .tones sub-AxisMan.
- The smurfgaps flags are loaded from the per-stream frame data
  and put into `.flags.smurfgaps_{stream_id}`.
- When user passes "no_headers=True", the smurf header fields will
  not be unpacked (so there is no .primary).
@mhasself
Copy link
Member Author

Addresses #1160. Here's what smurfgaps looks like (obs_id='obs_1745071718_satp1_1111111')

>>> tod.flags
FlagManager(smurfgaps_ufm_mv18[samps], 
 smurfgaps_ufm_mv19[samps], 
 smurfgaps_ufm_mv22[samps], 
 smurfgaps_ufm_mv48r1[samps], 
 smurfgaps_ufm_mv50r2[samps], 
 smurfgaps_ufm_mv51[samps],
 smurfgaps_ufm_mv52r1[samps],
 dets:LabelAxis(11935), samps:OffsetAxis(475151))

>>> for k, v in tod.flags._fields.items():
...   print(k, v.ranges())
... 
smurfgaps_ufm_mv18 [[90134 90171]
 [90172 90322]]
smurfgaps_ufm_mv19 [[90134 90141]]
smurfgaps_ufm_mv22 []
smurfgaps_ufm_mv48r1 []
smurfgaps_ufm_mv50r2 []
smurfgaps_ufm_mv51 [[90134 90138]]
smurfgaps_ufm_mv52r1 [[90134 90220]]

@mhasself
Copy link
Member Author

Also addresses #1190. E.g:

obs_id = 'obs_1744840989_satp1_0000010'
tod = ctx.get_obs(obs_id, special_channels=True)

Then:

>>> tod.tones
AxisManager(stream_id[tdets], band[tdets], channel[tdets], signal[tdets,samps], 
tdets:LabelAxis(23), samps:OffsetAxis(1318))

>>> tod.tones.tdets.vals
array(['ufm_mv52r1_0_38', 'ufm_mv52r1_0_377', 'ufm_mv52r1_2_326',
      ...
      'ufm_mv52r1_6_198', 'ufm_mv52r1_6_362'], dtype='<U16')

>>> tod.tones.signal
array([[ 1.6924602,  1.6927478,  1.6926519, ...,  1.692556 ,  1.6924602,
         1.691885 ],
       [ 2.4248402,  2.4247444,  2.4246483, ...,  2.4244566,  2.4244566,
         2.4244566],
       ...,
       [ 1.2802029,  1.280107 ,  1.280011 , ...,  1.2802987,  1.280107 ,
         1.280011 ]], dtype=float32)

@mhasself mhasself requested a review from kmharrington May 13, 2025 03:21
Copy link
Member

@kmharrington kmharrington left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!

Any chance there's a way to make operations books automatically load headers but not obs books? I suspect we'll usually want it when operation books are being loaded.

@mhasself
Copy link
Member Author

Any chance there's a way to make operations books automatically load headers but not obs books? I suspect we'll usually want it when operation books are being loaded.

Yes... this seems to be quite a big savings, so it's in place now. At least when you use context.get_obs.

I had assumed that loading headers was cheap, if one were already loading signal. But looks like that's not the case ... my random test suggests 40% increase in load time, when reading headers.

I cleaned up a couple other things. Will merge when tests pass unless there are further concerns.

@mhasself mhasself merged commit 164ac3a into master Jun 4, 2025
5 checks passed
@mhasself mhasself deleted the mhasself/more-loader-opts branch June 4, 2025 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants