Skip to content

High memory usage when frequency averaging MWA correlator fits files #1624

@ldress

Description

@ldress

I have read in a ~13 GB set of MWA correlator fits files (from one 2 minute observation at 40kHz) whose frequency resolution I'd like to reduce. However, when I run frequency_average(n_chan_to_avg=2) on this data, the process consumes more than my available system memory (96 GB) and fails. Averaging over 4 channels succeeds but still uses almost all available memory. As a workaround, I can manually process a few files at a time and concatenate the results, but it might be nice to have a built-in process that doesn’t load as much data into memory at once.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions