Improve performance of margin mapping stage

[hombit]

Bug report

"Main" catalog import pipeline optimizes the mapping stage by loading ra/dec columns only. We should do the same for the margin mapping stage. The motivation is generating margins for catalogs with massive rows, especially when fetching data from a remote filesystem.

Motivated by trying to create margin cache for mmu_manga catalog, see details here:
UniverseTBD/mmu-hdf-to-hats#59 (comment)

• I have described the situation in which the bug arose, including what code was executed, and any applicable data others will need to reproduce the problem.
• I have included information about my environment, including the version of this package (e.g. hats_import.__version__)
• I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
• If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

[tom-hehir]

Hi Konstantin -- this reminded me that this was the motivation for disabling the margin cache when building mmu manga, but you figured that out already 😉

[nevencaplar]

@tom-hehir How critical is this? Is this blocking creating margins for mmu manga?

[hombit]

@nevencaplar yes, it is kinda a blocking issue for manga margins - memory consumption is huge. I can try to rerun it with 512GB ram later next week

[tom-hehir]

From what I remember, the margin cache building stage was going to roughly double the time to build the dataset (it needed similar resources to building the 'main' dataset). It's not completely blocking as we could run it with enough compute + time, but Konstantin's proposed fix would be a high-impact optimization for scenarios like that posed by MaNGA.

[dougbrn]

Should at least be improved by #687, seeing a ~2x speedup in initial local testing with mmu_manga for margin mapping