Skip to content

Import DP2 pilot catalog in HATS on UW machines using the DASH pipeline #1329

@nevencaplar

Description

@nevencaplar

Import the DP2 pilot prep catalog on UW machines using the modification of the DASH pipeline, mirroring the successful import completed on USDF (see #1316). The goal is to test infrastructure for importing Rubin data on UW machines, and to test if all the required data has been transferred from USDF.

The data is at /astro/store/shire/rubin-data/dp2_prep_catalog. This is the data we already imported on USDF, in ticket #1316. The goal is to import the same data using UW machines.

Report back with any problems.
Compare times, memory consumption to the run on USDF
Flag any potential feasibility problems for larger runs (10TB and more of final data)
Compare the catalog on UW machines and USDF - e.g., size and number of rows, to confirm that the catalogs are indeed the same.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

To Do

Relationships

None yet

Development

No branches or pull requests

Issue actions