Import the DP2 pilot prep catalog on UW machines using the modification of the DASH pipeline, mirroring the successful import completed on USDF (see #1316). The goal is to test infrastructure for importing Rubin data on UW machines, and to test if all the required data has been transferred from USDF.
The data is at /astro/store/shire/rubin-data/dp2_prep_catalog. This is the data we already imported on USDF, in ticket #1316. The goal is to import the same data using UW machines.
Report back with any problems.
Compare times, memory consumption to the run on USDF
Flag any potential feasibility problems for larger runs (10TB and more of final data)
Compare the catalog on UW machines and USDF - e.g., size and number of rows, to confirm that the catalogs are indeed the same.
Import the DP2 pilot prep catalog on UW machines using the modification of the DASH pipeline, mirroring the successful import completed on USDF (see #1316). The goal is to test infrastructure for importing Rubin data on UW machines, and to test if all the required data has been transferred from USDF.
The data is at
/astro/store/shire/rubin-data/dp2_prep_catalog. This is the data we already imported on USDF, in ticket #1316. The goal is to import the same data using UW machines.Report back with any problems.
Compare times, memory consumption to the run on USDF
Flag any potential feasibility problems for larger runs (10TB and more of final data)
Compare the catalog on UW machines and USDF - e.g., size and number of rows, to confirm that the catalogs are indeed the same.