DESI-COSMOS and DESI-XMMLSS Spectroscopic Redshift VAC Review (DR2)

[weaverba137]

Contact Person: Joshua Ratajczak
2 catalog files (fits), 1 README

Initial Checks:

• Includes README
• Columns in ALLCAPS
• Extension names in ALLCAPS
• Files include units
• VAC publication passed CWR

Initial Notes:

• Thank you for including extension names.
• The FITS data tables have columns that are a mix of cases. We usually prefer ALLCAPS, but it appears that many of the columns are copied from other sources. If there is no harm in making the column names ALLCAPS, can you please do that?
• The data tables do not have any units (i.e. TUNIT keywords) that I can see. Can you please add units?
• There are a number of columns that appear to have TNULLxxx header keywords. Those could cause subtle issues and should only be used if really necessary.
• In both FITS files the first data HDU has invalid DATASUM and CHECKSUM keywords. The other HDUs do not have checksum information at all. Please add verified checksum information to all the HDUs.
• Finally, both files have a very unusual empty HDU 0, this example is from DESI-COSMOS.fits:

SIMPLE  =                    T / conforms to FITS standard
BITPIX  =                    8 / array data type
NAXIS   =                    0 / number of array dimensions
EXTEND  =                    T
M       =
E       =
R       =
G       =
E       =
END     

Can you please remove the MERGE?

[weaverba137]

PS, we will need to think of a short "stub" name and version. I suggest cosmos-xmmlss/v1.0, but other suggestions are welcome. The stub name will be used for the final directory as well as the documentation on https://data.desi.lbl.gov.

[jratajcz9]

Thank you Ben for all the comments. I have put updated catalogs in the same directory labeled DESI-COSMOS_v2.fits and DESI-XMMLSS_v2.fits.

Update notes:

• Changed all column names across all extensions to ALLCAPS
• Added units to all extensions
• Added DATASUM and CHECKSUM information.
• Edited HDU 0 and removed 'MERGE'. Added EXTENSION_NAME plus some other METADATA.
• stub name that you suggested is good!

Remaining Questions:

• TNULLxxx columns are columns where I have 'filled' entries. What are your suggestions for circumventing this in a way that will not cause issues for the users?
• HDU 0 is filled with some METADATA however I do not know if the METADATA I added is useful. Is there is other METADATA that is necessary or useful to add here?

[weaverba137]

Thank you @jratajcz9. I took a second look and have additional comments.

Changed all column names across all extensions to ALLCAPS

Looks good to me.

Added units to all extensions

This is going to need another round. The issue is that constructions like 1/deg^2 and 1/nanomaggy^2 are not valid in the FITS standard, and if you open these files with astropy.io.fits, you will get warnings about that. Those should be replaced with deg^-2 and nanomaggy^-2. We have some tools available to update units without disturbing other parts of a FITS file, so let me know if you need that.

Added DATASUM and CHECKSUM information.

Looks good to me.

Edited HDU 0 and removed 'MERGE'. Added EXTENSION_NAME plus some other METADATA.

Looks good to me.

stub name that you suggested is good!

I'll add that to the VAC planning document on Trac.

TNULLxxx columns are columns where I have 'filled' entries. What are your suggestions for circumventing this in a way that will not cause issues for the users?

OK, so those are "blank" in the original data sources then? If so, let's not try to change that, but be sure to document that in desihub/desidatadocs#168.

HDU 0 is filled with some METADATA however I do not know if the METADATA I added is useful. Is there is other METADATA that is necessary or useful to add here?

The HDU 0 metadata looks fine. However, you have SPECPROD = 'loa'. Is this catalog in fact based on loa/DR2? I was assuming that this was based on DR1, and therefore that all of the source data was already public.

[jratajcz9]

I have attempted fixes to the units issue although I am not sure they are resolved yet. I may need some guidance in resolving this, especially with columns with 'nanomaggy' as the units. The updated catalogs are now found here: DESI-COSMOS_v3.fits and DESI-XMMLSS_v3.fits

Notes:

• Attempted fix for units in proper formatting for deg^-2, mas^2/yr^2, nanomaggys^-2
• Updated some content to the catalog upon request of other collaborators
• The catalog is in fact based out of loa. We have permission from the previous spokespersons to release catalog level data through the end of Y3 observations as a VAC.

[weaverba137]

I think the units are good in v3. I would not make further changes to units at this point unless required by changes to content.

That's OK if you've got pre-approval for loa data, but it could change where we locate the data for public release, so let me consult on that.

For the purposes of the files, we can now finalize these. I recommend removing the older versions and renaming the v3 files to their final filenames.

[weaverba137]

Until DR2 becomes public, we recommend making the FITS files available under https://data.desi.lbl.gov/public/papers, but we would need to know which DESI working group the paper is associated with.

[jratajcz9]

I have cleared out that directory of extra files and renamed those files to their final version. The paper is associated with the C3 working group. Thanks!

[weaverba137]

OK, the next step is to work on the documentation. I've noted some issues with the existing README file in desihub/desidatadocs#168. Please take a look at that.

[jratajcz9]

I apologize I cannot seem to find the page for desihub/desidatadocs#168. Can you direct me to it?

[weaverba137]

I have sent an invitation to view that repository, please check your email.

[jratajcz9]

Great thank you!

[jratajcz9]

Hi Ben, We are looking to post to the arxiv tomorrow with the data set. What do you need from me to make the link with the catalogs live?

[weaverba137]

I've copied the catalogs to the public/papers/c3 directory, please take a look . It is technically already public, but won't be accessed very much until the paper appears.