Revised Masks to account for "dodgy astrometry"

[cheymans]

After several months of back and forth with ESO to get our test DR5 batch submission approved, we've finally got a green light to proceed with the bulk upload. One of the bonuses of this delay is that it's given the fabulous team the time needed to define the final set of masks.

I would argue that we do want the additional "dodgy astrometry" mask to be included in the DR5 release: it's bound to mess up other projects too.

@AngusWright: what format does it currently exist in? Ideally we would want to update the 10-band fits mask, and the MASK column in the superuser catalogues before we bulk upload them. This won't delay the upload of all the AW and THELI images though.

[hendrik1008]

Shouldn't we also include the "dodgy astrometry" information in the weight or flag images of the THELI data?

[cheymans]

I can't see ESO users using anything other than the AW-reduction, to be honest, so I think we would be justified in choosing to avoid that extra work?

[cheymans]

@AngusWright @hendrik1008, give me a shout when the catalogues and 10-band mask are updated. I'll upload the THELI imaging in the mean time!

[cheymans]

Ah, rather boringly I've realised my ESO conversion scripts for the THELI images, also convert the masks. It's not a massive pain to re-structure them - but it is a minor pain. If new mask fits files could be created sooner rather than later, I'd probably just hold off and wait for them (although I really do want to win the #31 race with Andrej).

[AngusWright]

@cheymans we discussed this at the cosmic shear call and there was consensus that it would be better to include the astrometric rejection metric as an additional data product, rather than using it to define a single mask.

The main reason is that: whether or not the selection we apply as fiducial is appropriate for arbitrary science is not obvious. This cut will be variously too aggressive or much too conservative if someone wants to, e.g., do a different scale cut, or use a different number of cosebis modes, or use different tomographic bins, or use a different shape-measurement, etcetc. Providing the means to easily tailor this selection as required enables other (& better) science down the road.
For ESO specifically, we're also talking about applying a predominately lensing-quality metric to the photometric catalogues which contain no lensing information. We've not established, e.g., that the majority of this area is problematic for photometric clustering with the organised randoms (the effect appears to be overwhelmingly in shape).
Delivering the additional data product also means that the MASK remains as is, so no additional work with data products for ESO, no further delays in the release, and avoids us having to write errata for the DR5 paper, as numbers there are clearly based on MASK. In the redshifts/cosmic shear papers we comment on the difference in number & area and explain that this change is motivated by our failed null test.
Lastly, there's also the possibility that this metric is not optimal: we know that it doesn't select 1:1 the differences between DR4 & DR5 ellipticities, so it would be a shame to blanket reject a chunk of area (unrecoverabely, as this mask would have to be incorporated into an existing MASK bit) for all science if a better metric comes along later.

However, if this doesn't sit well, then a simple compromise would be to add the metric as a new column to the superuser catalogues, just like we do with MASK. That would require changing the column metadata again in the superuser catalogues though.

And of course for the lensing sample, which we only release on the KiDS pages, we can do what we like really; tag all sources with the metric value, add a flag, do both, include the mosaic map of the metric, etcetc. I think lensing-sample wise this is an easy fix.

Sorry for the long message, let me know what you think 👌

[cheymans]

Thanks for this Angus. I know what you propose is the easiest route, but this is KiDS-Legacy, and you've all worked really hard to "do this right". My vote would be to put in the extra effort here to make this release as future-proof as possible. Let's take strong lensing and cluster finders as examples: in regions where the astrometry is so poor we're getting double images, they will flag up in an automated search of the multi-band catalogues as a dark lens/massive cluster. These communities aren't going to be reading the cosmic shear paper so won't know that we have this issue. I know nobody likes an erratum, but we need to flag what is problematic about the THELI-related products in an easily accessible way. And as the paper is not on the arxiv yet, the majority of readers won't even notice the difference.

On the ESO release:
I agree with you that our selection may not be optimal for everyone, and so you're right that a MASK bit is too black-and-white. Could we therefore follow your proposal of not changing the MASK.fits file, but adding two columns to the 10-band catalogue: the metric value and an 'our choice' flag. It's straightforward to update the scripts at my end to cope with the new additions, and also update the release document.

On the DR5 paper:
The erratum can be in the form of an extra appendix for the details, but some additions will also need to be made to the existing text. Specifically

• add additional rows in Table 12, 14
• add additional information to section 7.2 and 9 to quote the final neff we are using
• update Table I.1 and I.3 to include two new columns

Sorry to be a bore - I know the focus is 100% on the cosmic shear paper, but I'd vote to take the hit of a week or so delay on that paper to get this DR5 release right.

[cheymans]

Update:
@AngusWright has added three new columns to the catalogue:

TTYPE352  |  TTYPENA | max_dRADec_AW | meta.code | deg  | 1D | Max separation between THELI and AW coadd |
TTYPE353  |  TTYPENA | max_dRADec_THELI | meta.code | deg  | 1D | Max internal separation exposure centroid and stack |
TTYPE354  |  TTYPENA | ast_flag | meta.code |   | 1B | Fiducial astrometric accuracy selection |

Alas, not time to close this issue yet though, as the information needs updating in the data release document (#20) and the data paper.

[cheymans]

Alas, we have a handful of catalogues where the format of the three additional columns varies from the majority of catalogues. For the majority of fields we have

TTYPE382= 'max_dRADec_AW'      / Maximum spearation between THELI and AW coadd c
TFORM382= '1D      '           /                                                
TUNIT382= 'degree  '           /                                                
TTYPE383= 'max_dRADec_THELI'   / Maximum internal spearation between exposure ce
TFORM383= '1D      '           /                                                
TUNIT383= 'degree  '           /                                                
TTYPE384= 'ast_flag'           / Fiducial astrometric accuracy selection: max_dR
TFORM384= '1B      '           /          

But for 24 fields we have:

TTYPE382= 'max_dRADec_AW'      / Maximum spearation between THELI and AW coadd c
TFORM382= '1E      '           /                                                
TTYPE383= 'max_dRADec_THELI'   / Maximum internal spearation between exposure ce
TFORM383= '1E      '           /                                                
TTYPE384= 'ast_flag'           / Fiducial astrometric accuracy selection: max_dR
TFORM384= '1I      '           /    

This is the list of fields affected:

 KiDS_DR5.0_187.5_-3.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_46.7_-26.2_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_237.7_2.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_238.0_-0.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_238.0_0.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_238.0_1.5_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_238.0_-1.5_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_156.0_0.5_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_156.0_1.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_156.0_-0.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_156.0_-1.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_156.4_2.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_156.4_3.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_156.4_-2.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_156.4_-3.5_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_27.6_-34.1_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_45.6_-26.2_ugri1i2ZYJHKs_cat.fits 
 KiDS_DR5.0_47.8_-26.2_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_48.9_-26.2_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_329.4_-32.1_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_50.0_-26.2_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_51.1_-26.2_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_52.2_-26.2_ugri1i2ZYJHKs_cat.fits
 KiDS_DR5.0_53.3_-26.2_ugri1i2ZYJHKs_cat.fits

I imagine the fastest and safest thing would be to remake v4 cats for those 24 fields, sync them to Edinburgh, remake the ESO-compliant versions and then upload them to ESO? Alternatively I could try and modify the format here. Let me know your preference @AngusWright

[AngusWright]

I will remake these this morning. I'm pretty sure that I know why this has happened.

[AngusWright]

Files are updated in the THELI tree in Bochum, just need either @cheymans or @hendrik1008 to login and send them on to Cullin 👌

[hendrik1008]

Transferred!

[cheymans]

Success - that seems to have done the trick. Thanks for the rapid action @AngusWright and @hendrik1008.

Once we work out how to fix the curious case of KiDS_DR5.0_355.3_-31.2_r_sci.fits (#32) we can then move to the next 'deep dive' stage of faffing where the checker flags up double objects (objects that lie on the wcs cuts and noise falls them one way on one field, and one way on the other field so they end up appearing twice) and I manually remove them. There is a small chance this will happen this side of xmas, but it depends on Nausicaa's holiday plans. Let's see (crosses fingers).

Keeping issue open as a reminder that I need to update the release document with info about the flags.

[cheymans]

@AngusWright - we're tantalisingly close to finishing the DR5 release before the holidays (only one issue remains which only ESO can fix, and I'm hopeful as Nausicaa is on the case). That means it's time to finalise the release notes.

Do you have any draft text ready on the astrometry bug that I can summarise to explain what the additional columns mean?