Fix SSMIS configs

[jemrobinson]

Currently SSMIS datasets downloaded over FTP have NaNs in several places. This PR adds a NanToNum filter to replace these with numbers. I've currently chosen:

• ice_conc, raw_ice_conc_values => 0.0
• algorithm_standard_uncertainty, smearing_standard_uncertainty, total_standard_uncertainty => 99

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
icenet_mp/data_processors/filters
__init__.py
doubling_filter.py					16-17, 21-23
nan_to_num.py					13, 17
Project Total

This report was generated by python-coverage-comment-action

[jemrobinson]

@marianovitasari20 : are you happy with this?

[marianovitasari20]

Hi @jemrobinson,
Does that mean NaNs are being replaced with 0? If so, won’t that be confusing with real “no ice” values rather than NaN?

[marianovitasari20]

Is 99 defined in the SSMIS product specification as a fill value, or was it chosen arbitrarily?
I’m wondering whether, if it’s not an official fill value, it could be interpreted as a genuine uncertainty value later in the pipeline.

[jemrobinson]

@marianovitasari20

Does that mean NaNs are being replaced with 0? If so, won’t that be confusing with real “no ice” values rather than NaN?

We need to remove the NaNs because otherwise the pipeline will try and fail to predict them. Options that the pipeline will be able to predict are

1. a made-up non-zero value
2. 0

Note that the current OSISAF-SSMIS dataset we're using does one of these already (I think 2. but I haven't checked).

Is 99 defined in the SSMIS product specification as a fill value, or was it chosen arbitrarily?
I’m wondering whether, if it’s not an official fill value, it could be interpreted as a genuine uncertainty value later in the pipeline.

It was chosen arbitrarily. We're not currently using these variables (although we could consider doing so in future). Do you prefer a different fill value? We could consider something like -1 instead.

The best long-term solution is to only train on those portions of the grid that actually contain potential sea ice, but we're not doing so at the moment.

[marianovitasari20]

Thanks for explaining @jemrobinson! Happy to go ahead with 99 or -1 for now. It might be worth opening an issue to track the grid masking improvement (preprocessing step) so it doesn't get lost down the line.

[jemrobinson]

• Use of masks is Add land masks to data prep #134
• NaN replacement in Decide on correct NaN handling behaviour #220