Validation

This section presents some examples from the version 3 networks validation to give a potential user an idea of the SEVIRI_ML performance. A scientific manuscript with a full validation is in the making at this time.

All products (CMA/CPH/CTP/CTT) have been validated against CALIOP lidar observations which are assumed to have a higher quality than cloud parameters derived from passive instruments.

CALIOP CLAY v4.20 level-2 measurements have been temporally and spatially collocated with the SEVIRI pixels using atrain_match (https://github.com/foua-pps/atrain_match). SEVIRI pixels with scanning angles > 70° have been dropped from the collocations.

Cloud Mask

Tab.1: Cloud mask (CMA) validation statistics against CALIOP.

Score	Value
Hitrate	85%
Probability of Detection cloud	83%
Probability of Detection clear	89%
False Alarm Rate cloud	7%
False Alarm Rate clear	23%
Heidke Skill Score	0.70
Hansen-Kuipers Skill Score	0.72
CFC Bias	-6%
SEVIRI mean CFC	56%
CALIOP mean CFC	62%
Number of observations	580 414

Fig. 1: Cloud mask (CMA) validation statistics against CALIOP on a 5x5 degrees grid.

Cloud Phase

Tab.2: Cloud phase (CPH) validation statistics against CALIOP.

Score	Value
Hitrate	87%
Probability of Detection ice	88%
Probability of Detection liquid	87%
False Alarm Rate ice	11%
False Alarm Rate liquid	15%
Heidke Skill Score	0.74
Hansen-Kuipers Skill Score	0.74
CFC Bias	-1%
SEVIRI mean LCF	46%
CALIOP mean LCF	46%
Number of observations	267 900

Fig. 2: Cloud phase (CPH) validation statistics against CALIOP on a 5x5 degrees grid.

Cloud Top Pressure

Tab.3: Cloud top pressure (CTP) validation statistics against CALIOP.

Statistic	ALL	LIQ	ICE
Bias [hPa]	-4.4	-38.1	9.8
MAE [hPa]	85.1	62.6	45.0
RMSE [hPa]	155.9	99.1	76.1
SEVIRI mean [hPa]	445.6	705.8	227.3
CALIOP mean [hPa]	450.0	743.9	217.4
Number of observations	267 899	106 258	127 256

Fig. 3: Cloud top pressure (CTP) validation statistics against CALIOP.

Fig. 4: Cloud top pressure (CTP) validation statistics against CALIOP for each of the available CALIOP cloud types.

Cloud Top Pressure Uncertainty

Fig. 5: Cloud top pressure (CTP) uncertainty validation statistics. The uncertainty represents the 1-sigma uncertainty interval in which ideally 68% of all CTT values should be located. To validate the uncertainty we calculated the PICP, i.e. the percentage of all values falling within the 1-sigma prediction interval. (a) Mean CTP uncertainty interval for different CALIOP CTP bins. (b) Black: PICP as a function of CALIOP CTP bins. Red: Ideal 68% line. (c) Mean CTP uncertainty for each of the CALIOP cloud types. (d) Black: PICP for each of the CALIOP cloud types. Red: Ideal 68% line.

Cloud Top Temperature

Fig. 6: Cloud top temperature (CTT) validation statistics against CALIOP.

Multilayer Flag (experimental)

TBD