Enable flagging of dodgy observation data

[devsjc]

Moving some notes from Peter into a discussion here around the handling of dodgy input data:

It would be useful to be able to delete/ disactivsate some generation data

we sometimes get dodgy data from generation
it would be nice not to delete, but just flag

Possible Implementation

could add this flag to the metadata in the observation data
function to add this flag to observation data

[devsjc]

@peterdudfield It would be useful to get a bit of context on what makes the data dodgy. Currently there are guardrails on the database side of the Data Platform that prevent observations that

a) Are negative
b) Are over 110% of the capacity of the associated location

If this covers everything that might be descibed as dodgy data, we may not need this.

[peterdudfield]

For example a client might send 0.0 generation in the middle of the day. This is clearly wrong.

Another example could be the soar generation drops down to 10% for 10 mins, but the rest of the day is fine.

[devsjc]

So data being dodgy is something that would be be noticed after ingest, as opposed to there being any kind of check on the Quartz/User/Unified API that takes the data in?

And simply flagging it is preferable to completely deleting it?

[peterdudfield]

yea, I think we might not know until afterwards, of course, we should try to do this as automatically as possible.

I like the ideal of a user being able to flag it, rather than delete it. So its still in the database, but perhaps not used by the forecaster to make a bad forecast

[devsjc]

Out of interest, is this something we've had problems with already?

[peterdudfield]

yes, we get some blimpse in India sometimes, ive had to put something into the forecast-app to filter this out