Anomaly Detection on Great Expectations

[jmcorreia]

We have been using GX for checking the quality of our data, and it is great, but we have use cases for which we wanted to complement this Data Quality validations with anomaly detection (identifying unusual patterns, rare events, atypical behaviors, or outliers).
This would be important to automatically detect drifts, standard deviations, sudden changes in number of rows per load, sudden peak compared to min/max/avg of a few metrics and so on.
For this we would not only need adequate expectations (or we can develop custom expectations) but also some kind of metrics repository for the metrics to be dynamic according to the data.
There are already other tools like pydeequ (https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/examples/anomaly_detection_example.md) addressing problems like these. Is this something that GX has or considers having in the near/not so near Roadmap?

[cwdjankoski]

@jmcorreia Hi - I just came across your post by searching for the same thing. I just wanted to share that this (experimental) expectation exists, it seems to match the SD use case you mention.

https://greatexpectations.io/expectations/expect_column_values_to_not_be_outliers

hope it helps, curious also if you have solved it in a different way with some other expectation. thanks in front.

[priorwave]

It looks like this link is no longer valid, do we know what happened to this ?

[cwdjankoski]

hm, you are right the link is no longer valid but also i could not find the corresponding expectation anymore in the gallery.

[indresh-singh]

It would have been real good to have something like this, but unfortunately it's not visible anymore