Integrate DataBricks data quality checks also in GitHub packages

[AnneSchoenauer]

To have the same checks included in the tiltIndicatorAfter package as you run on the outputfiles it would be great if the data quality checks could be implemented also on GitHub. This ensures that users that don't have access to DataBricks can also run the checks. As I understand those checks could be implemented in the workflow to ensure that the output of the data is as we want it. I am reffering to those checks here:

How to best do it? Would it be a good thing if you @SKruthoff and @ysherstyuk show @kalashsinghal where to find these tests so that @kalashsinghal can integrate this in the tiltIndicatorAfter package?

@Tilmon could you please also have a look if you find your tests in June again so that we ensure that they are also integrated?

[Tilmon]

Hi @AnneSchoenauer @kalashsinghal @ysherstyuk @SKruthoff ,

"my" tests from data prep in June (at least the ones that I can still find) are all already integrated into tiltIndicatorAfter tests (see here https://github.com/2DegreesInvesting/tiltIndicatorAfter/tree/main/tests/testthat).

[bobresink]

Hi @Tilmon @AnneSchoenauer @SKruthoff ,

As just discussed, @SKruthoff will publish the Data Quality script in shared location, which can be accessed by Mauro and @kalashsinghal, in case they want to run the test locally. Test results will then be visible in the DQ dashboard. Once new checks need to be added, it is requested and added through a release, where Letizia can add them to both the script and dashboard. The documentation, data definitions and structure are added to this shared location.

Keeping this ticket open in the meantime