validation (report) customisation: future directions?

[eliarizzetto]

Hey there!

Thanks for your very valuable work on this useful tool.

As far as I could gather, there is no way to define custom validation rules for the documents beyond the constraints presented in the UI (e.g. specify data types or required fields).

I realise that such flexibility would be really hard to achieve, and maybe not even among the intended uses of ODE, but would it make sense to have at least the possibility of loading and visualising pre-made validation reports? Would you consider such an extension in the future?

In my use case, I already have a validator for CSV documents which outputs a JSON validation report (https://github.com/opencitations/oc_validator), therefore a tool to let (non-technical) users visualise errors and edit the table manually in a human-friendly interface, like ODE, would be very handy. For example, ODE could maybe be extended to support custom validation reports on condition that they are Frictionless-compliant. Converting my validation reports to make them adhere to the Frictionless framework would be well worth the effort if ODE could support them, and I think that such an extension would be useful also for any other scenarios where validation tools/pipelines are already in place, but still lack the usability features in which ODE… excels [pun intended😉].

Apologies if similar or related features are already present in ODE and I missed them, any guidance on where to find them (or even vague suggestions or tips on how to implement them) would be greatly appreciated.

Thanks again,
Elia

[pdelboca]

Hello @eliarizzetto !

This feature was discussed at several points and we agree that would be of great value. Some people wonder if it was possible for management or senior team members to pre-set validation rules so then users have a target to validate their files against. That use case is quite aligned with what you describe. Having a proper Table Schema to use when validating would do the trick as behind the scenes we are already validating against a simple one.

Sadly I cannot provide any timeline of how and when this could be part of ODE, but we will let you know. (You will see activity in this issue as well)

Context

Some context of why this isn't yet in ODE.

• Early versions of the application went full into trying to support the whole of Table Schema features but it resulted in a complex application that only savvy people in the Frictionless world understood. Our target audience didn't understood how to use it.
• Table Schema has a complex structure, designing a proper UX for it to be accessible and understandable requires some deep work. (Our early designs failed)
• The goal so far, was to develop a tool to fill the existing gap for people needing to validate data but not knowing how. Our target audience are daily Excel users that want to publish better data. That's why adding complex forms or Table Schema imports are, right know, an advanced feature that we haven't prioritize.
• With constraint resources (a.k.a no funding 😂 ) we decided to deliver a complete application for a target audience (non-technical), rather than pieces of features to a broader audience.

A hacky-test

If you have knowledge of Frictionless and want to give it a try, you could do a hacky thing that might work.

As I mentioned, we are already validating against simple schemas stored in .opendataeditor/tmp/.metadata/<filename>.json. I guess that if you have a complete enough Resource Descriptor you can hook it in that file and give it a try.

It is far from ideal, but if you feel like trying stuff you could play around!