Add coverage data to status.json

[Bjwebb]

Suggested by @drkane in ThreeSixtyGiving/test_registry#5

• for each field in the file:
  • field name
  • number of null/blank values
  • (future development) - number of values matching the required format (eg conform to regex)
  • (future development) - for fields that can be externally validated (eg identifiers) - number of values that are externally verified
  • whether the field appears in the 360 giving standard (plus guess at the field type if a non-standard field)

[stevieflow]

I would like to add (from discussion with @kindly & @Bjwebb) -

• Number of fields per unique identifier

By this I refer to the example whereby a single grant might have multiple locations (for example). If we only had a count of the instances of the fields then we might miss the potential to see if some grants are more populous than others.

eg: a dataset with a 1000 grants could have 1000 locations, but only in 1 grant

[Bjwebb]

There's now a pull request at ThreeSixtyGiving/datagetter#22

What's done:

• field name
• whether the field appears in the 360 giving standard
• Number of fields per unique identifier - actually I count the number of grant json objects with the field, which is almost always the same thing

What's not done:

• number of null/blank values
• (guess at the field type if a non-standard field)
• (future development) - number of values matching the required format (eg conform to regex)
• (future development) - for fields that can be externally validated (eg identifiers) - number of values that are externally verified

Here's the coverage.json generated by that branch https://storage.googleapis.com/datagetter-360giving-output/branch/11-coverage/coverage.json

Here's a snippet of what the coverage data looks like:

    "datagetter_coverage": {
      "/grants": {
        "grants_with_field": null,
        "standard": true,
        "total_fields": 1
      },
      "/grants/Grant Application Scheme": {
        "grants_with_field": 66,
        "standard": false,
        "total_fields": 66
      },
      "/grants/amountAwarded": {
        "grants_with_field": 228,
        "standard": true,
        "total_fields": 228
      },
      "/grants/awardDate": {
        "grants_with_field": 228,
        "standard": true,
        "total_fields": 228
      },
      "/grants/beneficiaryLocation": {
        "grants_with_field": 228,
        "standard": true,
        "total_fields": 228
      },
      "/grants/beneficiaryLocation/countryCode": {
        "grants_with_field": 210,
        "standard": true,
        "total_fields": 221
      },
      "/grants/beneficiaryLocation/name": {
        "grants_with_field": 228,
        "standard": true,
        "total_fields": 239
      },

@stevieflow @drkane How does this look?

[drkane]

Looks great.

Would it always be a separate file or would it populate the status.json file? We would presumably then end up with a three-tiered approach: data.json has the main registry in DCAT format, status.json has the additional datagetter_metadata and datagetter_aggregates and then coverage.json also includes datagetter_coverage.

Something that could cause confusion is that the fields are identified using the proper schema representation, but in the file they are likely to be in the Recipient Org:Name format. It would be helpful to have the original field name, perhaps as a name key in the object for each field, to help map these fields to the data itself. Not sure if that's possible with how the coverage file is generated at the moment though.

[drkane]

Just looked at coverage.py, looks like it's probably not possible.

[stevieflow]

We would presumably then end up with a three-tiered approach: data.json has the main registry in DCAT format, status.json has the additional datagetter_metadata and datagetter_aggregates and then coverage.json also includes datagetter_coverage.

I'd be interested to know if this is the direction we take, as have made similar comments elsewhere. @drkane - as a User (!) - would you find this useful>?

[Bjwebb]

@drkane wrote:

Something that could cause confusion is that the fields are identified using the proper schema representation, but in the file they are likely to be in the Recipient Org:Name format. It would be helpful to have the original field name, perhaps as a name key in the object for each field, to help map these fields to the data itself. Not sure if that's possible with how the coverage file is generated at the moment though.

Just looked at coverage.py, looks like it's probably not possible.

Yeah this would be reasonably involved to do. Options are to reconstruct the title by looking in the schema, or to get flatten-tool to produce its "heading source map" (this is how CoVE knows the real name of the column for validation errors).

[Bjwebb]

We would presumably then end up with a three-tiered approach: data.json has the main registry in DCAT format, status.json has the additional datagetter_metadata and datagetter_aggregates and then coverage.json also includes datagetter_coverage.

I'd be interested to know if this is the direction we take, as have made similar comments elsewhere. @drkane - as a User (!) - would you find this useful>?

@drkane Would be useful to know how well you think it works. The separation is partly based on what you said on a call about maybe keeping the data in status.json small enough that you can read through the file more easily.

[drkane]

@drkane Would be useful to know how well you think it works. The separation is partly based on what you said on a call about maybe keeping the data in status.json small enough that you can read through the file more easily.

For me personally, I'd be using the most detailed version of the data, which is currently coverage.json. Not sure if file size will become a problem - I was thinking aloud when we were talking about it, and actually the file size doesn't get that big.

Where it might make sense to have 3 separate files is that if there is a data.json > status.json > coverage.json separation you can tell what processes have been run on each file, and so what state you can expect it to be in. If it overwrites the file you wouldn't be able to tell what status.json contains at a glance, depending on which script has been run.