-
Notifications
You must be signed in to change notification settings - Fork 202
Inventory Beta Re‐design
The inventory application managed by data.gov helps agency/bureau/department data managers inventory and catalog data assets through metadata entry. This application has the following features/attributes:
- Agency/organization silos for metadata management and managed teams of users isolated from each other.
- Metadata entry based on the DCAT-US1.1 metadata standard, created by OMB for open data.
- DCAT-US1.1 and 3.0 export by organization (along with error reporting).
- Data storage by file (and API access if compatible file type such as CSV), and made publicly available.
The current setup utilizes a modified CKAN implementation to remove SOLR dependency. This implementation has the following weaknesses:
- UI revamp/rewrite required to be 508 compliant
- Forked code requires custom management of libraries and updates, no expectation this will ever merge with CKAN upstream
- No public testing or usage available, either gov user with PIV card and confirmed with agency/organization or no access to the tool will be granted.
- Custom React app data entry form based on 1.1 and will need complete re-write for 3.0
Given that the CKAN metadata model is not very compatible with the nested object classifications of DCAT-US3.0, we are recommending moving away from CKAN to a custom solution. This solution will remove the need for login and users (but will maintain the option), and will have an import/export system to start from a current DCAT-US3.0 catalog (or any DCAT-US3.0 object). The objects will be stored in browser memory or in a back end if the user is logged in. The classes and objects will be able to be created/defined for re-use by metadata providers, and descriptions of the fields will be available from the definition of the schema file. This will keep agencies consistent within their catalogs, while allowing flexibility (within schema definitions) across different organizations.
We want to be clear about the current features of inventory, and what will and won't be supported in the new version.
| Feature | Inventory Support | Planned Beta MVP Plan | Planned Long Term Inventory Plan |
|---|---|---|---|
| DCAT-US1.1 dataset form entry | Yes | No | No |
| DCAT-US3.0 full feature catalog form entry (entry by class and re-use) | No | Yes | Yes |
| Government data manager user login | Yes | Yes | Yes |
| Metadata draft state (in process for entry, not exported) | Yes | Yes | Yes |
| Public usage of inventory - creating and/or ingesting DCAT-US3.0 catalogs - editing DCAT-US3.0 catalogs (saved in browser memory) - export browser saved DCAT-US3.0 catalog (see below) |
No | No | Yes |
| Data file storage and public retrieval (inventory can host datasets) | Yes | Yes | Yes |
| Direct inventory metadata available and harvest possible from harvest.data.gov (inventory can host metadata files) | No | No | Yes |
| Export of DCAT-US metadata (with success and error reporting) | Yes | Yes | Yes |
| Have "show what this will look like in catalog" button for dataset (and datasetSeries and dataService) | No | No | Yes |
| Store versions of classes/documents with editor information | Maybe? | Yes | Yes |
Users have access to catalogs (read, read/write edit permissions, or admin permissions to read and write as well as change permissions). Catalogs created by users can be shared with other users with those level of permissions.
Since catalogs can be embedded in other catalogs (see documentation), catalogs can be shared with catalogs. Catalogs can only be shared if they are no longer in draft state. Classes/objects in draft state are not exported in the catalog.
Since users can create their own catalogs and mark them as draft or live, there is no reason to lock accounts. With PIV card authentication, there is no further level of identity management that the data.gov team can make to make sure a user is who they say they are.
We will maintain historical object tracking along with user edit information. All changes will be available for auditability.
We will build an audit process highlighting the following potential issues:
- All catalogs with no admin (no longer accessible/manageable)
- Live catalog (no longer in draft) URL scanning (for malicious URL content within catalogs)
- Deeper audit for cross-referenced harvest list with direct inventory harvest (future feature, future audit)