GNIP 92: non-spatial structured data as (pre)viewable FAIR datasets

[gannebamm]

GNIP 92 - non-spatial structured data as (pre)viewable FAIR datasets

Overview

We need to store structured non-spatial datasets besides geodata as GeoNode resources. The non-spatial datasets shall provide a simple viewer as a preview and should be able to get used as part of dashboards. The datasets should be findable, accessible and provided in an interoperable way, therefore complying with the FAIR principles.

Proposed By

Florian Hoedt, Thünen-Institute Centre for Information Management

We intend to fund this development as part of an upcoming tender process. This GNIP shall start a discussion about how the developed feature could get upstreamed to the main project.

Assigned to Release

This proposal is for GeoNode 4.0

State

• Under Discussion
• In Progress
• Completed
• Rejected
• Deferred

Motivation

Status Quo: Non-spatial but structured datasets like csv/ Excel files can be uploaded as documents. As document objects, these datasets do inherit the resource base metadata models but can not be viewed in a meaningful way.
As a research institute, our scientists often use PostgreSQL databases and tables to store and structure their research data. Currently, those datasets can not be published in any way in GeoNode. As a research institute, we need to store/register structured non-spatial datasets besides geodata as GeoNode datasets (in the meaning of a v4.0 dataset).

Objective: Implement a new category of RessourceBase for structured non-spatial datasets. Instead of using the GeoServer importer to ingest, e.g. shapefiles, into the PostGIS enabled backend, you should be able to define a connection string to the table to use [?]. The non-spatial datasets shall provide a simple viewer as a preview and should be able to get used as part of dashboards.

Proposal

How to realize the above-mentioned feature is still to be discussed.

As part of an internal discussion, we thought about using PostgREST as an accessible and interoperable tabular data provider. One major aspect is to synchronise authorization mechanisms with the new service. Currently, Django and GeoServer do synchronise their roles via GeoFence. Something similar should be implemented by the tabular service provider. There seem to be options to use JWT as part of the django-rest-framework to grant such authorization as explained here: https://gitter.im/begriffs/postgrest?at=61f06b40742c3d4b21b63843

Apart from using PostgREST as a tabular data provider we also considered the new OGC APIs. These may provide enough functionality for this GNIP. For example the EDR (https://ogcapi.ogc.org/edr/).

Backwards Compatibility

It is not intended to backport this GNIP to 3.x

Future evolution

Explain which could be future evolutions.

Feedback

See discussion below...

Voting

Project Steering Committee:

• Alessio Fabiani: 👍
• Francesco Bartoli:
• Giovanni Allegri:
• Toni Schoenbuchner: 👍
• Florian Hoedt: 👍

Links

Remove unused links below.

• Email Discussion
• Pull Request
• Mail Discussion
• Linked Issue

[afabiani]

@gannebamm my +1 here, the proposal is actually very good. Of course we need to carefully choose how to convert the structured data into a "standard" format that GeoNode can use later on.
It would be nice if you could prepare/provide few samples of possible datasets or provide an idea of the complexity of the structures. You are speaking about Excel documents, but those ones might be very complex. We might need to envisage some hooks/harvesters able to parse and store specific formats.
We had a similar use case for the Afghanistan Risk Data portal. In that case we had to create some brand new data structures and parsers able to ingest very specific Excel files for each hazard type.

[afabiani]

Added the GNIP to the wiki page

[gannebamm]

Thanks @afabiani for adding it to the wiki page!

It would be nice if you could prepare/provide few samples of possible datasets or provide an idea of the complexity of the structures. You are speaking about Excel documents, but those ones might be very complex.

here you go: soil example dataset

You can switch the metadata language to English and use the BZE_LW English Version. The site.xlsx is the spatial dataset we currently upload as a point layer. The other two xlsx files (LABORATORY_DATA, HORIZON_DATA) are examples of non-spatial datasets. I know, in the end, everything is spatial somehow since the lab and horizon datasets do explicitly or implicitly reference a sample site. Nonetheless, we would like to publish those as non-spatial datasets and enable custom applications to fetch those in an accessible and interoperable way through an API. An example of this kind of custom application can be seen at soilgrids. If you click a coordinate you will derive loads of additional data like this:

However, most of our data is already stored in PostgreSQL databases. I know other research institutes also have working databases that could maybe just get integrated. If we use some ORM like sqlalchemy we could even open this up for a more diverse set of SQL-like data providers, as explained here. But maybe that one is out of scope and we should stay true and close to our current stack, which does use PostgreSQL.

I will ask my colleagues to provide some more examples.

[t-book]

My +1. thanks Florian

[giohappy]

@gannebamm this proposal is the natural prosecution of the conceptual change we did from "layers" to "datasets" in GeoNode.
One of the reasons for the change of the name of these entities is exactly to make room for non-spatial datasets, which are not well represented as "layers".

Before starting the discussion about their presentation (web client, API, standards, whatever) I wonder where we imagine storing these datasets. The first option that comes to my mind is geonode_data DB, which is the one employed by Geoserver At the moment GeoNode has no direct connection with that DB, but I was thinking of this since a while. If we made GeoNode "aware" of the geonode_data DB and models we could:

• build more advanced and custom analysis and visualization tools on vector spatial datasets
• expose vector spatial datasets also to the functionality that will be built for non-spatial datasets
• we have a single data store for both non-spatial and vector datasets, even though the latter are directly managed by Geoserver

I know that this goes against the general advice to keep the services models separate, and in theory, we should only rely on OGC standard interfaces to query spatial datasets, but in the case of GeoNode and Geoserver:

• they're services composing a single product, so we have total control over the two and their models
• keeping GeoNode itself tied to the standard interfaces with Geoserver limits the functionality (or make it far more complex and less performant) that could be built

[matthesrieke]

As I had an offline discussion with @gannebamm on the topic, I am sneaking in on the discussion.

However, most of our data is already stored in PostgreSQL databases. I know other research institutes also have working databases that could maybe just get integrated.

I think @gannebamm is having a slightly different workflow in mind (correct me if I am wrong). The data would not be imported into a central data store, but managed as reference to an existing database. I guess this is the most flexible and scalable approach as otherwise you would need to make sure to preserve the structure in the geonode_data, without conflicts across datasets.

On the other hand, this would bring in the requirement of some sort of default structure anyways, if you do not want to implement special visualizers for each dataset. Maybe it could also be a mappable structure, filled out by the user during import.

Maybe also both scenarios (1. existing DB; 2. import into geonode_data) could be covered. The same questions would still require answers.

[gannebamm]

Maybe the https://github.com/rvinzent/django-dynamic-models technology evaluated and used (?) for the SOS integration (contrib module) can help for this feature request. See: https://github.com/GeoNode/geonode-contribs/issues/172

[matthesrieke]

Dear @giohappy, together with @gannebamm @mwallschlaeger we have started to iterate the requirements and the concept behind a non-spatial dataset feature for GeoNode. We have started a small prototype by setting up a Django App / Contrib module. At the moment, uploading data is achieved by providing a CSV file with a sidecar JSON Tabular Data Resource that describes the schema and types of fields.

@gannebamm pointed us to the new geonode-importer module and we were wondering if this would be a good fit for ingesting the data. It looks to be designed in a way that it would allow the addition of custom/new handlers. Do you think it would fit our purpose?

[giohappy]

Dear @matthesrieke sorry for the late reply.

First of all, a GNIP for the new importer is on its way, we want to make it a community module asap. At the moment it's hosted under GeoSolution's own repo.

As you note, the new importer lets you implement specific handlers, and it can assume complete control of the lifecycle of a resource. For example, the handler is in charge of doing any housekeeping when a resource backed by specific data and tables is deleted.
@mattiagiupponi can tell you much more about it since he's the module's author.

So, the primary use case here is to map a GeoNode resource to an external DB. If we generalize this I'd say that this case isn't strictly related to non-spatial datasets. In our vision, a non-spatial dataset could still be served by Geoserver, that way we can benefit from all the services and WFS-based client tools that we already have. They can work for non-spatial data too.
So, follow me, we have two "dimensions" here:

• implement support for non-spatial datasets on top of the existing tools (here we don't care where the data is located)
• implement support for alternative DBs to geonode_data (this can work both for spatial and non-spatial datasets)

IMHO we should agree on the first point first, which is the subject of this GNIP.
I'm a bit concerned with creating new data models and services. We can improve the current ones (always with back compatibility in mind!), but I'd try our best to avoid adding complexity.

[gannebamm]

@giohappy

I am not sure if I understand the two dimensions stated.

on top of the existing tools (here we don't care where the data is located)

We care where the data is located and would like to ingest it into the PostgreSQL backend for later use.

In our vision, a non-spatial dataset could still be served by Geoserver, that way we can benefit from all the services and WFS-based client tools that we already have.

I think because this is not the intended way to use WFS tools like QGIS are likely to fail to understand the non-spatial data served in a WFS. Did anyone test this successfully?

@afabiani @matthesrieke @t-book @francbartoli ->

Maybe we should schedule a talk to discuss this? It is getting quite complex, and I think it would help to dig deep into the pros and cons of the possible approaches and define our needs. Maybe @mattiagiupponi can provide a short intro into the importer and the non-spatial data serving capabilities of GeoServer, and @matthesrieke can describe the prototype he developed to test the approach. In the end, less complexity is always welcome.

If other community developers are interested in coming by, I can host a public meeting. Scheduling this will be rough, though.
What do you think?