Common cloud-based time series data format

[huhabla]

In openEO we handle large time series data of satellite scenes. IMHO we need a data type to describe such data that reflects the cloud based storage approach. This data type should be a s simple but sophisticated as possible, so that time series data that was generated by services or data searches describe the generated/stored data sufficiently for further cloud processing. I would suggest a simple text/json based description of time series or image collections that includes the url's that points to the actual data (jp2000, geotiff, XML in an object storage) and provides additional metadata like time stamps, bounding box and others.

For example:
A search for Sentinel2A scenes can return this data type that includes all links to the scenes, bands and metadata. The result of the search can directly be used in processes that compute the NDVI for a time series of Sentinel2A scenes. This process will return the same data type to describe the resulting image collection and their links in a cloud storage. The result of the NDVI computation can then be used in other services that process time series data an accept the suggest time series data type as input.

[Mec-iS]

Geospatial standards do not address time format usually. I think that a nice point to start from may be GeoJSON-events

[GreatEmerald]

I'm not sure if the definition of "time series" is clear enough in this case... What you're describing sounds like a collection to me. We already have a sort of definition of "time series" as a JSON response that includes timestamps and values for a single pixel (see "Getting the timeseries" in https://github.com/Open-EO/openeo-python-client/blob/master/examples/notebooks/Compositing.ipynb), though it's not an official definition yet as per #46.

[m-mohr]

I agree with @GreatEmerald. For me the description sounds like a collection, too. But the collections are handled like data cubes in openEO and working on individual files as suggested in the first post doesn't seem suitable here. @huhabla Could you please explain what exactly you meant?
If it's just about linking to the data files, then we could simply use STAC, which we plan to use anyway to some extent. Not all back-ends might want to list their data files though and this should be optional.

[huhabla]

I am not referring to the data formats that is used in the openeo backends.
What i meant is a data exchange format that represents an image collection with time stamps for each image. The best case would be a cloud-optimized solution.

For example howto download the the image collection of a NDVI computation? The input data of the computation was a time series of Landsat images (image collection) with different time stamps and spatial extents, in the worst case scattered around the globe. Should this be handled as a data cube using netcdf? What is the best export format for this? I would suggest a cloud compatible format like CO-GeoTiff for the images. These images are stored in a directory that contains the time stamps of the GeoTiff files in a text file. This directory can be compressed. The key is that HTTP GET requests can be used to download parts of the GeoTiff files as well as parts of the archive when hosted on a HTTP server or in an object storage.

My point is: What to do in case the user don't want to store the resulting image collections in the database of the backend, but in on a HTTP server or an object storage and want to use it in further processing? Then we should use a data format, that can be directly accessed in the openeo backends, using HTTP GET range requests or virtual file systems.

GDAL supports for this case a virtual file system:
http://www.gdal.org/gdal_virtual_file_systems.html#gdal_virtual_file_systems_vsis3