Merge pull request #12 from metno/9-update-internal-format-docs

9 update internal format docs

Author: havardf

docs/internal_format.md

@@ -1,40 +1,82 @@
 # Forti's internal data format
 
-Forti uses an internal data format for storing data. This format is used both in the blob storage and by `rawdataforecaster`. It also dictates parts of the contents of the grpc protocol.
-Each area/version consists of one or several separate forecasts grouped by their grid resolution.
+This document is intended for developers who need to read or write Forti’s internal data format directly.
 
-## Data for each area/version
+Forti uses an internal data format for storing data. This format is used both in the blob storage and by `rawdataforecaster`. 
 
-Each area/version has one or several grids (strictly speaking, it is not a grid, but a collection of latitude/longitude coordinates within a particular area). Each grid is supposed to have the same geographic extent, but differs in what resolution the grid points in each grid has.
+## Overview
 
-The data for each grid resolution is described in a uniquely-named subfolder under `/<area>/<version>/` in the blob storage. The data for each resolution is described below.
+Data is split into areas and versions.
+An area refers to to a limited geographic area, typically the domain of a single forecast model. 
+Version numbers are used to identify which forecast for an area is the newest.
+Highest version number is newest.
 
-Each combination of `area` and `version` has accompanying metadata. These data are described in `DatasetMeta` in [the code](../upload/pkg/fortiblob/collector.go). On the blob store, these data are stored in a file called `complete.json`.
+A single rawdataforecaster instance can serve data from several areas, but only one area for a single request. 
+This is done by selecting the area with a grid point which is closest to the requested location.
+It will only ever serve the latest version for each area, as determined by that area's version number.
 
+The data in the single area is expressed as one or more lists of latitude/longitude values with accompanying data for several parameters.
+If there are more than one list of latitudes and longitudes, they are expected to cover the same area, but with different values for latitude and longitude.
+This allows some parameters to have a different resolution than others, even if the cover the same area.
 
-## Format for each grid resolution
+All data for each area/version is placed in a uniquely-named subfolder under `/<area>/<version>/` in the blob storage. 
+The structure of this is described below.
 
-The format consists of four pieces of data:
 
-* forecast values
-* metadata
-* latitudes
-* longitudes
+## Object store layout
 
-### Forecast values
+Under a single area/version in the blob storage, the following layout is expected:
 
-The forecast values are the actual values for the forecast. The data is organized so that all values for each location is stored together, one location after the other. Each value in the data is merely a little-endian encoded `int16`, and the meaning of each one is defined in the metadata.
+* complete.json
+* sub-folders, containing the following objects:
+  * meta.json
+  * data
+  * longitude
+  * latitude
 
-On the blob store, these data are stored in a file called `data`.
+### complete.json
 
-### Metadata
+This contains metadata about the area/version itself. 
+Its format is described in `DatasetMeta` in [the code](../fortiup/pkg/fortiblob/collector.go).
 
-The metadata describes the meaning of the forecast values. The data structure is described in `MetaCollection` in [the code](../upload/pkg/fortiblob/collector.go). There is also [an example](../upload/pkg/fortiblob/collector_test.go) available for how to interpret raw data.
+When uploading data to the object store, this is supposed to be the last file uploaded, as its existence will trigger an update on `rawdataforecaster`.
 
-In the blob store, this is json-encoded in a file called `meta.json`.
+### Sub-folders
 
-### Latitudes and longitudes
+Different data for the same geographic area can have different resolutions.
+This will be expressed as different values for longitude and latitudes.
+For each of these resolutions, a subfolder is made.
 
-In the blob store, latitudes and longitudes exist in two separate files. These define the lat/lon of each forecast point in the data, and each latitude/longitude is binary encoded as a `float32` in the files.
+The name of this sub-folder is expected to be unique for each set of lat/lon lists.
+For example, the name can be equal to the md5 sum of the concatenated latitude and longitude lists.
 
-The index of each lat/lon pair matches the index of the forecast in the `data` file. So, if you want to look up the data for latitude/longitude index `x`, that data starts in the data file at index `(x * meta.LocationCount)`. Of course, if you make a lookup into a file, you must multiply this by 2 to account for int16 taking up two bytes.
+Four files are expected to exist here:
+
+* meta.json
+* longitude
+* latitude
+* data
+
+#### meta.json
+
+This describes the meaning of the forecast values. 
+The data structure is described in `MetaCollection` in [the code](../fortiup/pkg/fortiblob/collector.go). An example of such data can be found in [the same folder](../fortiup/pkg/fortiblob/collector_test.go)
+
+#### longitude and latitude
+
+In the blob store, latitudes and longitudes exist in two separate files. 
+These define the lat/lon of each forecast point in the data, and each latitude/longitude is binary encoded as a `float32` in the files.
+The ordering of the values are not important, as long as the same ordering is used in the logintude, latitude and data files.
+
+#### data
+
+Data contains the actual values for the forecast. 
+It consists of a series of little-endian encoded `ìnt16` values, and their meaning is defined in meta.json.
+
+To look up data for a specific location, you need two things:
+* An index from the latitude and longitude arrays.
+* The length of the relevant data - this is the metadata's number_of_points value.
+
+Multiply the two values to get the starting index.
+You can then use the metadata to interpret the relavant values.
+[The example](../fortiup/pkg/fortiblob/collector_test.go) shows how to do the interpretation.

fortiup/cmd/mktestdata/main.go

@@ -101,11 +101,11 @@ func writeMeta(path, parameters string) (*fortiblob.MetaCollection, error) {
 		pm := fortiblob.ParameterMeta{
 			Units:     "u_" + match[1],
 			Times:     times,
-			SliceFrom: meta.LocationCount,
+			SliceFrom: meta.NumberOfPoints,
 		}
-		meta.LocationCount += len(times)
+		meta.NumberOfPoints += len(times)
 		if len(times) == 0 {
-			meta.LocationCount++
+			meta.NumberOfPoints++
 		}
 		meta.Parameters[match[1]] = pm
 	}
@@ -124,20 +124,20 @@ func writeMeta(path, parameters string) (*fortiblob.MetaCollection, error) {
 }
 
 func writeData(path string, meta *fortiblob.MetaCollection, lat, lon []float32) error {
-	totalSize := len(lat) * meta.LocationCount
+	totalSize := len(lat) * meta.NumberOfPoints
 
 	data := make([]int16, totalSize)
 
 	for i := range lat {
 		for _, pMeta := range meta.Parameters {
 			if len(pMeta.Times) == 0 {
 				value := (i * 100) + pMeta.SliceFrom
-				idx := (i * meta.LocationCount) + pMeta.SliceFrom
+				idx := (i * meta.NumberOfPoints) + pMeta.SliceFrom
 				data[idx] = int16(value) * 10
 			}
 			for t := range pMeta.Times {
 				value := (i * 100) + (t * 10) + pMeta.SliceFrom
-				idx := (i * meta.LocationCount) + pMeta.SliceFrom + t
+				idx := (i * meta.NumberOfPoints) + pMeta.SliceFrom + t
 				data[idx] = int16(value) * 10
 			}
 		}

fortiup/internal/blob2blob/collector/grid/simpledatagroup.go

@@ -58,7 +58,7 @@ func collectSimpleDataGroup(ctx context.Context, out io.Writer, source *modelpro
 
 	return &fortiblob.MetaCollection{
 		Parameters:    pMeta,
-		LocationCount: elements,
+		NumberOfPoints: elements,
 	}, nil
 }
 

fortiup/internal/nc/store/collect/collect.go

@@ -28,7 +28,7 @@ func Collect(ctx context.Context, variables []*netcdf.Variable, out io.Writer) (
 func getMetaCollection(ctx context.Context, variables []*netcdf.Variable) (*fortiblob.MetaCollection, error) {
 	ret := fortiblob.MetaCollection{
 		Parameters:    make(map[string]fortiblob.ParameterMeta),
-		LocationCount: 0,
+		NumberOfPoints: 0,
 	}
 
 	for _, v := range variables {
@@ -56,12 +56,12 @@ func getMetaCollection(ctx context.Context, variables []*netcdf.Variable) (*fort
 		meta := fortiblob.ParameterMeta{
 			Units:       units,
 			Times:       times,
-			SliceFrom:   ret.LocationCount,
+			SliceFrom:   ret.NumberOfPoints,
 			ScaleFactor: getScaleFactor(v),
 		}
 
 		ret.Parameters[v.Name] = meta
-		ret.LocationCount += len(times)
+		ret.NumberOfPoints += len(times)
 	}
 
 	return &ret, nil

fortiup/internal/upload/upload_test.go

@@ -49,7 +49,7 @@ func makeTestingClient() fortiblob.Client {
 				SliceFrom: 3,
 			},
 		},
-		LocationCount: 4,
+		NumberOfPoints: 4,
 	}
 	if err := u.SetGridMeta(ctx, &paMeta, area, version, gridid); err != nil {
 		panic(err)

fortiup/pkg/fortiblob/collector.go

@@ -31,20 +31,20 @@ type GeographicArea struct {
 	SRS string `json:"srs"`
 }
 
-// MetaCollection is defines the meaning of the forecast data values. It
+// MetaCollection defines the meaning of the forecast data values. It
 // refers to a set of indexable data consisting of int16 values, where index 0
 // contains the first piece of data for a particular location.
 type MetaCollection struct {
 
 	// Parameters maps parameter names to metadata about that parameter.
 	Parameters map[string]ParameterMeta `json:"parameters"`
 
-	// LocationCount is the total number of forecast values for a single
+	// NumberOfPoints is the total number of forecast values for a single
 	// location. It is equal to the sum of the length of all times slices
 	// under Parameters, and is therefore redundant.
 	// We keep it as a separate value to avoid having to calculate it over
 	// and over again upon usage.
-	LocationCount int `json:"number_of_points"`
+	NumberOfPoints int `json:"number_of_points"`
 }
 
 // ParameterMeta contains metadata about a forecast for a single parameter

fortiup/pkg/fortiblob/collector_test.go

@@ -1,45 +1,43 @@
 package fortiblob
 
 import (
+	"encoding/json"
 	"fmt"
-	"time"
 )
 
 // ExampleMetaCollection shows how to use MetaCollection to interpret raw data.
 func ExampleMetaCollection() {
 	sampleData := []int16{
 		142, 139, 92, 0, 1,
 	}
-	meta := MetaCollection{
-		Parameters: map[string]ParameterMeta{
-			"temperature": {
-				Units: "c",
-				Times: []time.Time{
-					time.Date(2021, 3, 3, 0, 0, 0, 0, time.UTC),
-					time.Date(2021, 3, 3, 1, 0, 0, 0, time.UTC),
-				},
-				SliceFrom:   0,
-				ScaleFactor: 0.1,
-			},
-			"altitude": {
-				Units: "m",
-				Times: []time.Time{
-					{},
-				},
-				SliceFrom:   2,
-				ScaleFactor: 1,
-			},
-			"precipitation": {
-				Units: "kg/m²",
-				Times: []time.Time{
-					time.Date(2021, 3, 3, 0, 0, 0, 0, time.UTC),
-					time.Date(2021, 3, 3, 1, 0, 0, 0, time.UTC),
-				},
-				SliceFrom:   3,
-				ScaleFactor: 0.1,
-			},
-		},
-		LocationCount: len(sampleData),
+	metaJSON := `{
+  "parameters": {
+    "temperature": {
+      "units": "c",
+      "times": ["2021-03-03T00:00:00Z", "2021-03-03T01:00:00Z"],
+      "slice_from": 0,
+      "scale_factor": 0.1
+    },
+    "altitude": {
+      "units": "m",
+      "times": ["0001-01-01T00:00:00Z"],
+      "slice_from": 2,
+      "scale_factor": 1
+    },
+    "precipitation": {
+      "units": "kg/m²",
+      "times": ["2021-03-03T00:00:00Z", "2021-03-03T01:00:00Z"],
+      "slice_from": 3,
+      "scale_factor": 0.1
+    }
+  },
+  "number_of_points": 5
+}
+`
+
+	var meta MetaCollection
+	if err := json.Unmarshal([]byte(metaJSON), &meta); err != nil {
+		panic(err)
 	}
 
 	for parameter, meta := range meta.Parameters {

fortiup/pkg/fortiblob/sampleblob/sampleblob.go

@@ -82,7 +82,7 @@ func (c *client) GetGridMeta(ctx context.Context, d *fortiblob.DatasetMeta, grid
 				ScaleFactor: 0.1,
 			},
 		},
-		LocationCount: 5,
+		NumberOfPoints: 5,
 	}, nil
 }
 

rawdataforecaster/internal/server/forecast/dataset/values/blob/blob.go

@@ -44,20 +44,20 @@ func (r *Reader) Read(idx int) (*values.LocationDataCollection, error) {
 	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
 	defer cancel()
 
-	reader, err := r.source.GetDataRange(ctx, &r.datasetMeta, r.grid, r.gridMeta.LocationCount*idx*2, r.gridMeta.LocationCount*2)
+	reader, err := r.source.GetDataRange(ctx, &r.datasetMeta, r.grid, r.gridMeta.NumberOfPoints*idx*2, r.gridMeta.NumberOfPoints*2)
 	if err != nil {
 		return nil, err
 	}
 	defer reader.Close()
 
-	buffer := make([]int16, r.gridMeta.LocationCount)
+	buffer := make([]int16, r.gridMeta.NumberOfPoints)
 	if err := binary.Read(reader, binary.LittleEndian, &buffer); err != nil {
 		return nil, err
 	}
 
 	ret := values.LocationDataCollection{
 		ParameterMeta: r.gridMeta.Parameters,
-		Data:          make([]float32, r.gridMeta.LocationCount),
+		Data:          make([]float32, r.gridMeta.NumberOfPoints),
 	}
 
 	for _, meta := range r.gridMeta.Parameters {

rawdataforecaster/internal/server/forecast/dataset/values/memory/memory_test.go

@@ -55,7 +55,7 @@ func getSampleReader() *MemoryReader {
 				ScaleFactor: 0.1,
 			},
 		},
-		LocationCount: 3,
+		NumberOfPoints: 3,
 	}
 
 	mad := allocate(3 * 4)