Skip to content

exporters.md

Latest commit

 

History

History
174 lines (113 loc) · 7.98 KB

exporters.md

File metadata and controls

174 lines (113 loc) · 7.98 KB

BAS Assets Tracking Service - Exporters

Exporters are interfaces between this application and clients and tools that use asset location information.

Exporters make data on tracked assets and their positions available in a format and over a protocol other applications accept. Some exporters are more generic and used by other, more specific, exporters to avoid repeating work.

Exporter classes

Exporters inherit from an abstract assets_tracking_service.exporters.base_exporter.Exporter base class which defines a public interface. These public methods typically call private methods to:

  • select required data
  • format it for use
  • output and host data in a suitable file format or push directly into other applications

Selecting data

Exporters MAY use Database views to select relevant data.

Exporters MAY depend on the outputs of other exporters (and so require these to be enabled and suitably configured).

Exporters manager

A central assets_tracking_service.exporters.exporters_manager.ExportersManager class exports data for all enabled Exporters using their common public interface.

Disabling exporters

See the Configuration documentation for options to disable exporters.

Note

An error will occur if an exporter that another (enabled) exporter depends upon is disabled.

Exporter Credentials

See Infrastructure documentation for credentials required by these exporters.

Available exporters

ArcGIS

ArcGIS is a geospatial data hosting, publishing and analysis platform used by BAS.

The ArcGIS exporter publishes each Layer to ArcGIS Online using the BAS Esri Utilities library as a:

Tip

These exports are intended for use by end-users and downstream services (such as the Embedded Maps Service 🛡️).

Feature Layer items are associated with an underlying Feature Service, which for this project is seeded from a GeoJSON item, updated by this application. Separate layers and services are hosted in the ArcGIS Online subscription for each Environment.

A folder within the AGOL user account used for publishing, and a group within each ArcGIS subscription, are created automatically to organise content:

Note

This exporter is opinionated as it always:

  • assumes all Layers are vector (and can/should be published as Feature Layers)
  • creates a set of GeoJSON, Feature Service/Layer and OGC API Features Layer items
  • stores items it creates in a prj-assets-tracking-service folder
  • adds items to an Assets Tracking Service group, except GeoJSON items (as they are an implementation detail)
  • shares items publicly, except GeoJSON items (which are private as they are an implementation detail)
  • sets ArcGIS item metadata from, and links to, an associated BAS data catalogue record

ArcGIS schemas

Feature layer's schemas are based on the source view for each Layer.

An additional ObjectID attribute is automatically included in each schema by ArcGIS. This attribute MUST be considered an implementation detail and values MUST be hidden from end-users to the extent possible.

Warning

ObjectID values MUST NOT be considered stable and MAY change or reused/reassigned without warning.

ArcGIS permissions

Content is shared publicly, except for underlying GeoJSON items, as they are implementation detail that should not be relied upon.

ArcGIS requirements

Required exporters (MUST be enabled):

Within each ArcGIS Online organisation a user for publishing content MUST be provisioned, which:

  • MUST be assigned the creator user type
  • MUST be assigned the administrator role (to create restricted groups)
  • MUST NOT have multi-factor authentication enabled (to allow non-interactive authentication)
  • SHOULD be the conventional app_automation_bas / app_automation_bas_test shared automation user

ArcGIS configuration options

Required options:

  • EXPORTER_ARCGIS_USERNAME: username of publishing user
  • EXPORTER_ARCGIS_PASSWORD: password of publishing user
  • EXPORTER_ARCGIS_BASE_ENDPOINT_PORTAL: base endpoint for the ArcGIS portal instance
  • EXPORTER_ARCGIS_BASE_ENDPOINT_SERVER: base endpoint for the ArcGIS hosting server associated with the portal
  • EXPORTER_ARCGIS_FOLDER_NAME: folder exporter will publish content within
  • EXPORTER_ARCGIS_GROUP_INFO: information needed to create the group published content will be shared with

The base endpoints are currently assumed to be ArcGIS Online organisations which use the form:

  • https://<org>.maps.arcgis.com
  • https://services<hive>.arcgis.com/<org>/arcgis/

See the Infrastructure documentation for values to use for BAS AGOL organisations.

The folder name and group information are hard-coded values and do not need to be set. File references within the group information are relative to: src/assets_tracking_service/resources/arcgis_group/

BAS Data Catalogue

The BAS Data Catalogue is data discovery tool used to find, evaluate and access datasets, products, services and other resources produced, managed or used by the British Antarctic Survey and UK Polar Data Centre.

This exporter creates and maintains metadata records for each Layer using an associated Record within a Collection.

Data Catalogue collection

Records are grouped by a collection record representing all outputs from this project.

See the README for more information.

Data Catalogue metadata standards

Records created by this exporter use the BAS ISO 19115 JSON schema, maintained as part of the BAS Metadata Library.

Records additionally conform to the BAS MAGIC Discovery Profile (v1).

Records additionally conform to the BAS MAGIC Administration (v1).

Data Catalogue metadata sources

Metadata records combine information from multiple sources:

  • Records provide static values for properties such as title, abstract, etc.
  • Layers, managed by the ArcGIS exporter, define distribution options and extents

Data Catalogue publishing

The Non-Interactive Data Catalogue publishing workflow is used to publish records exported to the directory specified by the EXPORTER_DATA_CATALOGUE_OUTPUT_PATH config option.

Data Catalogue configuration options

Required options:

  • EXPORTER_DATA_CATALOGUE_OUTPUT_PATH:
    • path to the directory that will contain catalogue records for publishing
    • the application will try to create any missing parent directories to this directory if needed
    • e.g. ./exports
  • EXPORTER_DATA_CATALOGUE_ADMIN_METADATA_ENCRYPTION_KEY_PRIVATE
    • JSON encoded JSON Web Key (JWK) for encrypting administrative metadata
  • EXPORTER_DATA_CATALOGUE_ADMIN_METADATA_SIGNING_KEY_PRIVATE
    • JSON encoded JSON Web Key (JWK) for signing administrative metadata