Skip to content

CHANGELOG.md

Latest commit

 

History

History
945 lines (642 loc) · 63.9 KB

CHANGELOG.md

File metadata and controls

945 lines (642 loc) · 63.9 KB
Raw

Changelog

All notable changes to this project will be documented in this file.

This project relies on continuous integration for new features. So we do not yet have explicitly versioned releases. Releases are simply built continuously, automatically tested, deployed to a development environment and then to production.

Note that the openEO API provides a way to support stable and unstable versions in the same implementation: https://openeo.org/documentation/1.0/developers/api/reference.html#operation/connect

If needed, feature flags are used to allow testing unstable features in development/production, without compromising stable operations.

0.73.0

  • Starting batch jobs is made more lenient to support processes not known by the web application.
  • Add soft_error feature on missing products to S1BackscatterOrfeoV2 (#1719)

0.72.0

  • aspect and slope processes now return radians instead of degrees
  • YARN batch jobs: configure spark.driver.maxResultSize dynamically based on driver memory (max of 5g and driver memory) instead of hardcoded 5g (#631)
  • Better support for spatial filtering with geometries that have internal overlap (#1645)
  • Serialize STAC ItemCollection from load_stac (#1618)
  • Batch job results: STAC item geometry should match asset geometry (#756)
  • Support experimental corsa_compress_v2 and corsa_decompress_v2 processes (Open-EO/openeo-geotrellis-extensions#702)
  • add SimpleEtlApiConfig.from_env() helper
  • Global extent in post-dry-run: at least apply align from resample_spatial when staying withing CRS (#1662)
  • Kubernetes batch jobs: support driver-corerequest job option for fractional CPU allocation to the driver (e.g. "500m"), using the coreRequest field in the Spark operator spec (#1658)
  • ETL API: support logging added value for synchronous requests (#1436)
  • load_collection through load_stac: add support for applying raster:scale and raster:offset through a feature flag in collection metadata (#1666)
  • Global extent in post-dry-run: always apply align from resample_spatial, regardless of correspondence between the CRS'es of the requested spatial extent and requested resampling (#1662)
  • Add job option stac_api_filter_by_geometry to disable fine-grained geometry based filtering in STAC API queries (#1689)
  • Support loading layer catalog from GZIP'ed JSON file or ZIP of JSON files (#1691)
  • load_stac: avoid unnecessary (and sometimes brittle) normalization of STAC Item self URL (#1702)
  • load_stac: support deduplication based on "properties" extracted from item id (eu-cdse/openeo-cdse-infra#1127)

0.71.0

  • sar_backscatter: avoid segmentation fault on Sentinel-1 products with missing parts (#1479)
  • Ensure counter-clockwise bbox geometries in STAC metadata (#1384)
  • "stac-version" job option is not considered experimental; a value of "1.1" also generates a derived_from document (#1488)
  • load_stac based load_collection: support granule_metadata asset in SENTINEL2_L2A for azimuth/zenith angle bands (eu-cdse/openeo-cdse-infra#804)
  • run_udf now supports running CWL scripts. (#1476)
  • new property_filter_adaptations feature flag (under load_stac_feature_flags) for load_stac powered collections to inject a property filtering adapter that translates legacy (e.g. OpenSearch) property filters to STAC property filters (eu-cdse/openeo-cdse-infra#803, eu-cdse/openeo-cdse-infra#804)
  • Guard STAC metadata parsing against invalid "item_assets" usage (Open-EO/openeo-python-client#853)
  • load_collection through load_stac: support band name aliases/variants (eu-cdse/openeo-cdse-infra#803, eu-cdse/openeo-cdse-infra#804)
  • load_collection through load_stac: new apply_sentinel2_reflectance_offset feature flag (eu-cdse/openeo-cdse-infra#803, eu-cdse/openeo-cdse-infra#804)
  • StacApiWorkspace: do not require export-workspace-enable-merge (#1030)
  • Support latest pystac (currently 1.14.3) (#1496)
  • Support property filtering in load_stac.construct_item_collection in post-dry-run phase (#1530)
  • load_collection through load_stac: fix resolution selection for low-res/non-UTM cases with aliases (#1539)
  • load_stac harden STAC "band" metadata parsing when "name" field is missing (Open-EO/openeo-python-client#860)
  • load_stac: increase default STAC API per page limit to 100 (#1150)
  • Fix ZeroDivisionError from dry-run determine_global_extent caused by invalid proj:transform metadata (#1557)
  • Align load_stac cube's temporal extent better with load_collection behavior (#1578)
  • Improve load_stac bbox filtering across anti-meridian (#1568)
  • load_stac: avoid handling irrelevant items (with corrupt bbox) returned by CDSE STAC API (#1592)
  • load_stac: do property filtering in STAC API queries without properties. prefix by default now (#1584, stac-api-extensions/filter#31)
  • STAC API usage (from load_stac or STAC powered load_collection) with bounding box across the antimeridian: split and merge query for each side to workaround broken antimeridian handling of some STAC API implementations (#1568)
  • Added initial support for /collections/{collectionId}/queryables for STAC based collections (#536)
  • ObjectStorageWorkspace: treat AccessDenied error as new Collection (Open-EO/openeo-geotrellis-extensions#590)
  • ObjectStorageWorkspace: merge argument can be a plain (not nested) file (#1257)
  • Further improve detection of wrong proj:transform metadata (#1570)
  • load_stac: avoid unbounded iteration through STAC API response pages (#1591)
  • load_stac: support sparse spatial filtering (from aggregate_spatial or filter_spatial geometries) (#1225)
  • Support loading Sentinel-3 OLCI L1B from file (eu-cdse/openeo-cdse-infra#644)
  • Support loading Sentinel-3 SLSTR L1B from file (eu-cdse/openeo-cdse-infra#644)

0.70.0

  • load_stac: remove load_stac_apply_lcfm_improvements feature flag and make it the standard behavior (#1306)
  • Experimental support for dropping derived_from links on the STAC Collection level in favor of a link, to a dedicated document with input features, on the item level (#1278)
  • sar_backscatter: under specific circumstances, data was cropped during processing, leading to missing outputs (#1403)
  • Improve vector cube support in export_workspace (#1404)
  • load_stac: support detection of projection metadata from asset "bands" (#1391)
  • load_stac: support item deduplication (for now with opt-in feature flag "deduplicate_items") (#1361)
  • load_stac: added cellsize feature flag. It will be used as fallback when no pixel size can be derived from the source assets.
  • load_stac: the cellsize feature flag is automatically taken from the layercatalog when available, if not set explicitly in the process graph.
  • Initial implementation of new "post-dry-run" hook to improve "global extent and alignment" computation for load_collection (#1299)
  • Remove udf_python_dependencies_folder_path from PYTHONPATH in batch jobs (k8s context) (eu-cdse/openeo-cdse-infra#842)
  • Check UDF code for managed UDF dependencies before doing (possibly) more expensive IO/S3 operations (eu-cdse/openeo-cdse-infra#842)
  • post-dry-run: initial support for global extent alignment with load_stac (#1299, openeo-python-driver#406)
  • load_stac: added asset_id_to_bands_map feature flag in order to support collections that do not contain band name information (e.g. COPERNICUS_30 on CDSE).
  • Start using "global_extent" calculated from post-dry-run phase when possible (#1299, openeo-python-driver#406)
  • Handle "global_extent" calculation from STAC based load_collection through load_stac logic (#1299, openeo-python-driver#406)
  • Support experimental corsa_compress and corsa_decompress processes (Open-EO/openeo-geotrellis-extensions#563, Open-EO/openeo-geotrellis-extensions#577)

0.69.0

  • Added new API UdfRuntimeImageRepository for centralized configuration and handling of container images and corresponding UDF runtimes (#1279, Open-EO/openeo-python-driver#415, eu-cdse/openeo-cdse-infra#721)
  • Support overriding default ETL source ID from K8s app label (eu-cdse/openeo-cdse-infra#612)
  • Add S3StacIO and support custom StacIO in openeogeotrellis.load_stac.load_stac (#1133)
  • Drop override to "python38" image for batch jobs with UDFs (eu-cdse/openeo-cdse-infra#763, eu-cdse/openeo-cdse-infra#169, #1387)
  • Port STAC Collection/Catalog part of load_stac to new (experimental) ItemCollection API (#1307)
  • run_udf inspect calls in apply_metadatadid not show up in logs (#1394)
  • Port STAC API part of load_stac to new (experimental) ItemCollection API (#1307)

0.68.0

  • Experimental support for unified asset keys in job results STAC items by means of the "stac-version-experimental" job option (#1111, Open-EO/openeo-geotrellis-extensions#402)
  • Avoid workaround with EJR to obtain job results metadata in the context of a failover (#1255)
  • Avoid 413 "Payload Too Large" response from EJR upon job results metadata update (#1200)
  • Disable async_task by default (eu-cdse/openeo-cdse-infra#387)
  • export_workspace: add md5 and mtime metadata to assets exported as objects (#1318)
  • Avoid in-memory credentials cache conflict in IPA tooling (eu-cdse/openeo-cdse-infra#660)
  • resample_cube_spatial: support resampling cube without time dimension, where target cube has a time dimension
  • Log warning when requesting bands in unexpected order for temporal NetCDF STAC catalogs. (#1153)
  • Expose get_oidc_access_token helper method for use with StacApiWorkspace (eu-cdse/openeo-cdse-infra#633)
  • Add integrations.vault.VaultClient with "cert" auth support (eu-cdse/openeo-cdse-infra#671)
  • Port single STAC Item part of load_stac to new (experimental) ItemCollection API (#1307)
  • Port "own job" part of load_stac to new (experimental) ItemCollection API (#1307)
  • Avoid ZeroDivisionError with resample_spatial with resolution 0 for UTM layers. (openeo-geotrellis-extensions#506)
  • Less often used Spark metrics for batch jobs are now disabled by default. They are enabled when log-level is set to 'debug'.
  • filter_temporal should also work in half-open fashion when applied after load_collection/load_stac with non-empty temporal_extent (Open-EO/openeo-geotrellis-extensions#498)
  • load_stac: ignore empty roles listing for assets to decide if asset is a "band asset" (1356)
  • Make osgeo.gdal more clearly an optional Eliminate osgeo.gdal as runtime dependency (#1363, eu-cdse/openeo-cdse-infra#718, eu-cdse/openeo-cdse-infra#733)
  • Support returning job result STAC items in addition to assets (#1111)
  • apply_neighborhood: allow minimum X/Y size of 4 instead of 32

0.67.0

  • Configurable usage of async_task (eu-cdse/openeo-cdse-infra#387)
  • Add job option "omit-derived-from-links" to omit "derived_from" links in batch job results metadata (ESA-WEED-project/eo_processing#175)
  • Better freeIPA configurability for proxy user lookup (#1261, eu-cdse/openeo-cdse-infra#626)
  • load_stac: temporal filtering of STAC Items: prefer more descriptive start_datetime+end_datetime over nominal datetime if possible (#1293)
  • Restore proper bbox and geometry for STAC items with netCDF assets (#1294)
  • load_stac: drop (unused) support for eo:bands with integer indices (#619)

0.66.0

  • 'Batch job options' are now advertised following the openEO processing parameters extension (Open-EO/openeo-python-driver#307)
  • use region specific config for workspace actions
  • apply_neighborhood/apply_dimension: support changing band names via apply_metadata (#1155)
  • StacApiWorkspace: support arbitrary paths in merge; the last part of a path becomes the collection ID in the STAC API (#1074)
  • Fix compatibility with Shapely 2 (#1161)
  • Change default use_zk_job_registry config to False (#632, #863, #1165)
  • Add timeout to metadata request on "stac" data sources in layer catalog (eu-cdse/openeo-cdse-infra#525)
  • Improve detection of apply_metadata function in UDF code (related to #1182)
  • Fix require_application_id filter of list_active_jobs in job tracker (#1197)
  • Remove outdated and defunct ZooKeeper "cleaner" scripts and code paths (#1219, #1123)
  • Move integrations.s3 to Python driver #1195
  • Add config ejr_preserialize_process to enable process graph pre-serialization when storing new jobs in EJR (#1232)
  • load_stac: start supporting STAC 1.1 style "common" bands" metadata (in addition to legacy "eo:bands") (#1015)
  • load_stac: improve resilience of STAC API item search (#1096)
  • Harden IPA lookup of proxy user in YARN batch job submit code path (#1261)
  • Make sure presigned URLs against S3 proxy also allow HEAD requests #1265
  • Work around 413 "Payload Too Large" response from EJR upon job results metadata update (#1200)
  • Basic support for batch jobs without a job registry or job tracker (#1005)
  • Fix thresholds in linear_scale_range to trigger type casting (#1275)
  • load_stac: fully switch to _StacMetadataParser from openeo.metadata, which has wider STAC (1.1) support (#1015, #1138)
  • load_ml_model: switch to group openeo_results for ml_models folder (eu-cdse/openeo-cdse-infra#609)
  • resample_cube_spatial avoid going out of memory when resampling from lower to much higher resolution (Open-EO/openeo-geotrellis-extensions#478)

0.65.0

  • sar_backscatter: soft-errors can now be a fraction, allowing to tolerate a certain failure percentage provided as a number between 0 and 1. (#443)
  • save_result: write GeoTiff assets with valid tile size; override with tile_size format option. (#1083)
  • sar_backscatter support: automatically use backend-specific coefficient default even if there is no explicit sar_backscatter in the process graph (Open-EO/openeo-python-driver#376)
  • Add capabilities_extras config to easily extend capabilities document (Open-EO/openeo-python-driver#384)
  • Add CalrissianS3Result.generate_presigned_url() (#937)
  • Restore COG layout for GTiff output format (Open-EO/openeo-geotrellis-extensions#393)
  • Add CalrissianS3Result.download() (#937)
  • Support additional Sentinel 3 collections (eu-cdse/openeo-cdse-infra#380)
  • Add CalrissianS3Result.generate_public_url() (#937)
  • Replace ConfigParams.layer_catalog_metadata_files with GpsBackendConfig.layer_catalog_files (#285, #1084)
  • StacApiWorkspace: log body of error response from STAC API for better root cause analysis (#1116)
  • export_workspace: add "derived_from" links to STAC Collection (#1050)
  • Calrissian integration: avoid unnecessary pulls of alpine image ([#1132]Open-EO#1132)
  • Calrissian integration: refactor config to a CalrissianConfig sub-config (#1009)
  • save_result: support zarr format (experimental)
  • save_result: allow non-string values in GTiff file_metadata (#1142)
  • Add udp_registry_zookeeper_client_reuse config for KazooClient reuse in ZooKeeperUserDefinedProcessRepository (#1037)
  • GpsBackendConfig: be more forgiving about unknown config keys to better support use cases that involve backward/forward incompatible configurations (Open-EO/openeo-python-driver#322)
  • Improved API alignment between DoubleJobRegistry and JobRegistryInterface/ElasticJobRegistry (#863, #1123)
  • Add use_new_feature_extent_intersection_2 option to load_collection: To use new intersection code to work with products crossing the antimeridian. (#1072)
  • Add add_bands_statistics format option to include the statistics in asset metadata of netCDF bands (Open-EO/openeo-geotrellis-extensions#406)

0.64.1

  • load_custom_processes: allow specifying path directly

0.63.0

  • Add bbox and geometry to exported STAC items pertaining to GeoTiff assets of a spatial data cube (eu-cdse/openeo-cdse-infra#418)
  • Support UDF dependency extraction from remote process definitions (URL based UDPs) (#1063)
  • StacApiWorkspace: improve STAC requests resilience (#1073)
  • StacApiWorkspace: reject unsupported characters in merge (eu-cdse/openeo-cdse-infra#415)
  • load_stac: fix delay in driver due to expensive item/asset processing (#1081)

0.62.0

  • load_stac: improve STAC requests resilience (#818)
  • Extract demo CWL content to package resources (#1057)
  • Disable EJR health check from batch job context (#1066)
  • load_stac: support empty data cubes (#1049)
  • Remove proof-of-concept CWL processes (now in openeo-geotrellis-kubernetes) from generic openeo-geopyspark-driver (#1057/#1038)

0.61.0

  • python-memory: make job option available on YARN, add a default config
  • load_stac: optimize resolution with regard to requested bands (experimental) (#1043)
  • load_stac: apply offset (experimental) (#1051)
  • Deprecate non-standard "logging-threshold" job option in favor of standardized "log_level" job creation parameter (#909)
  • Fail fast on UDF dependency installation failure (#1048)
  • load_stac: avoid OOM on global low-res assets (#1055)

0.60.1

  • load_stac: restore spatial dimensions of data cube loaded from STAC Collection that lacks cube:dimensions. (#1036)
  • sar_backscatter: report soft-errors fraction in usage metrics
  • apply_neighborhood: for t='P1D', add date to dataarray.attrs with key 't'

0.60.0

  • Make environment variables to be passed from web app driver to batch job driver configurable (#867)
  • load_collection/load_stac: support parameters in properties (Open-EO/openeo-python-driver#327)

0.59.0

  • load_stac: cube creation is now cached, just like load_collection (#993)
  • logs: Provide a performance summary at the end of a batch job.
  • StacApiWorkspace: support filepath_per_band (#867)
  • load_stac: use STAC API Filter Extension to prevent driver OOM (#979)

0.58.1

  • Fix spatial and temporal extents of exported STAC Collection (#867)

0.58.0

  • Improve "App not found" logs to avoid red herrings in root cause analysis (eu-cdse/openeo-cdse-infra#147)
  • Avoid pixel shift when source data is not aligned in load_stac. (#648)
  • Fix when outputting an empty vector cube to GeoParquet (#987)
  • Fix for outputting a vector cube to legacy timeseries.json format. (#342)
  • Add gdalinfo json files next to tiff files. (openeo-geotrellis-extensions#352)
  • Support exporting assets for different collections to different paths (#867)
  • load_collection: bugfix in 'global extent' computation, increases performance (#334)

0.57.0

  • Initial support for S3 profiles and tokens during batch job execution (#969)

0.56.0

  • Initial support for CWL based processes with Calrissian on a Kubernetes cluster (#936)

0.55.0

  • Support file_metadata format option to set file-specific metadata on GTiff output assets (#970)
  • load_collection for Sentinel-3 Level2 data: avoid data corruption in specific case (#755)

0.54.0

  • export_workspace: experimental support for merging into STAC API (#867)

0.53.1

  • export_workspace: fix KeyError: 'alternate' upon merging into existing STAC collection (#677)

0.53.0

  • export_workspace: experimental support for merging STAC Collections (#677)
  • Add support for orgId in ETL resource/cost reporting (#671)
  • load_stac: Align output pixels with source pixels if source data is UTM and has an offset. (#648)

0.52.0

  • Throw error when trying to use unsupported target_dimension in aggregate_spatial (#951)
  • Allow specifying region name for an ObjectStorageWorkspace (#955)
  • Better print ApiException. (#962)
  • Include job title by default in user job listings (#963)
  • Support pagination of user job listings (#959/Open-EO/openeo-python-driver#332)

0.51.0

  • load_stac: omit datetime parameter from STAC API item search request if no temporal_extent specified (#950)

0.50.1

  • Fix reduce_dimension of bands for GeoTIFF output in batch job (#943)

0.50.0

  • Fix type of ZLEVEL option for GTiff format
  • Add filepath_per_band to save_result options. (#877)
  • Allow pointing to custom processes with OPENEO_CUSTOM_PROCESSES env var (related to #936)

0.49.1

  • Py4j log level is now always 'WARN' to avoid spurious messages.
  • Fix removal of multiple original job result assets (#883)
  • Fix using asset_per_band with large extends giving partial tiff files (Open-EO/openeo-geotrellis-extensions#329)

0.49.0

  • Fix load_stac for collections from stac.terrascope.be (#862)
  • Point href of job result asset to workspace URI if original was removed (#883)

0.48.2

  • Fix resample_spatial of Sentinel-3 data cube (#920)

0.48.0

  • Expose filename_prefix format option for netCDF output assets (#876)
  • Make sure OPENEO_BACKEND_CONFIG env var is set in K8s executors

0.47.0

0.46.0

  • Automatic Python UDF dependency handling: add option to work with ZIP archive instead of full tree in job work folder, to improve performance/stability in contexts where large file trees under the job work folder are not ideal, e.g. FUSE-mounted S3 storage (#845, docs)

0.45.0

  • Experimental support for removal of originals of assets exported to workspace (#883)
  • A rounding bug was fixed in a downstream library that in specific cases leads to a change in the number of pixel rows/columns in the output. We mainly observe this when the input bounding box is not well aligned to the pixel grid of the Copernicus data. #297
  • Fixed an issue where jobs asset_per_band sometimes returned empty tiff files. (Open-EO/openeo-geotrellis-extensions#329)

0.44.1

0.44.0

  • Job tracker: only consider jobs updated in last 2 weeks (#902)

0.43.0

0.42.0

  • Job tracker (still based on DoubleJobRegistry): only consider last 2 weeks of "trackable" jobs (#902)

0.41.0

  • quantiles, when used in apply_dimension was corrected to use the interpolation method that is prescribed by the openEO process definition.
  • return STAC Items with valid date/time for time series job results (#852)
  • filter_labels now also supported for collections that use sar_backscatter and use the internal Orfeo toolbox based method to compute backscatter on the fly. For example: SENTINEL1_GRD (Open-EO/openeo-geotrellis-extensions#320)
  • support mask assets in load_stac (#874)
  • align DataCubeParameters with load_collection (#812)
  • apply/apply_dimension(dimension='bands'): nodata tiles were removed as an optimization, but this could lead to unexpected results depending on subsequent steps. They are now replaced with a memory efficient implementation. ([WorldCereal issue][WorldCereal/worldcereal-classification#141])
  • load_collection with an excessive extent (temporal or spatial) will now be blocked to avoid excessive resource usage. This check can be disabled with job_options.do_extent_check=False (#815)
  • Mixing bands with signed and unsigned data types could lead to negative values being misrepresented. This is now fixed by using the correct data type for the output.
  • Logging output is being reduced to focus on most relevant messages from a user perspective.
  • Support multiple export_workspace processes (eu-cdse/openeo-cdse-infra#264)
  • Fix export_workspace process not executed in process graph with multiple save_result processes (eu-cdse/openeo-cdse-infra#264)

0.40.1

  • Fix load_stac of GeoTiff batch job results by returning compliant GeoJSON in "geometry" of STAC Items. #854

0.40.0

  • a new 'python-memory' option allows to more explicitly limit memory usage for UDF's, sar_backscatter and Sentinel3 data loading. The executor-memoryOverhead option can be reduced or removed when using the new python-memory option.
  • The default processing chunk size can now be configured for backends. If not set, the default may be determined automatically. We observe that a lower default, like 128 pixels, allows running jobs with less memory. (Open-EO/openeo-geotrellis-extensions#311)
  • aggregate_spatial: trying to use the probabilities argument in a single 'quantiles' reduces was throwing an error. (#821)
  • sar_backscatter: when a target resolution is provided via resample_spatial, it is now immediately taken into account for computing backscatter, reducing memory usage.
  • the temporary folder which is created for aggregate_spatial now contains a timestamp to aid cleanup scripts.
  • apply_neighborhood: support applying UDF on cubes without a time dimension
  • Add "separate_asset_per_band" to save_result options. Currently, for TIFF only.
  • load_stac: presence of eo:bands is no longer a hard requirement; band name defaults to asset key #762.
  • Optionally sleep after automatic UDF dependency install (config udf_dependencies_sleep_after_install) (eu-cdse/openeo-cdse-infra#112)

0.39.0

  • Correctly apply the method parameter in resample_spatial and resample_cube_spatial, when downsampling to lower resolution, and the sampling is not applied at load time. (Open-EO/openeo-geotrellis-extensions#303)
  • Use band names as column name in GeoParquet output (#723)
  • Prevent nightly cleaner from failing a job tracker run (eu-cdse/openeo-cdse-infra#166)
  • Sentinelhub collections handle non zero nodata better (openeo-geotrellis-extensions#300)
  • Support allow_empty_cubes job option (#649)
  • Cross-backend jobs: support running main job on backends that lack async_task infrastructure. (#786)
  • Support save_result processes in arbitrary subtrees in the process graph i.e. those not necessarily contributing to the final result (#424)

0.38.6

  • Increase default resource usage configs for sync processing (eu-cdse/openeo-cdse-infra#158)

0.38.5

0.38.4

  • Fix load_stac from unsigned job results URL in batch jobs (#792)

0.38.3

  • Automatically include declared/installed UDF dependencies in PYTHONPATH on YARN deploys (#237)

0.38.2

0.38.1

  • fix load_stac from MS Planetary Computer STAC API (#784)

0.38.0

  • Initial, experimental support for automatic installing declared UDF dependencies (#237)

0.37.2

  • load_stac: incorporate STAC Item geometry (#778)

0.37.1

  • load_stac from STAC API: fix CRS and resolution of output assets (#781)

0.37.0

  • Support additional options in FreeIpaClient.user_add() (eu-cdse/openeo-cdse-infra#56)

0.36.0

  • Job tracker: skip jobs where application id can't be found (instead of giving status "error") to be less destructive in distributed contexts with partial replication (related to eu-cdse/openeo-cdse-infra#141)

0.35.0

  • Add config zk_job_registry_max_specification_size to set a limit on the size of the process graph items when registering a new batch job with ZkJobRegistry. Jobs with a process graph that is too large will be partially stored in the registry: most metadata will be available, but use cases that try to get the process graph itself will fail with a JobNotFound-like error. This is intended to be combined with ElasticJobRegistry through DoubleJobRegistry to allow ElasticJobRegistry to act as fallback for jobs that are too large for ZkJobRegistry. (related to #498, eu-cdse/openeo-cdse-infra#141)

0.34.0

  • load_stac from unsigned job results URL: fix CRS and resolution of output assets (#669)
  • Align job registry implementations to omit "process" and "job_options" in user job listings (related to #498, eu-cdse/openeo-cdse-infra#141)

0.33.1

  • array_create bugfix to support mixed data types (#287)

0.33.0

  • Support correlation ID in job tracker logs (#707)

0.32.0

  • Always enable allow_dynamic_etl_api from synchronous processing (drop feature flag) (#531, eu-cdse/openeo-cdse-infra#114)

0.31.1

  • Initial support for job_options handling in OpenEoBackendImplementation.request_costs() (#531, eu-cdse/openeo-cdse-infra#114)

0.31.0

  • vector_to_raster now returns a valid raster cube (Open-EO/openeo-python-driver#273)
  • aggregate_spatial can now be used as input for vector_to_raster (#663)
  • raster_to_vector now returns a valid vector cube (Open-EO/openeo-python-driver#276)
  • raster_to_vector now includes the pixel values of the output polygons (#578)
  • raster_to_vector now adds an id to each polygon including band name, date, and index (#578)
  • resample_spatial now has better performance when resampling to a high resolution (#265)
  • support property filters for STAC based collections (#460)
  • performance improvement when reading from object storage (#740)
  • support Microsoft Planetary Computer in load_stac (#760)

0.30.2

  • load_stac: fix filtering by Item properties
  • Fix common cause of 'TopologyException' errors in vector processing
  • sample_by_feature will now use the 'feature_id_property' setting for naming generated assets. #722

0.30.1

  • Reinstate ejr_credentials_vault_path config option.

0.30.0

  • Remove deprecated and unused ejr_credentials_vault_path config option.

0.29.0

  • aggregate_temporal(_period) Performance improvement to use number of bands in metadata rather than computing the band count.
  • Added initial implementation of FreeIpaClient (eu-cdse/openeo-cdse-infra#56)

0.28.2

  • Mask process now also works when the mask is in a different projection, so resample_cube_spatial is needed in fewer cases.
  • Improve resilience by retrying ETL API requests (#720).

0.28.1

0.28.0

  • Export to JSON is now more robust, supports datetime objects returned by dimension_labels, and will default to the string representation.
  • GDAL upgraded to 3.8.4 and Orfeo Toolbox to 8.1.2. This mainly reduces the volume of bytes read from object storage by GDAL. (#571)
  • Size of incoming requests is now limited to 2MB by default (Open-EO/openeo-python-driver#254)
  • load_stac: support loading netCDF multiple netCDF items with a time dimension, as produced with 'sample_by_feature' option
  • In batch result STAC metadata proj:shape is fixed to be in Y-X order, as prescribed by the standard. (#693)
  • Copy batch job output assets to a workspace with the export_workspace process (#676).
  • Support vector cubes loaded from load_url in "sample_by_feature" feature (#700)
  • Keep polygons and multipolygons sorted when calling aggregate_spatial (#60)

0.27.1

  • Add timeout to requests towards ETL API to unblock JobTracker (#690).

0.27.0

  • Expose "bbox" and "geometry" for spatial STAC Item with netCDF assets (#646)

0.26.2

  • MultiEtlApiConfig: don't fail-fast on missing env vars for credentials extraction, just skip with warnings for now

0.26.1

Bugfix

  • fix load_stac from unsigned job results URL in batch job (#644)

0.26.0

  • Introduce MultiEtlApiConfig to support multiple ETL API configurations (#531)

0.25.0

  • The default for the soft-errors job option is now set to 0.1 and made configurable at backend level. This value better recognizes the fact that many EO archives have corrupt files that otherwise break jobs #617.
  • Support GeoParquet output format for aggregate_spatial (#623)

Improved datatype conversion

A rather big improvement in this release is the handling of datatypes. OpenEO does not have very explicit rules when it comes to datatypes, and in this implementation, in most cases the datatype was simply preserved, like in most programming languages.

For most users, this resulted in unexpected behaviour, for instance when dividing integer dataypes, or subtracting two unsigned 8 bit numbers, and expecting to get negative values.

This implementation will now try to use wider datatypes when necessary. For instance by switching to floating point when performing a division. This change makes writing formulas more intuitive, and should save time debugging issues.

When there is still a need to get a smaller datatype, users can use the 'linear_scale_range' process. This process for instance will convert to 8 bit unsigned integers if the target range uses integer values and fits in the [0,255] range.

Relevant issues:

0.24.0

  • Start using DynamicEtlApiJobCostCalculator in job tracker. Effective ETL API selection strategy is to be configured through EtlApiConfig

Bugfix

  • added max_processing_area_pixels custom option to sar_backscatter, avoiding going out of memory when processing too large chunks

0.23.1

Bugfix

  • Requests towards Job Registry Elastic API are unreliable; reconsider ZK as primary data store.

0.23.0

Added

  • Support disabling ZkJobRegistry (#632)

0.22.3

Bugfix

  • Restore batch job result metadata; this reverts the Zookeeper fix introduced in 0.22.2

0.22.2

Bugfix

  • Prevent Zookeeper from blocking requests (Open-EO#639)

0.22.1

Bugfix

  • Prevent usage duplication in ETL API (#41)

0.22.0

Added

  • Added config use_zk_job_registry to disable ZkJobRegistry usage

Bugfix

  • apply_neighborhood: fix error if overlap is null/None (#519)

0.21.5

  • Initial implementation of DynamicEtlApiJobCostCalculator and added caching feature to get_etl_api() (#531)

0.21.4

  • Support for reading GeoPackage vector data
  • move legacy-vs-dynamic ETL selection logic to get_etl_api() (#531)

0.21.3

  • job tracker: move app state mapping to CostDetails construction time

0.21.2

  • job tracker: pass job_options to JobCostsCalculator through CostDetails (related to #531)

0.21.1

  • job tracker: do job info iteration in streaming fashion (instead of loading all job info in memory at once)

0.21.0

  • Initial support for dynamic ETL API configuration (#531)

0.20.1

  • finetune zookeeper_set.py script and concurrent_pod_limit logic

0.20.0

  • Introduce GpsBackendConfig.zookeeper_hosts and GpsBackendConfig.zookeeper_root_path

0.19.4

0.19.3

Feature

  • Experimental support for filter_labels: only works with catalog based collections, and when used close to the load_collection call #559
  • Support for UDF signature that works directly on XArray DataArray, avoiding the need for openEO specific wrapper class.
  • Support filtering on tileId with a wildcard. Open-EO/openeo-opensearch-client#25

Bugfix

  • Error fixed when doing aggregate_temporal + merge_cubes Open-EO/openeo-geotrellis-extensions#201
  • Avoid 'out-of-memory' errors when writing large netCDF files. Allows files of >600MB without custom memory settings. #199
  • netCDF output will generate a more useful warning in case of a mismatch with cube band metadata

0.19.2

  • (Temporarily) disable extensive /validation checks on production (related to #566, #575)

0.18.0a1

Removed

  • Remove old "v1" job_tracker script (#545)

0.17.0a1

Feature

Bugfix

2023-09-18 (0.9.5a1)

Important change: time intervals are now left closed. Workflows that are sensitive to exact time intervals may need to be updated.

Feature

  • load_stac support, allowing to load STAC collections that conform to the mainstream approach for organizing metadata in STAC. (Open-EO#402)
  • First support for UDF's that change resolution, in Python-Jep runtime. (Open-EO/openeo-geotrellis-extensions#197)
  • Improved support for running UDF's on vector cubes.
  • Support load_geojson and load_url processes to create vector cubes. (Open-EO/openeo-python-driver#211)
  • The 'partial' query parameter is now supported when requesting job results, load_stac supports loading unfinished results. (Open-EO#489)
  • Support new (experimental) vector_to_raster process, allowing to combine data from a vector source with EO data. (Open-EO#423)

Bugfix

Changed

2023-07-30 (0.9.5a1)

Feature

  • array_element: Support band selection by label (Open-EO#43)
  • apply_neigborhood: Support applying function over time intervals (Open-EO#415)

2023-06-30 (0.9.5a1)

Feature

Changed

2023-03-30 (0.9.5a1)

Bugfix

  • Fix "Permission denied" issue with run_udf usage on vector date cube (#367)
  • Fix: Extent in STAC result metadata should be lat lon (#321)
  • Single row/line results with SentinelHub (#375)
  • Fix: Creodias: Download asset from object storage (S3) before extracting projection metadata (#403)

Changed

  • /validation now detects if the amount of pixels that will be processed is too large (#320)
  • Add projection extension metadata to batch job results (openeo-geotrellis-extensions/#72)

2023-03-08 (0.9.3a1)

Build 20230307-1166, with components: openeo-geopyspark-0.9.3a1.dev20230307+1073, openeo_driver-0.37.0a1.dev20230307+441, openeo-0.15.0, geotrellis-extensions-static-2.3.0_2.12-SNAPSHOT-5822561

Feature

  • Add "filename_prefix" to format_options.

Bugfix

2023-02-27 (0.7.0a1)

Build 20230221-1118

Note: this deploy was rolled back to previous build 20230117-966 the same day.

  • GeoParquet support to allow loading large vector files
  • Improved specific log messages
  • Better support for multiple filter_spatial prcesses in same process graph (Open-EO#147)
  • Bugfix for sampling sentinelhub based collections (Open-EO#279)
  • vector_buffer: Throw an error when a negative buffer size resuls in invalid geometries (Open-EO/openeo-python-driver#164)
  • batch jobs now also report usage of credits (Open-EO#272)
  • non-utm collections should now have a better alignment to the original rasters, if the process graph does not apply an explicit resampling (Open-EO/openeo-geotrellis-extensions#69)

2023-02-07 (0.6.7a1)

Build 20230117-966

  • Added initial support for the inspect process. It can be used on datacubes and in callbacks.
  • The size of a single chunk is now automatically increased for larger jobs, to improve IO performance.
  • resample_cube_spatial is no longer needed in all cases when using merge_cubesor mask
  • Better detection of duplicate products in source catalogs
  • The 'if' process will no longer evaluate the branch that is not accepted Open-EO/openeo-python-driver#109

2023-01-20 (0.6.7a1)

2022-11-28 (0.6.3a1)

  • Added an experimental job option 'udf-dependency-archives' to pass on archives of UDF dependencies

2022-10-27 (0.6.3a1)

  • Reprojection is performed at load time whenever possible, by pushing down parameters from resample_spatial and resample_cube_spatial
  • PROBA-V collections can now be loaded at original resolution
  • Overlap between original products is now handled based on the footprint in STAC/Opensearch metadata
  • Logging for synchronous jobs is now more complete
  • First prototype for running vector data UDF's on Spark
  • Bugfix: allow large (multiple GB) CSV output
  • Try to avoid going out of memory by reducing default partition size

2022-09-21 (0.6.3a1)

  • Expose logging from UDF's
  • Feature id's from GeoJSON are used to name timeseries in netCDF export
  • NetCDF's are now cropped to provided extent
  • Support remote STAC collections
  • Sentinelhub usage is now recorded for batch jobs
  • "task-cpus" job option to control number of cpu's for a single Spark task. Mostly relevant for UDF's that use multi-threaded libraries such as Tensorflow.
  • New processes:
    • array_find
    • exp

2022-05-04 (0.6.3a1)

  • Enable JSON logging from batch_job.py (and inject user_id/job_id)
  • New processes:
    • predict_catboost (not-standard)
    • predict_random_forest
    • fit_class_random_forest
    • array_interpolate_linear
  • Faster sar_backscatter both for large areas and sparse sampling
  • STAC metadata for random forest models
  • Colormap support in PNG's
  • Support custom Sentinelhub collections, e.g. PlanetScope data
  • 'soft-errors' job option to allow failure of individual Sentinelhub requests

2022-04-07 (0.6.2a1)

  • EP-4012: implement collection source selection based on product availability (e.g. collection "SENTINEL2_L2A" "forwards" to "TERRASCOPE_S2_TOC_V2" when possible, but falls back to "SENTINEL2_L2A_SENTINELHUB" when there are missing products for the selected spatiotemporal extent.

2021-11-17

Feature

  • Support load_result
  • Allow raster masks to filter a collection before loading any data
  • Caching of Sentinelhub data
  • Streaming writing of netCDF files
  • Support filter_spatial
  • Support first and last processes
  • Jep based UDF implementation

2021-07-14

Changed

  • Add support for openeo.udf based UDFs and keep backward compatibility with openeo_udf based UDFs (EP-3856, #78, #93)

2021-04-08

Feature

  • Add support for (multiple) default OIDC clients (for EGI Check-in OIDC provider) (EP-3700, Open-EO/openeo-api#366)

2021-03-30

Feature

  • Add support for Sentinelhub layers on different endpoints (e.g. Landsat-8, MODIS)
  • In batch jobs, write one geotiff per date as opposed to reducing all dates into a single pixel
  • Improved CARD4L metadata generation for atmospheric_correction

2021-03-12

  • Fix support for UDPs in batch jobs (EP-3754)
  • Fix support for custom processes in batch jobs (EP-3771)

2021-01-26

Feature

Add an experimental resolution_merge for Sentinel-2 based on the implemntation in FORCE.

Support reading Copernicus Global Land NetCDF files.

Support the Sentinelhub batch process API to generate Sentinel-1 backscatter data.

The atmospheric_correction process can now apply iCor on SentinelHub layers.

2021-01-25

Feature

  • Add implementation of on-the-fly Sentinel1 Backscatter (Sigma0) calculation using Orfeo Toolbox on Creodias (EP-3612)

2020-12-06

Performance

Performance improvement for requests with small spatial extents. The backend was loading too much tile metadata.

2020-11-11

Feature

Support the "if" process:https://processes.openeo.org/#if

Major performance improvements for SentinelHub layers. The UTM projection is now used by default when processing these layers. The datatype is no longer set to float by default.

2020-10-28

Internal

Refactored internal process graph parsing: first to a dry-run processing to extract information that can help loading initial data sources. (EP-3509)

2020-10-14

Feature

Support "PNG" output format (non-indexed only).

2020-10-06

Performance improvement

Geotiff (GTiff) output format is faster to generate, and works for larger areas.

Compatibility

Copernicus projections stored in UTM are now also processed and returned in UTM, as opposed to web mercator. This affects processing parameters that depend on a specific projection, like the size of pixels in map units.

This change also improves memory usage and performance for a number of layers.