-
Notifications
You must be signed in to change notification settings - Fork 19
feat(cogify): Update cogify to support the topo raster processes. BM-1116 #3388
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
24087c3 to
61f2b1a
Compare
|
I've re-based this branch to |
|
Some functions have been refactored several times and therefore have outdated docs. I am correcting such instances now. Edit: I've also refined many of the Topo STAC creation functions as they had remnants of past refactorings that needed to be re-optimised. |
…ween topogrpahic creation and generic cog creation
) ### Motivation As a Basemaps user, I want to consume the NZTopo 50 & 250 Maps [as a tile service]. ### Modification This work defines a new Argo `topo-maps-standardising` workflow and an update to the existing `imagery-import-cogify` workflow. We have used this work to generate and publish the [NZ Topo Gridless Maps] layer to basemaps. 1. **`topo-maps-standardising` (new)** This work defines a new Argo [topo-maps-standardising] workflow for coordinating the standardisation of an NZTopo Map Series imagery collection. We have designed the workflow to handle the following collections stored in the AWS S3 [TopoReleaseArchive] directory: ``` s3://topographic-upload/TopoReleaseArchive/NZTopo50_GeoTif_Gridless/ s3://topographic-upload/TopoReleaseArchive/NZTopo250_GeoTif_Gridless/ ``` To support this workflow, we have introduced new [Topo Raster processes] to the `basemaps\cogify` package. This workflow executes these new processes as part of its process flow. **This workflow has two key steps:** For an NZTopo Map Series imagery collection: - Generate a directory structure of STAC Item and Collection files - Standardise the contained GeoTIFF imagery files and save them within the generated directory structure From this, we can pass the resulting directories into the [imagery-import-cogify] workflow to process the standardised imagery into basemaps, either for previewing, or publishing. 2. **`imagery-import-cogify` (update)** This work also updates the [imagery-import-cogify] workflow to support the new background parameter for the CLI `create` command of the `basemaps/cogify` package. This allows the command to override any transparent pixels with a fixed colour. In this work's context, this allows us to eliminate [imagery artifacts] that used to appear around the COGs, particularly for NZTopo Map Series imagery. <!-- external links --> [NZ Topo Gridless Maps]: https://basemaps.linz.govt.nz/@-41.8899962,174.0492437,z5?style=topo-raster&i=topo-raster [as a tile service]: linz/basemaps#3365 (comment) [TopoReleaseArchive]: https://linz-topographic-upload.s3.ap-southeast-2.amazonaws.com/topographic/TopoReleaseArchive/ [Topo Raster processes]: linz/basemaps#3388 [topo-maps-standardising]: https://github.com/linz/topo-workflows/pull/932/files#diff-985cd50a4d3d95216b2aac800f8f1590a1ac94d4e3d1041ade2f09ce0e1b5bc6 [imagery-import-cogify]: https://github.com/linz/topo-workflows/pull/932/files#diff-5b88dd6435a818dfac26161a8270f7f4ae499f9771f91e7c8ac08d807f033845 [background parameter]: linz/basemaps#3379 [imagery artifacts]: linz/basemaps#3379 (comment)
[7.15.0](v7.14.0...v7.15.0) (2025-03-17) ### Bug Fixes * **lambda-analytic-cloudfront:** process logs upto one hour ago ([#3409](#3409)) ([8d9609d](8d9609d)) ### Features * CITM2000 debug tile matrix ([#3397](#3397)) ([4bb91bd](4bb91bd)) * **cogify:** Update cogify to support the topo raster processes. BM-1116 ([#3388](#3388)) ([4366df6](4366df6)), closes [/github.com//issues/3365#issuecomment-2445444420](https://github.com//github.com/linz/basemaps/issues/3365/issues/issuecomment-2445444420) * **shared:** support multiple credential locations ([#3407](#3407)) ([92de7c4](92de7c4)), closes [/github.com/linz/argo-tasks/blob/master/src/fs.register.ts#L132](https://github.com//github.com/linz/argo-tasks/blob/master/src/fs.register.ts/issues/L132)
Motivation
As a Basemaps user, I want to consume the NZTopo 50 & 250 Maps as a tile service.
Modifications
This work comprises a series of additions and modifications to the
basemaps/cogifypackage.These changes allow us to generate STAC files for an NZTopo Map Series imagery collection. They also allow us to generate standardised and cloud-optimised GeoTIFFs from the map sheet files of such collections. We have adjusted the
basemaps/cogifypackage to handle the following collections stored in the AWS S3 TopoReleaseArchive directory:There is also a new Argo workflow in development that depends on this work as part of its process flow. The workflow's purpose is to automate the standardisation of an NZTopo Map Series imagery collection.
Generating STAC files
This works defines a new process for generating a nested directory structure of STAC files for an NZTopo Map Series imagery collection. The process operates as follows:
Outputs
The command groups the images by EPSG and then structures the generated StacItem and StacCollection files, as illustrated:
The command then saves the generated tree of folders and files into the target location directory.
Processing Map Sheet files
This work extends the
createCLI command of thebasemaps/cogifypackage. This extension allows the command to recognise and process the map sheet files of an NZTopo Map Series imagery collection by way of particular properties added to the accompanying STAC files.