feat(cli-vector): New cli to create mbtiles for vector map. BM-1268 #3435
+2,954
−157
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Wentao-Kuang
changed the title
ccbblin
reviewed
Apr 30, 2025
ccbblin
reviewed
May 5, 2025
ccbblin
reviewed
May 5, 2025
Wentao-Kuang
commented
May 20, 2025
… properties BM-1268 (#3440) 1. This pull request extends the [feat/etl-create] branch, not the [master] branch. 2. The changed files contain short-circuiting code to enable faster debugging that needs to be purged before approval. --- ### Motivation We are currently migrating our Vector ETL system into basemaps as a new `cli-vector` package. We are also re-building the system in the process. As a part of the re-build, we have streamlined the system's architecture in the following four steps: 1. `extract` 2. `create` 3. `join` 4. `analyse` This work implements an extension to the base branch's implementation of the `create` CLI command. This work introduces a series of functions for handling dataset features of specific Shortbread layers for which we require special tagging. ### Modifications | Diagram | | - | | ![][diagram1] | 1. Implements a switch-style pattern for identifying the Shortbread layer to which a feature is assigned. 2. Implements bespoke functions for overriding a feature's metadata and properties for the following Shortbread layers: - `contours` - `place_labels` - This one is weird. We will need to discuss the approach for this one. - `pois` - This one doesn't actually do anything as it never receives any features that will meet its conditions for special tagging. We can discuss this also. - `public_transport` - `street_labels` - This one's logic is very similar to the `streets` logic. - `streets` - This one's logic is very similar to the `street_labels` logic. - `water_polygons` #### Processing a Vector Stac Item The system processes a Stac Item file created via the **cli-vector** package's new `extract` command like so: 1. Parse the file as a runtime `VectorStacItem` type. 2. Download the source file (i.e. a GeoPackage file). 3. Convert the source file into an NDJSON file. 4. Parse each line (i.e. feature) of the NDJSON file, generalise it (i.e. simplify, add/remove attributes, or discard). 5. Process the collection of generalised features into an mbtiles file. 6. Upload a copy of the mbtiles file to the same directory as the Stac Item file. 7. Update the Stac Item file's contents and overwrite it. | Diagram | | - | | ![][diagram2] | ### Verification 1. Established test cases for the special tagging functions for the following Shortbread layers: - `contours` More to come if desired. <!-- external links --> [feat/etl-create]: https://github.com/linz/basemaps/tree/feat/etl-create [master]: https://github.com/linz/basemaps/tree/master [diagram1]: https://github.com/user-attachments/assets/6a405417-59fb-4127-8630-cb0901c12618 [diagram2]: https://github.com/user-attachments/assets/2994bd5d-c668-4b99-957e-57623da7c6a1 --------- Co-authored-by: Wentao Kuang <[email protected]>
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Wentao-Kuang
added
container
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Modifications
Verification