Fix layer priority - #50
Closed
acalcutt wants to merge 32 commits into
Closed
Conversation
Wall mode speedup by about 10 the speed of writing in the database Create index on tiles for efficient access to this table
Otherwise, the file .mbtiles-wal becomes huge (same size as the final file). The result is the need to have twice the size of the final Mbtiles on the hard drive
This was originally 0 until i changed it for the error "densify_pts must be at least 2 if the output is geograpic". i assume this was set to 0 for performance, so I am going to set this to the new minimum of 2 since I'm not sure we need the extra padding. densify_pts (uint, optional) – Number of points to add to each edge to account for nonlinear edges produced by the transform process. Large numbers will produce worse performance. Default: 21
work around 'Python int too large to convert to SQLite INTEGER' error
* Update cli.py * Update cli.py * Update cli.py * Update cli.py * list resampling test * Revert "list resampling test" This reverts commit 424e2ba. * Update cli.py * try to fix db path issue * Update merger.py * Update merger.py * test * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update database.py * test db changes * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * test * Update merger.py * Update merger.py * Update merger.py * Update cli.py * Update merger.py * test * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update database.py * test * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * test * Update merger.py * Update merger.py * Update merger.py * Update image.py * Update image.py * Update image.py * Update merger.py * Update merger.py * Update merger.py * disable inverse y indexing * Update database.py * Update image.py * masking test * Update merger.py * try to make mbtiler use less memory * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update image.py * Update image.py * Update mbtiler.py * Update mbtiler.py * test * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update cli.py * test * Update image.py * Update image.py * Update image.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update image.py * Update mbtiler.py * Update mbtiler.py * remove kwargs * remove kwargs * use format * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * test * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * test gaussian_blur_sigma * Update README.md * move blue after reprojection * add zoom based gaussian_blur_sigma * add retry * add retry, try to get mbtiler working again * fomat table appends * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * update metadata * Update merger.py * Update merger.py * Update merger.py * start raster merger * add bounding tile option * try to fix tile generation * test raster merger * try to fix bounds * Update cli.py * Update cli.py * Update cli.py * sort tiles z, x, y * Update cli.py * Update cli.py * Update cli.py * try to fix cli * Update cli.py * Update cli.py * Update mbtiler.py * Revert "Update mbtiler.py" This reverts commit 218f294. * Revert "sort tiles z, x, y" This reverts commit dfd9382. * add verbose logging flag * Update cli.py * Update raster_merger.py * Update raster_merger.py * Update merger.py * Update merger.py * fix output path * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * try to fix bounds * try to make dynamic chunk size * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * add bounds_source - test * fix processed message * Update mbtiler.py * test * test 2 * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * claud test * revert dynamic chunk size * Update README.md fix run documentation * Update README.md * Update README.md * add output_nodata option * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update cli.py * Update merger.py * Update merger.py * inverse y * Update merger.py * Update merger.py * Update merger.py * add sparse tiles option * remove quantized_alpha * Update test_cli.py * add missing end quote * try to pass zoom * try to fix source_zoom * Revert "try to fix source_zoom" This reverts commit e853bb7. * Revert "try to pass zoom" This reverts commit 688000f. * use source_zoom * check if all upscalled before resampling * skip NaN tiles in sparse tile mode * remove quantized alpha * Update merger.py * add tests * add psutil * tests * Fix all 13 CI test failures from round 3 - merger.py: remove double height_adjustment application from _merge_tiles (was already applied in _decode_tile, causing doubled elevation offsets) - database.py: INSERT OR REPLACE to handle duplicate tiles on re-run - test_cli.py: skip test_cli_good_elev (GeoTIFF output mode removed); use MBTilesDatabase for test_merge_command sources (was empty files) - test_encoders.py: use np.allclose(atol=0.5) for roundtrip tests (terrarium ignores interval; float32 precision causes ~0.15 max error) - test_mbtiler.py: skip test_RGBtiler_format_fails (format validation moved to CLI click.Choice) - test_merger.py: fix _merger() helper to default num_sources=2; use _WORLD_BOUNDS to avoid mercantile InvalidLatitudeError at lat=±90 * Fix 2 remaining CI failures - merger.py: fix source priority — mask should be np.isnan(result) so higher-priority (first) sources win; previous mask ~np.isnan(resampled_data) caused every subsequent source to overwrite earlier ones (last wins bug) - cli.py: fix resampling choice 'gaussian' -> 'gauss' to match rasterio's Resampling enum member name; tests correctly pass 'gauss' * Add live GEBCO+JAXA fixture-based merge tests - test/download_fixtures.py: script to fetch z0-z2 tiles from both tile servers into MBTiles fixture files (auto-decompresses gzip responses) - test/fixtures/gebco_sample.mbtiles: 21 WebP tiles from ocean-rgb endpoint - test/fixtures/jaxa_sample.mbtiles: 21 WebP tiles from jaxa_terrainrgb_webp endpoint - test/test_merger.py: TestLiveMerge (6 tests) mirroring merge_bathymetry.json: JAXA land (priority 1) over GEBCO bathymetry (priority 2), z0-z2, webp output - produces output with tiles at all 3 zoom levels - output tiles are valid WebP RGB 512x512 images - JAXA land takes priority (positive elevation in East Asia z=2/2/1 tile) - sparse_tiles mode produces <= full mode tile count - output_nodata config key accepted without error Tests skip automatically when fixture files are absent. * Add pixel-level expected output comparison for live merge tests - Add test/expected/z{0,2}_x*_y*.png: lossless PNG reference tiles generated by running the GEBCO+JAXA merger once and frozen as ground truth - Add test/generate_expected_tiles.py: regeneration script to re-run whenever merger behaviour is intentionally changed - Add TestLiveMerge.test_output_matches_expected_tiles: decodes output tiles to elevation arrays and asserts np.allclose(atol=1.0) against references - Update _EXPECTED_TILES_DIR to point to existing test/expected/ convention
The squash in 512d65f inverted the merge priority. It now reads mask = np.isnan(result) & ~np.isnan(resampled_data) so the *first* source wins and later ones only fill its holes. Three things say it should be the last source that wins: - The README: "the last input source will be the base layer for tiles", and bounds, max_zoom and bounds_source all default to the last file. - merge_example.json, which lists bathymetry first and terrain after it. Under first-wins the bathymetry paints over the land. - The production configs, which list the base tileset first and the top one last. Concretely, a config of [GEBCO z0-z8, detailed planet z0-z16] under first-wins lets coarse GEBCO win everywhere it has data -- and since _extract_tile falls back to a parent, its upscaled z8 tile keeps winning above z8 too, so every zoom past 8 loses its detail. Two further fixes in the same function: - height_adjustment was applied twice on the merge_sparse branch, once in _decode_tile and again here. master already had this right; the comment now says why the decode-time site is the only one it can live at (mask_values are compared against raw decoded heights, so the shift has to come after masking). - The sparse all-NaN check ran after output_nodata had replaced every NaN, where it could not fire. Moved above the substitution. It is still unreachable -- has_native_with_data returns first for every input that would give an all-NaN result -- but that guard asks a stricter question, so the two are not interchangeable, and the comment says so rather than leaving it looking like a live safety net. The tests here were written against the inverted priority, so two move with it. The unit test now asserts the last source wins, and the GEBCO and JAXA fixtures are listed bottom first rather than JAXA first. That swap is exactly equivalent to the old order under the old rule, which is why the expected-tile references still match pixel for pixel: test_output_matches_expected_tiles passes untouched. generate_expected_tiles.py gets the same swap so regenerating stays consistent. test_merger_regressions.py covers all three fixes at the unit level. 42 merger tests pass, 54 overall. The six failures are get_context("fork") on Windows and are unrelated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.