Add pmtiles support - #47
Closed
acalcutt wants to merge 30 commits into
Closed
Conversation
Wall mode speedup by about 10 the speed of writing in the database Create index on tiles for efficient access to this table
Otherwise, the file .mbtiles-wal becomes huge (same size as the final file). The result is the need to have twice the size of the final Mbtiles on the hard drive
This was originally 0 until i changed it for the error "densify_pts must be at least 2 if the output is geograpic". i assume this was set to 0 for performance, so I am going to set this to the new minimum of 2 since I'm not sure we need the extra padding. densify_pts (uint, optional) – Number of points to add to each edge to account for nonlinear edges produced by the transform process. Large numbers will produce worse performance. Default: 21
work around 'Python int too large to convert to SQLite INTEGER' error
* Update cli.py * Update cli.py * Update cli.py * Update cli.py * list resampling test * Revert "list resampling test" This reverts commit 424e2ba. * Update cli.py * try to fix db path issue * Update merger.py * Update merger.py * test * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update database.py * test db changes * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * test * Update merger.py * Update merger.py * Update merger.py * Update cli.py * Update merger.py * test * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update database.py * test * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * Update image.py * test * Update merger.py * Update merger.py * Update merger.py * Update image.py * Update image.py * Update image.py * Update merger.py * Update merger.py * Update merger.py * disable inverse y indexing * Update database.py * Update image.py * masking test * Update merger.py * try to make mbtiler use less memory * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update image.py * Update image.py * Update mbtiler.py * Update mbtiler.py * test * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update cli.py * test * Update image.py * Update image.py * Update image.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update image.py * Update mbtiler.py * Update mbtiler.py * remove kwargs * remove kwargs * use format * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * test * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * test gaussian_blur_sigma * Update README.md * move blue after reprojection * add zoom based gaussian_blur_sigma * add retry * add retry, try to get mbtiler working again * fomat table appends * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * update metadata * Update merger.py * Update merger.py * Update merger.py * start raster merger * add bounding tile option * try to fix tile generation * test raster merger * try to fix bounds * Update cli.py * Update cli.py * Update cli.py * sort tiles z, x, y * Update cli.py * Update cli.py * Update cli.py * try to fix cli * Update cli.py * Update cli.py * Update mbtiler.py * Revert "Update mbtiler.py" This reverts commit 218f294. * Revert "sort tiles z, x, y" This reverts commit dfd9382. * add verbose logging flag * Update cli.py * Update raster_merger.py * Update raster_merger.py * Update merger.py * Update merger.py * fix output path * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * try to fix bounds * try to make dynamic chunk size * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * add bounds_source - test * fix processed message * Update mbtiler.py * test * test 2 * Update mbtiler.py * Update mbtiler.py * Update mbtiler.py * claud test * revert dynamic chunk size * Update README.md fix run documentation * Update README.md * Update README.md * add output_nodata option * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update merger.py * Update cli.py * Update merger.py * Update merger.py * inverse y * Update merger.py * Update merger.py * Update merger.py * add sparse tiles option * remove quantized_alpha * Update test_cli.py * add missing end quote * try to pass zoom * try to fix source_zoom * Revert "try to fix source_zoom" This reverts commit e853bb7. * Revert "try to pass zoom" This reverts commit 688000f. * use source_zoom * check if all upscalled before resampling * skip NaN tiles in sparse tile mode * remove quantized alpha * Update merger.py * add tests * add psutil * tests * Fix all 13 CI test failures from round 3 - merger.py: remove double height_adjustment application from _merge_tiles (was already applied in _decode_tile, causing doubled elevation offsets) - database.py: INSERT OR REPLACE to handle duplicate tiles on re-run - test_cli.py: skip test_cli_good_elev (GeoTIFF output mode removed); use MBTilesDatabase for test_merge_command sources (was empty files) - test_encoders.py: use np.allclose(atol=0.5) for roundtrip tests (terrarium ignores interval; float32 precision causes ~0.15 max error) - test_mbtiler.py: skip test_RGBtiler_format_fails (format validation moved to CLI click.Choice) - test_merger.py: fix _merger() helper to default num_sources=2; use _WORLD_BOUNDS to avoid mercantile InvalidLatitudeError at lat=±90 * Fix 2 remaining CI failures - merger.py: fix source priority — mask should be np.isnan(result) so higher-priority (first) sources win; previous mask ~np.isnan(resampled_data) caused every subsequent source to overwrite earlier ones (last wins bug) - cli.py: fix resampling choice 'gaussian' -> 'gauss' to match rasterio's Resampling enum member name; tests correctly pass 'gauss' * Add live GEBCO+JAXA fixture-based merge tests - test/download_fixtures.py: script to fetch z0-z2 tiles from both tile servers into MBTiles fixture files (auto-decompresses gzip responses) - test/fixtures/gebco_sample.mbtiles: 21 WebP tiles from ocean-rgb endpoint - test/fixtures/jaxa_sample.mbtiles: 21 WebP tiles from jaxa_terrainrgb_webp endpoint - test/test_merger.py: TestLiveMerge (6 tests) mirroring merge_bathymetry.json: JAXA land (priority 1) over GEBCO bathymetry (priority 2), z0-z2, webp output - produces output with tiles at all 3 zoom levels - output tiles are valid WebP RGB 512x512 images - JAXA land takes priority (positive elevation in East Asia z=2/2/1 tile) - sparse_tiles mode produces <= full mode tile count - output_nodata config key accepted without error Tests skip automatically when fixture files are absent. * Add pixel-level expected output comparison for live merge tests - Add test/expected/z{0,2}_x*_y*.png: lossless PNG reference tiles generated by running the GEBCO+JAXA merger once and frozen as ground truth - Add test/generate_expected_tiles.py: regeneration script to re-run whenever merger behaviour is intentionally changed - Add TestLiveMerge.test_output_matches_expected_tiles: decodes output tiles to elevation arrays and asserts np.allclose(atol=1.0) against references - Update _EXPECTED_TILES_DIR to point to existing test/expected/ convention
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.