Skip to content

Map Compression with LZMA for size benefit - #1579

Open
christf wants to merge 7 commits into
navit-gps:trunkfrom
christf:mapsize
Open

Map Compression with LZMA for size benefit#1579
christf wants to merge 7 commits into
navit-gps:trunkfrom
christf:mapsize

Conversation

@christf

@christf christf commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This allows us to use LZMA in maptool and navit, utilizing multiple cores during compression.

Despite the multi-core aspect, this about doubles the runtime of maptool for me for a size gain of 37%

@christf christf changed the title Mapsize Map Compression with LZMA for size benefit Jul 3, 2026
@christf
christf force-pushed the mapsize branch 2 times, most recently from 66aca3f to ccdb48f Compare July 3, 2026 17:23
@jkoan jkoan added this to the 0.5.8 milestone Jul 21, 2026
jkoan
jkoan requested changes Jul 21, 2026
View reviewed changes
Comment thread navit/maptool/misc.c Outdated
worker_threads[i].compression_method = compression_method;
#ifdef HAVE_LZMA
if (compression_level > 0) {
uint64_t mem = lzma_easy_encoder_memusage(compression_level);

@jkoan jkoan Jul 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is lzma hardcoded here. My understanding is that this method should be used for both methods, hence the compression_method parameter.

@christf christf Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to think about that one a bit because at first I thought this would not actually be a problem because of that ifdef HAVE_LZMA.
However, even in the zlib case, this lzma-code is being called if general lzma support is baked in, so, yes indeed this needs a change. Will do on next push.

@christf christf Jul 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I have resolved this now. If lzma capabilities are baked into the binary, there is code added that checks whether the current compression method is lzma and in this case sets up the arena allocator.

And it sits in that function because it must be set up before starting the worker threads.

Comment thread navit/maptool/maptool.c Outdated
Comment thread CMakeLists.txt
Comment thread navit/maptool/zip.c
@christf
christf force-pushed the mapsize branch 2 times, most recently from 7391f8b to ce6a05d Compare July 25, 2026 00:04
christf
christf commented Jul 26, 2026
View reviewed changes
Comment thread navit/maptool/maptool.c
p.zip64 = 1; /* default to 64 bit zip */
p.compression_method = ZIP_COMPRESSION_DEFLATE; /* default to zlib */
#if defined(HAVE_ZLIB) || defined(HAVE_LZMA)
p.compression_level = 6;

@christf christf Jul 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not entirely sure if we want it done that way.
Using lzma level 9 is hardly providing much space savings above level 6 but it does increase the time of processing significantly. For zlib, again, space difference is also small.

At the same time, doing different default for zlib / lzma does not seem right. The proposal here is to switch for both, just putting a lantern on it so it is a conscious choice.

@christf
christf force-pushed the mapsize branch 2 times, most recently from 08c4182 to 0ba86a3 Compare July 26, 2026 15:30
@christf
christf marked this pull request as ready for review July 27, 2026 13:14
@christf
christf force-pushed the mapsize branch 3 times, most recently from 77b503a to e93cbf7 Compare August 4, 2026 07:58
@christf
christf force-pushed the mapsize branch 2 times, most recently from 9ba8ceb to 432c792 Compare August 17, 2026 19:43
christf added 7 commits August 18, 2026 00:07
@christf
fix(maptool): avoid invalid free of -t timestamp
71c04ce 
The default timestamp is heap-allocated by current_to_iso8601() and freed
in main(), but the -t override pointed at optarg, whose memory is not
freeable, so the free in main() crashed under ASAN. Own the value with
g_strdup like the other string options so main()'s free is always valid.
@christf
feat(navit): add LZMA decompression for binfile map reading
5cd9084 
Add LZMA decompression support to the navit runtime so it can read
LZMA-compressed binfile maps. When a map is stored with LZMA, the
reader now decodes it transparently alongside the existing deflate path.

- Add CMake HAVE_LZMA detection and lzma.h include guard
- Define ZIP_COMPRESSION_NONE/STORED/DEFLATE/LZMA constants
- Add lzma_uncompress_int() and file_data_read_compressed_method()
- Route binfile read by lfh->zipmthd instead of hardcoded values
@christf
feat(maptool): add LZMA compression and refactored zip writing
9464c6e 
Add LZMA compression to the zip write path and extract a reusable
compress_for_zip() that handles both LZMA and deflate with automatic
fallback. This also adds write_zipmember_raw() for writing
pre-compressed entries, enabling parallel compression in a later commit.

- Add compress_lzma_int() using lzma_easy_buffer_encode
- Extract compress_for_zip() with shared compression buffer reuse
- Add write_zipmember_raw() for pre-compressed zip entries
- Rewrite write_zipmember() to delegate to compress_for_zip()
@christf
feat(maptool): add --compression-method CLI option
89d5416 
Add a -C/--compression-method option to select between zlib and LZMA
compression when building maps. Defaults to zlib for backward
compatibility.

- Add compression_method field to maptool_params
- Wire zip_set_compression_method() through to zip_info
- Bump index version to reflect new tile format
- Use named MAP_INDEX_VERSION constant
- Lower default compression level from 9 to 6 for LZMA practicality
@christf
feat(maptool): add threaded tile processing
c58ab10 
Add a GLib-based thread pool for compressing and writing tile data
in parallel. A persistent pool of worker threads is reused across
slices to avoid repeated thread creation overhead.

- Add process_tile_worker() with per-thread scratch buffer and LZMA arena
- Add tile_worker_pool_init/fini for persistent pool lifecycle
- Extract write_tiles_for_slice() to dispatch tiles to workers
- Auto-detect CPU count when --threads is not specified
- Reuse slice allocation across slices via g_realloc
@christf
docs(maptool): document -C, -T and update -z in man page
bdc2a84 
- -C (--compression-method) selects zlib or lzma
- -T (--threads) sets worker thread count for tile compression
- -z (--compression-level) documents the default value of 9 and ranges
- Synopsis includes the new options in both XML and PBF usage lines
@christf
fix(maptool): write tiles in zipnum order after threaded compression
aeeb130 
The threaded tile processing wrote tiles to the zip file in worker
completion order (non-deterministic) instead of tile_head_root order
(zipnum assignment order). Navit's binfile reader uses sequential
central directory index lookups (zipfile * cde_size) to map zipnum
to tile data, so non-deterministic write order caused zipnum
mismatches and missing/unreachable tiles.

Split write_tiles_for_slice into two phases: first drain all worker
completions (validation only), then write tiles in tile_head_root
order matching the zipnum assignment from write_tilesdir.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jkoan jkoan jkoan requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

0.5.8

Development

Successfully merging this pull request may close these issues.

2 participants

@christf @jkoan