Refactor:maptool:Replace hand-written OSM parsers with libosmium

maptool has four hand-written OSM input parsers: osm_xml.c, osm_o5m.c,
osm_protobuf.c and osm_protobufdb.c. osm_xml.c scans XML line by line.
The two protobuf parsers use generated protobuf-c code. All four parsers
call the same eight osm_add_* and osm_end_* callbacks in osm.c.

This change replaces the four parsers with one libosmium handler in
osm_libosmium.cpp. The handler calls the same callbacks. The code after
the parse stage does not change: item_bin, the tiles and the zip
assembly stay the same. The change deletes about 2000 lines and adds
about 180 lines.

Reasons for the change:

- The build no longer needs protobuf-c. This change also deletes the
  generated .pb-c sources, osmformat.proto and fileformat.proto.
- maptool detects the input format from the file name. A plain -i reads
  .osm, .osm.gz, .osm.bz2, .pbf and .o5m files. Compressed XML no longer
  needs an external bzcat pipeline.
- libosmium decodes PBF data with more than one thread. maptool parses
  France in 280 s. The old parser needed 354 s. These values are the
  medians of three runs.
- The old XML parser needed one element for each line. If a valid OSM
  file contained an empty line, the parser stopped with a fatal error.
  libosmium reads these files.
- maptool now reports progress during a PBF parse. osm_protobuf.c never
  armed the progress alarm, so phase 1 was silent for minutes on a large
  extract.
- One parser is easier to maintain than four.

Changes in behavior:

- The options -P and -M are now format hints for standard input only.
  With -i, maptool detects the format from the file name.
- This change removes the undocumented options -B and -O. These options
  only used osm_protobufdb.c, which has no replacement.
- maptool now needs a C++14 compiler. DISABLE_CXX therefore has no
  effect if BUILD_MAPTOOL is on. All C++ code is in one file behind a C
  interface.
- The old build excluded PBF on MSVC. libosmium supports MSVC, so this
  restriction is gone.
- osm_psql.c (-d, HAVE_POSTGRESQL) does not change. libosmium has no
  PostgreSQL reader.

Dependencies:

libosmium and protozero contain only headers. The build prefers the
system copies, libosmium2-dev and libprotozero-dev. The build downloads
them only if they are absent. Distribution builds and offline builds
therefore do not need the network. This change deletes
cmake/FindPROTOBUF_C.cmake. It also updates the dependency lists in
docs/development/targets, in scripts and in
CPACK_DEBIAN_PACKAGE_DEPENDS.

Verification:

A comparison of the maps byte by byte does not work. The threaded phases
of maptool put the items of a tile in a variable order. Two runs of the
same binary give maps that differ by some hundred bytes.

The comparison therefore ignores the order. Each tile becomes a multiset
of item type, coordinates and sorted attributes. The comparison includes
every per-phase counter.

- Geofabrik France, 5.05 GB. The input has 524,532,569 nodes,
  73,128,562 ways and 1,098,637 relations. The map has 3.47 GB. All
  114,981 members are identical. All counters are identical. Both maps
  index the same five countries.
- BBBike Denver metro, 1.4 million split ways. All 330 members are
  identical.
- Liechtenstein PBF and an Overpass XML extract. All members are
  identical.

These input paths also give correct results: XML with -i, XML from
standard input, PBF with -i, PBF from standard input, .osm.gz with -i,
.osm.bz2 with -i, and bzcat into standard input.

Real o5m data is not tested. No o5m file and no converter were
available. The o5m reader starts and rejects a bad header correctly.

Author: pgrandin

CMakeLists.txt

@@ -6,7 +6,7 @@ set(MACOSX_BUNDLE_BUNDLE_NAME "Navit")
 set(CPACK_PACKAGE_CONTACT "navit@navit-project.org")
 set(CPACK_DEB_COMPONENT_INSTALL ON)
 #set(CPACK_DEBIAN_MAPTOOL_PACKAGE_DEPENDS "")
-set(CPACK_DEBIAN_PACKAGE_DEPENDS "libglib2.0-0 (>= 2.10.0), libprotobuf-c1")
+set(CPACK_DEBIAN_PACKAGE_DEPENDS "libglib2.0-0 (>= 2.10.0)")
 message(STATUS "Building with CMake V${CMAKE_VERSION}")
 if (DISABLE_CXX)
 	project(navit C)
@@ -746,12 +746,6 @@ if(CMAKE_SIZEOF_VOID_P LESS 8)
 	set_with_reason(BUILD_MAPTOOL "maptool works only on 64 bit architectures" FALSE)
 endif()
 
-if(BUILD_MAPTOOL)
-	find_package(PROTOBUF_C REQUIRED)
-	if(NOT PROTOBUF_C_FOUND)
-		set_with_reason(BUILD_MAPTOOL "PROTOBUF-C not found" FALSE)
-	endif()
-endif()
 
 set(LOCALEDIR "${LOCALE_DIR}")
 

cmake/FindPROTOBUF_C.cmake

@@ -31,7 +31,10 @@ Translations for the user interface
 Maptool
 -------
 
-**protobuf-c-compiler libprotobuf-c-dev**
+**libosmium2-dev libprotozero-dev**
+
+If these packages are not available, the build downloads them. Both libraries
+are header-only.
 
 GTK+
 ----

docs/development/targets/Debian_dependencies.rst

@@ -18,8 +18,8 @@ Optionals
 
 To build maptool:
 
--  protobuf-c
--  libprotobuf-c-devel
+-  libosmium-devel
+-  protozero-devel
 
 
 GTK Gui

docs/development/targets/OpenSuse_dependencies.rst

@@ -33,8 +33,8 @@ Minimum requirements
 --------------------
 
 -  cmake
--  protobuf-c-compiler
--  libprotobuf-c-dev
+-  libosmium2-dev
+-  libprotozero-dev
 -  zlib1g-dev
 -  libpng12-dev
 -  libgtk2.0-dev
@@ -110,5 +110,5 @@ Everything in one command
 
 | ``sudo apt-get install cmake zlib1g-dev libpng12-dev libgtk2.0-dev librsvg2-bin \``
 | ``g++ gpsd gpsd-clients libgps-dev libdbus-glib-1-dev freeglut3-dev libxft-dev \``
-| ``libglib2.0-dev libfreeimage-dev gettext protobuf-c-compiler  libprotobuf-c-dev``
+| ``libglib2.0-dev libfreeimage-dev gettext libosmium2-dev  libprotozero-dev``
 

docs/development/targets/Ubuntu_dependencies.rst

@@ -98,7 +98,7 @@ From .pbf
 """""""""
 .. code-block:: bash
 
- maptool --protobuf -i my_OSM_map.osm.pbf my_Navit_map.bin
+ maptool -i my_OSM_map.osm.pbf my_Navit_map.bin
 
 Processing the whole Planet
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/user/configuration/maps/osm.rst

@@ -2,19 +2,28 @@
 .SH NAME
 maptool \- parse osm textfile and convert to Navit binfile format
 .SH SYNOPSIS
-.B For OSM XML data:
-.B bzcat planet.osm.bz2 | maptool mymap.bin
-[\-h] [\-6] [\-a <level>] [\-c] -[\-d <connect string]
-[\-e <phase>] [\-i <file>] [\-k] [\-M] [\-N] [\-o] [\-r <file>] [\-s <phase>]
+.B maptool \-i planet.osm.pbf planet.bin
+[\-h] [\-6] [\-a <level>] [\-c] [\-d <connect string>]
+[\-e <phase>] [\-k] [\-M] [\-N] [\-o] [\-P] [\-r <file>] [\-s <phase>]
 [\-S <size>] [\-w] [\-W] [\-U] [\-z <level>]
 
-.B For OSM Protobuf/PBF data:
-.B maptool \-\-protobuf \-i planet.osm.pbf planet.bin
-[\-h] [\-6] [\-a <level>] [\-c] [\-e <phase>]
-[\-i <file>] [\-k] [\-M] [\-N] [\-o] [\-P] [\-r <file>] [\-s <phase>]
-[\-S <size>] [\-w] [\-W] [\-U] [\-z <level>]
+.B bzcat planet.osm.bz2 | maptool planet.bin
 .SH DESCRIPTION
-maptool parses osm textfile and converts it to Navit binfile format
+maptool reads OSM data and converts it to Navit binfile format.
+
+If you give an input file with \-i, maptool detects the format from the file
+name. maptool recognizes
+.BR .osm ,
+.BR .osm.gz ,
+.BR .osm.bz2 ,
+.B .pbf
+and
+.BR .o5m .
+The options \-P and \-M are not necessary for these files.
+
+maptool cannot detect the format of data from standard input. For standard
+input, maptool assumes OSM XML. The \-P option selects pbf data. The \-M option
+selects o5m data.
 .SH OPTIONS
 .TP
 \-h (\-\-help)
@@ -43,13 +52,15 @@ end at specified phase
 enable experimental features (if available)
 .TP
 \-i (\-\-input-file) <file>
-specify the input file name (OSM), overrules default stdin
+specify the input file name (OSM), overrules default stdin. maptool detects the
+format from the file name suffix
 .TP
 \-k (\-\-keep-tmpfiles)
 do not delete tmp files after processing. useful to reuse them
 .TP
 \-M (\-\-o5m)
-input data is in o5m format
+the data from standard input is in o5m format. With \-i, maptool detects the
+format, and this option is not necessary
 .TP
 \-n (\-\-ignore-unknown)
 do not output ways and nodes with unknown type
@@ -58,7 +69,8 @@ do not output ways and nodes with unknown type
 process only nodes
 .TP
 \-P (\-\-protobuf)
-input data is in pbf (Protocol Buffer) format
+the data from standard input is in pbf (Protocol Buffer) format. With \-i,
+maptool detects the format, and this option is not necessary
 .TP
 \-r (\-\-rule-file) <file>
 read mapping rules from specified file

man/maptool.1

@@ -1,24 +1,42 @@
 if(BUILD_MAPTOOL)
 
-	find_package(PROTOBUF_C REQUIRED)
-	include_directories(${PROTOBUF-C_INCLUDE_DIR})
-	include_directories(${CMAKE_CURRENT_BINARY_DIR})
-	#PROTOBUF_C_GENERATE_C(PROTO_SRCS PROTO_HDRS zfs.proto)
+	enable_language(CXX)
+
+	find_package(ZLIB REQUIRED)
+	find_package(EXPAT REQUIRED)
+	find_package(BZip2 REQUIRED)
+	find_package(Threads REQUIRED)
+
+	# libosmium and protozero are header-only. Prefer the system copies
+	# (libosmium2-dev / libprotozero-dev on Debian and Ubuntu) so distro and
+	# offline builds never hit the network; only fall back to downloading them.
+	find_path(OSMIUM_INCLUDE_DIR osmium/version.hpp)
+	find_path(PROTOZERO_INCLUDE_DIR protozero/version.hpp)
+	if(OSMIUM_INCLUDE_DIR AND PROTOZERO_INCLUDE_DIR)
+		include_directories(${OSMIUM_INCLUDE_DIR} ${PROTOZERO_INCLUDE_DIR})
+	else()
+		message(STATUS "libosmium or protozero not found on the system, downloading them")
+		# SOURCE_SUBDIR points nowhere on purpose: we only want the headers,
+		# not the libraries' own CMake projects.
+		include(FetchContent)
+		FetchContent_Declare(libosmium
+			URL https://github.com/osmcode/libosmium/archive/refs/tags/v2.20.0.tar.gz
+			SOURCE_SUBDIR non-existent)
+		FetchContent_Declare(protozero
+			URL https://github.com/mapbox/protozero/archive/refs/tags/v1.7.1.tar.gz
+			SOURCE_SUBDIR non-existent)
+		FetchContent_MakeAvailable(libosmium protozero)
+		include_directories(${libosmium_SOURCE_DIR}/include ${protozero_SOURCE_DIR}/include)
+	endif()
 
 	add_definitions( -DMODULE=maptool ${NAVIT_COMPILE_FLAGS})
 
 	add_executable (maptool maptool.c)
 	add_library (maptool_core boundaries.c buffer.c ch.c coastline.c itembin.c
-		itembin_buffer.c itembin_slicer.c misc.c osm.c osm_o5m.c osm_psql.c
-		osm_relations.c sourcesink.c tempfile.c tile.c zip.c osm_xml.c)
-
-	if(NOT MSVC)
-		PROTOBUF_C_GENERATE_C (PROTO_SRCS PROTO_HDRS osmformat.proto)
-		PROTOBUF_C_GENERATE_C (PROTO_SRCS PROTO_HDRS fileformat.proto)
-		target_sources(maptool_core PRIVATE osm_protobuf.c osm_protobufdb.c
-			fileformat.pb-c.c osmformat.pb-c.c)
-		target_link_libraries(maptool_core ${PROTOBUF_C_LIBRARY})
-	endif(NOT MSVC)
+		itembin_buffer.c itembin_slicer.c misc.c osm.c osm_psql.c
+		osm_relations.c sourcesink.c tempfile.c tile.c zip.c osm_libosmium.cpp)
+	set_property(TARGET maptool_core PROPERTY CXX_STANDARD 14)
+	target_link_libraries(maptool_core ZLIB::ZLIB EXPAT::EXPAT BZip2::BZip2 Threads::Threads)
 
 	if(NOT MSVC)
 		SET(NAVIT_LIBS ${NAVIT_LIBS} m)