A navigation database builder for GA (General Aviation) Experimental aircraft GPS systems, written in C++20. It imports and combines OpenStreetMap, FAA, OpenAIP, NOAA, and USGS/Copernicus data into a single PostGIS database that a homebuilt-aircraft glass panel or EFB can query directly — roads and terrain for moving-map display, magnetic declination for heading correction, charted airspace boundaries, and FAA obstacle data, alongside a full worldwide airport/navaid database.
At its core is a high-performance OpenStreetMap PBF importer (the original purpose of this project) with a custom Mercator schema, parallel processing, and efficient random-access node lookup via a memory-mapped flat file — built to handle a full-planet import, not just regional extracts.
Most users want this, not a full-planet import. A regional bundle
(<region>.gpsxdb.tar.gz) is a self-contained export — roads, terrain,
airspace, WMM declination, airports, everything — for one region of the
world (10 natural continents plus a few border corridors; see
Regional bundles below, scoped by include/Regions.h).
Given one, installing it into a fresh database is a single command (build
regional_install first — see Building, same one command
produces it and everything else here):
createdb <your_db>
psql -d <your_db> -c "CREATE EXTENSION postgis;"
./build/regional_install <region>.gpsxdb.tar.gz \
-s <your_db_server> -d <your_db> -u <your_user_id> \
--nodes-file <region>.nodes.datThat's it — the bundle's own manifest and embedded region polygon handle
the rest, and the install registers itself so osm_import -m poll
afterward automatically stays region-scoped (no separate flag; see
Region-aware poll) instead of needing the full
master database. See Regional bundles below for the
full region list, how bundles get built, and installing more than one
region into the same database.
Everything else in this README — running a full-planet osm_import,
building bundles yourself, tuning thread counts for a bulk import — is for
producing regional bundles in the first place, not for installing one.
Skip ahead to Data sources if you just want to know
what's inside before you commit to it.
- Multi-threaded node processing — parallel Mercator projection across configurable node threads
- LZ4-compressed per-thread shard files — compact on-disk node store during import, merged into a flat mmap for O(1) random coordinate lookup during way/relation processing
- Direct PostgreSQL COPY — no ORM, no temp tables, no ON CONFLICT overhead; bulk-loads via
pqxx::stream_to - Custom Mercator schema — all coordinates stored as EPSG:3857 (Web Mercator) to match tile rendering pipelines
- Normalized tag tables — separate
*_tagstables per entity type rather than hstore columns - Phase-resumable — blob-aligned PBF offset checkpoints allow restarting at any phase (nodes, merge, ways, relations, indexing, airports, FAA obstacles, WMM, airspace) without re-processing earlier phases
- Delta/replication support — applies OSC change files (
.osc/.osc.gz) with tag diffing; polling mode tracks sequences via the database - Integrated OurAirports data — loads airports, runways, frequencies, navaids, countries, and regions from OurAirports as a final step
- FAA obstacle data — all known US man-made obstacles affecting aeronautical charting (towers, antennas, wind turbines, etc.)
- Terrain elevation — USGS 3DEP (US) and Copernicus DEM GLO-30 (worldwide) raster elevation, plus derived elevation-band polygons for sectional-chart-style terrain tinting
- WMM magnetic declination — a direct port of NOAA's public WMM spherical-harmonic model, computed globally and stored as both a raster and declination-band polygons
- Charted airspace — FAA Class Airspace and Special Use Airspace (US), plus OpenAIP international airspace (rest of world)
- Post-import data quality check — spot-checks a sample of points against public records (real airport/obstacle facts, an independent WMM implementation) after each import
- GiST spatial indexes — built after all data is loaded, not during import, for maximum efficiency
- Regional bundles — export any subset of the planet as a self-contained, installable
.gpsxdb.tar.gz, with a region-aware poll process that stays current afterward without depending on the source database — see Regional bundles
| Table | Description |
|---|---|
nodes |
Point geometries with Mercator coordinates |
ways |
Linear geometries (roads, paths, rivers, etc.) |
areas |
Polygon geometries — closed ways (positive IDs) and multipolygon/boundary relations (negative IDs = -relation_id). The negative ID convention disambiguates the two since OSM way and relation IDs are separate namespaces that can share the same integers. |
roads |
Route relations tagged route=road or highway=* |
relations |
Relation geometries (MultiLineString, nullable) |
node_tags |
Key/value tags for nodes |
way_tags |
Key/value tags for ways |
area_tags |
Key/value tags for areas |
road_tags |
Key/value tags for roads |
relation_tags |
Key/value tags for relations |
| Table | Description |
|---|---|
airports |
Airports with Mercator point geometry |
frequencies |
Radio frequencies per airport |
runways |
Runway endpoints with Mercator geometry |
navaids |
Navigation aids with Mercator geometry |
countries |
Country reference data |
regions |
Region reference data |
tags |
Unified key/value tag table for airports and navaids |
| Table | Description |
|---|---|
faa_obstacles |
FAA Digital Obstacle File — all known man-made obstacles affecting aeronautical charting (towers, antennas, wind turbines, buildings, etc.) with AGL/AMSL heights, lighting, marking, accuracy codes, and point geometry |
All geometry columns use PostGIS geometry type. The default SRID is EPSG:3857 (Web Mercator). Pass -L at import time to store as EPSG:4326 (WGS84) instead.
| Table | Description |
|---|---|
terrain |
Elevation data (USGS 3DEP for the US, Copernicus DEM GLO-30 elsewhere) as PostGIS raster tiles. Query directly via ST_Value(rast, point), or use the elevation_at_point_ft/elevation_along_bearing_ft SQL functions below. |
terrain_tiles |
Tracks which tiles have been loaded, for idempotent/incremental loading. |
terrain_load also (re)creates two SQL functions on every run, for the point/corridor elevation lookups the consuming GPS application needs:
| Function | Description |
|---|---|
elevation_at_point_ft(lon, lat) |
Elevation in feet at a single point, sampled directly from terrain. |
elevation_along_bearing_ft(lon, lat, bearing_deg, distance_km, interval_m) |
Elevation samples every interval_m meters along a geodesic bearing out to distance_km, for terrain-ahead lookahead profiles. |
Loaded separately via the standalone terrain_load tool — see below.
| Table | Description |
|---|---|
wmm |
Magnetic declination as PostGIS raster tiles, one 10-degree cell per row, always EPSG:4326 regardless of the destination SRID used elsewhere. Query point declination via ST_Value(rast, point). |
wmm_cells |
Tracks which 10-degree cells have been computed, for idempotent/incremental (re)loading. |
wmm_bands |
Declination-band area polygons (band_min_deg, band_max_deg, geog) derived from wmm — a color-tint layer suited to a sectional-chart-style display. Rebuilt from scratch each time wmm_load runs (unless --no-bands). |
Loaded separately via the standalone wmm_load tool — see below.
| Table | Description |
|---|---|
class_airspace |
FAA Class Airspace (US + territories) — Class A/B/C/D/E/G surface areas, Mode-C veils, with floor/ceiling altitudes. |
special_use_airspace |
FAA Special Use Airspace (US + territories) — Military Operations Areas, Restricted, Warning, Alert, and Prohibited areas, with times-of-use. |
international_airspace |
OpenAIP airspace covering every country except the US (FAA data is authoritative there). type/icao_class/lower_unit/lower_ref/upper_unit/upper_ref are OpenAIP's raw numeric codes, not decoded to names — see include/AirspaceLoader.h. |
Loaded separately via the standalone airspace_load tool — see below.
sudo apt install \
build-essential cmake \
libboost-program-options-dev \
libexpat1-dev \
zlib1g-dev \
libpqxx-dev \
libpq-dev \
libproj-dev \
protobuf-compiler \
libprotobuf-dev \
liblz4-dev \
postgresql postgresql-contrib postgisOn Raspberry Pi, if osmium reports a missing boost library:
sudo ln -s /usr/lib/aarch64-linux-gnu/libboost_program_options.so.1.90.0 \
/usr/lib/aarch64-linux-gnu/libboost_program_options.so.1.83.0
sudo ldconfigcmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)-j$(nproc) assumes RAM scales with core count, which isn't true on every
machine (e.g. a 16-core / 10GB laptop) — several files here are memory-heavy
to compile (anything using Boost.Geometry: RegionPolygons.cpp,
RegionIndex.cpp, regional_export.cpp; anything pulling in pqxx's
heavily-templated headers), and running many of them in parallel at once can
OOM-kill cc1plus even with several GB of swap available, since the peak
memory spike from N simultaneous compiles can arrive faster than swap
absorbs it. If you hit that, cap parallelism explicitly instead:
cmake --build build -j4 # or lower, depending on available RAMOne build produces every binary this project has, including the regional bundle toolchain used in Quick start and Regional bundles below — there's no separate build step for those, they come from this same command:
build/osm_import— main importer (full-planet import, delta/poll replication)build/regional_install— installs a<region>.gpsxdb.tar.gzbundle into a target databasebuild/regional_export— exports a region's node coordinates from the masternodes.datbuild/regional_db_export— exports a region's table data from the master databasebuild/airports_load— standalone OurAirports loaderbuild/faa_obstacles_load— standalone FAA obstacle loaderbuild/terrain_load— standalone terrain elevation loaderbuild/wmm_load— standalone WMM declination loaderbuild/airspace_load— standalone airspace loader
# Create database and enable PostGIS
createdb <your_db>
psql -d <your_db> -c "CREATE EXTENSION postgis;"
# Create OSM schema
psql -h localhost -U <your_user_id> -d <your_db> -f create.sql
# Create OurAirports schema (imported automatically at end of import)
psql -h localhost -U <your_user_id> -d <your_db> -f create_airports.sqlA ~/.pgpass file is required. None of the tools in this project
accept a password on the command line or via a -p/--password flag —
every pqxx::connection and psql/raster2pgsql shell-out relies
entirely on libpq's standard .pgpass
mechanism (or peer/trust auth for local connections). Add a line to
~/.pgpass (mode 0600) for each host/database/user you'll connect as:
hostname:port:database:username:password
* is a valid wildcard for any field (e.g. myhost:5432:*:myuser:mypass
covers every database on that host for that user). Without a matching
.pgpass entry (or passwordless local auth), every tool here will fail
to connect.
Add to postgresql.conf and restart before starting the import. These
scale with your machine's actual RAM/CPU rather than being fixed values —
plug in your own numbers using the formulas below:
| Setting | Formula | Why |
|---|---|---|
shared_buffers |
15–25% of total RAM | Standard baseline for a dedicated DB server — but lean toward the low end here specifically: nodes.dat is memory-mapped and benefits far more from OS page cache than from Postgres's own buffer pool, so over-allocating this just takes RAM away from that. |
effective_cache_size |
50–75% of total RAM | Tells the query planner how much RAM is realistically available for caching (OS page cache + shared_buffers combined) — doesn't allocate anything itself, safe to set generously. |
work_mem |
total RAM ÷ (2 × thread count, i.e. -t + -w) |
Import uses one DB connection per thread, not the hundreds a typical multi-user OLTP setting has to divide RAM across, so this can be much more generous. |
maintenance_work_mem |
1–2GB, more if RAM allows | Speeds up the primary-key/GiST index builds at the end of the ways/relations phases and after -I. |
max_wal_size |
2–4GB on a fast NVMe, less on spinning disk | Larger reduces checkpoint frequency during bulk load; bounded by how much WAL churn your disk can actually absorb before a checkpoint has to catch up. |
wal_buffers |
64MB | Rarely needs to scale with machine size — Postgres's own auto-tuning (-1, the default) is also a reasonable choice. |
Worked example for a 40GB RAM / 12-core machine running -t 3 -w 12 (15
import threads total):
shared_buffers = 8GB # ~20% of 40GB
effective_cache_size = 24GB # ~60% of 40GB
work_mem = 256MB # 40GB / (2 * 15 threads), rounded down
maintenance_work_mem = 2GB
max_wal_size = 4GB
wal_buffers = 64MB
synchronous_commit = off
synchronous_commit = off is a fixed recommendation regardless of hardware
— it trades some durability (a crash can lose the last few commits) for
significantly faster bulk-load throughput, which is an easy trade during a
from-scratch import you can just re-run if interrupted.
The importer automatically disables autovacuum at startup and re-enables
it after VACUUM ANALYZE completes — you do not need to manage this manually.
# Get max node ID from PBF
osmium fileinfo -e planet-latest.osm.pbf | grep "Max ID"
./build/osm_import \
-i planet-latest.osm.pbf \
-s <your_db_server> -d <your_db> -u <your_user_id> \
-I \
-n 20000000000 \
-f nodes.dat \
-v -l osm.logDon't already have planet-latest.osm.pbf? Just omit -i and osm_import
fetches the latest one from planet.openstreetmap.org before importing,
saving it to ./planet-latest.osm.pbf (equivalent to passing
--download-planet with no -i; pass --download-planet explicitly
alongside -i <path> instead if you want it saved somewhere else). The
file is on the order of 100GB, so this takes hours — the download is
resumable (safe to re-run after an interruption or crash) and
checksum-verified against upstream's published MD5 before the import
proceeds, and it happens before -I touches the database, so a failed
download never leaves you with a wiped schema and nothing to import:
./build/osm_import \
-s <your_db_server> -d <your_db> -u <your_user_id> \
-I \
-n 20000000000 \
-f nodes.dat \
-v -l osm.log-I drops and recreates all tables before the import begins — equivalent
to running create.sql and create_airports.sql manually. Safe to use on
a fresh database or when reimporting from scratch. Do not use -I with
-R resume flags — it will wipe data from phases already completed.
The defaults (-t 1 -w 6) are conservative. On a proper x86_64
desktop/server, increase both for much higher throughput — see
Hardware notes below (bulk import needs that kind of
hardware regardless — see that section for why Raspberry Pi 5 isn't
supported for this specific mode).
| Flag | Description | Default |
|---|---|---|
-i <file> |
Input PBF file | required |
-s <host> |
PostgreSQL host | required |
-d <db> |
Database name | required |
-u <user> |
Database user | required |
-t <n> |
Node threads | 1 (see Hardware notes) |
-w <n> |
Way threads | 6 (see Hardware notes) |
-q <n> |
Queue size per node thread | 10000 |
-f <file> |
Node mmap file path | nodes.dat |
-n <id> |
Max node ID (sizes nodes.dat) |
20000000000 |
-S <dir> |
Shard file directory | . |
-l <file> |
Log file | osm_import.log |
-v |
Verbose logging | off |
-I |
Initialize schema (drop + recreate all tables) before import | off |
-L |
Store coordinates as WGS84 lon/lat (EPSG:4326) instead of Web Mercator (EPSG:3857) | off (Mercator is default) |
-R <phase> |
Resume at phase (see below) | nodes |
If an import is interrupted, resume at any phase without reprocessing earlier ones:
# Resume from merge phase (nodes already written to shards)
./build/osm_import -i planet.osm.pbf -s <your_db_server> -d <your_db> -u <your_user_id> \
-n 20000000000 -f nodes.dat -R merge
# Resume from ways (nodes.dat already fully merged)
./build/osm_import -i planet.osm.pbf -s <your_db_server> -d <your_db> -u <your_user_id> \
-n 20000000000 -f nodes.dat -R ways
# Resume from relations (ways already in DB)
./build/osm_import -i planet.osm.pbf -s <your_db_server> -d <your_db> -u <your_user_id> \
-n 20000000000 -f nodes.dat -R relations
# Re-run spatial indexing only
./build/osm_import -s <your_db_server> -d <your_db> -u <your_user_id> -R indexing
# Re-run airports loading only
./build/osm_import -s <your_db_server> -d <your_db> -u <your_user_id> -R airports
# Re-run WMM declination loading only
./build/osm_import -s <your_db_server> -d <your_db> -u <your_user_id> -R wmm
# Re-run airspace loading only
./build/osm_import -s <your_db_server> -d <your_db> -u <your_user_id> -R airspace
# Re-run terrain elevation loading only
./build/osm_import -s <your_db_server> -d <your_db> -u <your_user_id> -R terrain
# Re-run VACUUM ANALYZE only
./build/osm_import -s <your_db_server> -d <your_db> -u <your_user_id> -R vacuumResume phases: nodes | merge | ways | reindex | relations | indexing | airports | faa | wmm | airspace | terrain | vacuum
On the first successful import, two checkpoint files are written alongside nodes.dat:
nodes.dat.offset— byte offset of first non-node blob in the PBF (skips node section on resume)nodes.dat.relations_offset— byte offset of first relation blob (skips nodes + ways on-R relations)
Apply a single OSC change file:
./build/osm_import -m delta -o changes.osc.gz \
-s <your_db_server> -d <your_db> -u <your_user_id> \
-f nodes.dat -n 20000000000Continuously apply replication diffs from planet.openstreetmap.org:
./build/osm_import -m poll -r minute \
-s <your_db_server> -d <your_db> -u <your_user_id> \
-f nodes.dat -n 20000000000
# Start from a specific sequence number
./build/osm_import -m poll -r minute -Q 5123456 \
-s <your_db_server> -d <your_db> -u <your_user_id> \
-f nodes.dat -n 20000000000While polling, osm_import also checks every 6 hours whether the OurAirports
or FAA obstacle datasets have been refreshed upstream (via HTTP
Last-Modified) and automatically reloads whichever has changed — no
separate -R airports/-R faa run needed. The same 6-hour check also
triggers a WMM declination refresh every ~3 months (matching how slowly
secular variation actually drifts declination) and an airspace refresh
every month (simpler than matching FAA's ~8-week NASR/SUA publish cycle,
still current enough) — neither has a single upstream "has this changed"
signal to check the way airports/FAA obstacles do, so these are
fixed-interval instead. The last-seen timestamp for each is tracked in the
external_data_state table.
Replication granularities: minute | hour | day
Delta and poll mode both auto-detect whether they're pointed at a regional
install — no separate flag needed. If -d/--database has any rows in
public.installed_regions (populated by regional_install, see
Regional bundles below), -f/--nodes-file is read as
that install's own (much smaller) node coordinate store instead of the
master nodes.dat, and every DB write is filtered to whichever region(s)
are actually installed there. A database with more than one region
installed is handled by a single poll process covering all of them — one
download of each replication diff, not one per region. Otherwise (the
master database, or any database with no installed_regions rows), poll
mode behaves exactly as documented above.
./build/osm_import -m poll -r minute \
-s <your_db_server> -d <your_regional_db> -u <your_user_id> \
-f <region>.nodes.dat./build/faa_obstacles_load -s <your_db_server> -d <your_db> -u <your_user_id>
# Use WGS84 coordinates instead of Mercator
./build/faa_obstacles_load -s <your_db_server> -d <your_db> -u <your_user_id> -4# Run create_airports.sql first if not already done
psql -h localhost -U <your_user_id> -d <your_db> -f create_airports.sql
./build/airports_load -s <your_db_server> -d <your_db> -u <your_user_id>Loads USGS 3DEP 1 arc-second (~30m) elevation tiles into a PostGIS raster
table (terrain), queryable via ST_Value(rast, point). US-only coverage
(3DEP doesn't cover outside the US/territories). Requires the postgis
package (for raster2pgsql, already a listed dependency) and the
postgis_raster extension (created automatically by -I, or run
CREATE EXTENSION postgis_raster; manually otherwise).
Takes an explicit bounding box rather than defaulting to full-CONUS —
coverage area × resolution drives storage size directly (a few thousand
tiles, 100+ GB, for the whole lower 48). Already-loaded tiles are tracked
in terrain_tiles and skipped on re-run, so calling this again with an
overlapping or expanded bbox only fetches what's new.
./build/terrain_load -s <your_db_server> -d <your_db> -u <your_user_id> \
--bbox <min_lon>,<min_lat>,<max_lon>,<max_lat>
# Example: Colorado
./build/terrain_load -s <your_db_server> -d <your_db> -u <your_user_id> \
--bbox -109,37,-102,41
# Use WGS84 coordinates instead of Mercator
./build/terrain_load -s <your_db_server> -d <your_db> -u <your_user_id> \
--bbox -109,37,-102,41 -4After loading tiles, terrain_load (re)creates the elevation_at_point_ft/
elevation_along_bearing_ft SQL functions (see the Terrain tables section
above) so they stay current against the newly-loaded raster.
# Override the thread count
./build/terrain_load -s <your_db_server> -d <your_db> -u <your_user_id> \
--bbox -109,37,-102,41 --threads 83DEP is US-only; pass --source copernicus to load Copernicus DEM GLO-30
instead for coverage anywhere else in the world. For the whole planet at
once, pass --global instead of --bbox:
./build/terrain_load -s <your_db_server> -d <your_db> -u <your_user_id> --global --threads 10This loads Copernicus DEM GLO-30 for all 9 populated non-US regions in one
call (10 natural-boundary regions total — coastlines + physical divides,
not administrative borders, see include/Regions.h and
tools/prepare_regions.py — minus north_america, which still needs its
continental-US portion covered separately). Deliberately excludes the
continental US (3DEP is authoritative there — run a separate --source 3dep pass; the north_america region's Copernicus sweep skips just the
CONUS tile cells, not the whole region, since it also covers Canada/Mexico/
Central America/Caribbean/Alaska/Greenland) and Antarctica (minimal
Copernicus coverage, no permanent civil GA population).
Computes World Magnetic Model declination directly from NOAA's public
WMM2025 coefficients (embedded in WMMLoader.cpp, no external data file or
network fetch needed for the model itself) and loads it globally by
default — unlike terrain, there's no per-region tile source, so a full
global (re)load takes only a few seconds. The globe is divided into
resumable 10-degree cells (wmm_cells), same idea as terrain_tiles.
./build/wmm_load -s <your_db_server> -d <your_db> -u <your_user_id>
# Restrict to a bbox, override band width, or use a specific date
./build/wmm_load -s <your_db_server> -d <your_db> -u <your_user_id> \
--bbox -109,37,-102,41 --band-deg 1 --year 2027.5Loads FAA Class Airspace and Special Use Airspace (US + territories, no
API key needed) and OpenAIP international airspace (everywhere else,
requires a free API key from openaip.net —
read from ~/.openaip_api_key by default, never committed to the repo).
./build/airspace_load -s <your_db_server> -d <your_db> -u <your_user_id>
# Load just one source
./build/airspace_load -s <your_db_server> -d <your_db> -u <your_user_id> --class-only
./build/airspace_load -s <your_db_server> -d <your_db> -u <your_user_id> --sua-only
./build/airspace_load -s <your_db_server> -d <your_db> -u <your_user_id> --intl-only
# Skip international airspace (e.g. no API key available)
./build/airspace_load -s <your_db_server> -d <your_db> -u <your_user_id> --no-intlExports any subset of the planet's data (see include/Regions.h for the
region list — 10 natural continents plus border corridors for regions that
straddle a continent boundary) as a single self-contained
<region>.gpsxdb.tar.gz, installable into a fresh database without needing
the source database again afterward. Built for customers/deployments that
only want their own region's data (e.g. a single-region install on a
Raspberry Pi 5 — see Hardware notes) rather than the
full planet.
./build_regional_bundles.sh -s <host> -d <db> -u <user> \
-f <master nodes.dat path> -n 20000000000 --out-dir <dir> \
[--regions name1,name2,...] [--region-batch-size N] [-v]This runs two tools in sequence for every requested region (all of them
if --regions is omitted), then bundles the result:
regional_db_export --big-tables-only— single client-side pass over the 5 big parent+child table pairs (ways/areas/roads/relations/nodes and their_tagschildren), classifying each row against every region's real polygon in one read instead of oneST_Intersectsquery per region. Also writes each region'sextra_vertices.bin.regional_export— one pass over the masternodes.datproducing each region's own node coordinate slice, widened (viaextra_vertices.binabove) to include every vertex of any way/area/road included in that region (not just nodes whose own coordinate falls inside the region polygon — needed so a region-aware poll process can always resolve a border-crossing way's full geometry locally, see below).regional_db_export --small-tables-only— the remaining, smaller tables (airports, WMM, terrain, airspace, etc.), one per region.--region-batch-sizebounds peak disk usage for the--big-tables-onlypass (all requested regions' big-table output otherwise lands on disk simultaneously) by splitting the region list into batches.
./build/regional_install <region>.gpsxdb.tar.gz -s <host> -d <db> -u <user> \
--nodes-file <target nodes.dat path> [--work-dir <dir>] [-v]Idempotent and safe to run more than once against the same target,
including with different or overlapping regions — every row is deduped via
INSERT ... ON CONFLICT (id) DO NOTHING, and --nodes-file is merged
(not overwritten) if it already exists from a prior install. Installing a
second region into an already-populated database:
- drops the GiST spatial indexes first (not needed for the dedup above, only the primary keys are) and rebuilds them once at the end, so every install gets the same efficient one-pass bulk index build the main importer uses, rather than incremental per-row index maintenance;
- registers the region in
public.installed_regions, which is what lets poll mode auto-detect which polygon(s) to filter against afterward (see Region-aware poll) — no--regionflag needed anywhere, and a database with several regions installed is fully represented, not just the first one.
dq_check.py spot-checks a live nav database after an import: it samples
100 points across the continental US (5 major hub airports, 5 minor GA
fields, 4 named landmarks, and 86 randomly-sampled points rejection-tested
for real OSM coverage) and checks six data layers at each — nearest
airport, FAA obstacles, charted airspace (class + special use), WMM
magnetic declination, nearby roads (ways), and nearby land-use areas
(areas). WMM declination is additionally cross-checked against
pygeomag (an independent NOAA WMM
implementation, not this project's own WMMLoader.cpp), and a handful of
stable public facts are checked directly (e.g. the real height of the
KVLY-TV mast) so a future regression has a chance of being caught without
a human eyeballing the SQL results. Meant to be rerun occasionally after a
fresh import, not part of the build.
pip install -r requirements-dq_check.txt
python3 dq_check.py -s <your_db_server> -d <your_db> -u <your_user_id>
# writes dq_report.html; add --json results.json to also dump raw per-point dataInitial run (2026-07-06, after the first full-planet import): all 6
golden-fact checks passed, all 5 major hubs correctly showed Class B
airspace, WMM matched the independent model on all 100 points (mean
deviation 0.032°, max 0.095° — consistent with the underlying raster's
0.25° grid resolution, not a defect). Two standout matches against famous
public height records: the KVLY-TV mast came back at 2,060 ft AGL against
its well-known public figure of 2,063 ft, and the tallest obstacle within
5nm of the Statue of Liberty (One World Trade Center) came back at 1,792 ft
AGL against its iconic 1,776 ft spire height. Real street names surfaced
next to the White House (Pennsylvania Avenue NW, E Street NW), and nearby
airport lookups correctly found the White House South Lawn Helipad and a
closed-status Crissy Field next to the Golden Gate Bridge, matching its
real decommissioned status.
During import, a status line is printed to stdout once per second:
0:45:12 60.2% N:2.1B A:0 W:0 R:0 Q:847 M:42.3% | 487.2K/s [Nodes]
| Field | Description |
|---|---|
0:45:12 |
Total elapsed time since process start (H:MM:SS) |
60.2% |
PBF file read progress (bytes read / file size) |
N:2.1B |
Total nodes processed (running total, never resets) |
A:137.4M |
Total areas processed (running total, never resets) |
W:85.5M |
Total ways processed (running total, never resets) |
R:122.9K |
Total relations processed (running total, never resets) |
Q:847 |
Way queue depth (entries waiting to be processed by way threads) |
M:42.3% |
Merge progress — only shown during the Merging phase |
487.2K/s |
Throughput for the current phase only — computed from the count accrued since the phase began, so it reflects current speed rather than being diluted by time spent in earlier phases |
[Nodes] |
Current phase |
| Phase | Description |
|---|---|
[Nodes] |
Reading nodes from PBF, projecting to Mercator, writing to shard files and nodes |
[Merging] |
Decompressing and merging per-thread shard files into the flat nodes.dat mmap |
[Ways] |
Reading ways/areas from PBF, looking up node coordinates from mmap, writing to ways/areas |
[Reindexing] |
Rebuilding primary key indexes on way/area/road/relation tables |
[Relations] |
Processing relation members, merging geometries, writing to relations/roads |
[Spatial Indexing] |
Building GiST spatial indexes on all geometry columns |
[Loading Airports] |
Downloading and loading OurAirports data |
[Loading FAA Obstacles] |
Downloading and loading FAA Digital Obstacle File |
[Loading WMM Declination] |
Computing and loading World Magnetic Model declination, globally |
[Loading Airspace] |
Downloading and loading FAA Class/Special Use Airspace + OpenAIP international airspace |
[Loading Terrain] |
Downloading and loading 3DEP (US) + Copernicus DEM GLO-30 (rest of world) elevation |
[Vacuuming] |
Running VACUUM ANALYZE on all tables |
[Done] |
Import complete |
PBF file
│
▼
OSMReader (producer, single thread)
│ parses PBF blobs, emits OSMEntry variants
│
├──▶ node_queues[0..N]
│ │
│ ▼
│ nodeThread × N
│ toMercator() + pointWKB() ← parallelized across N threads
│ OSMMMap::insert() ← writes to per-thread LZ4 shard files
│ NavDB::insertNode() ← bulk COPY to nodes
│ │
│ node_barrier
│ OSMMMap::merge() ← decompress + merge shards into flat mmap
│ OSMMMap::setRandomAccessHint()
│
└──▶ way_q
│
▼
wayThread × M
OSMMMap::select() ← O(1) random coord lookup via mmap
buildWayGeom()
NavDB::insertWay/Area() ← bulk COPY to ways / areas
│
way_barrier
│
▼
RelationEntry processing
NavDB::getWay() ← SELECT geog FROM ways UNION ALL areas
mergeWayGeoms() ← filters to LineString/MultiLineString only
NavDB::insertRelation()
│
▼
NavDB::createGistIndexes() ← GiST spatial indexes on all geog columns
│
▼
loadAirportsData() ← downloads + loads OurAirports CSVs
The node store (nodes.dat) is a flat binary file: node_id × 16 bytes = (lon_m: f64, lat_m: f64). Node ID is the implicit array index, so select(id) is a single pointer dereference with zero search overhead.
During the node phase, coordinates are written to per-thread LZ4-compressed shard files to avoid write contention. After all nodes are processed, shards are decompressed and merged into the flat mmap in a single pass.
For a full planet import, nodes.dat is sized at max_node_id × 16 bytes (a sparse file — actual disk usage reflects only written node IDs). With -n 20000000000, the logical size is ~298 GB but actual usage is ~200 GB for a 2026 planet dump.
In -m delta/-m poll mode, nodes.dat auto-grows in place if OSM assigns a node ID beyond the file's current capacity (25%+ growth per grow event, still sparse — cheap). This is durable across restarts: reopening the file with the original -n will not lose the extra capacity, since the actual on-disk file size is always used as a floor. Passing a larger -n explicitly still works too; it's just no longer strictly required to keep bumping it by hand to avoid silently dropping new nodes.
The bulk importer (-m import) is I/O-bound, not CPU-bound: the node phase
is dominated by mmap write throughput, and the ways/relations phase by
random node-coordinate lookups from nodes.dat. It requires a proper
x86_64 desktop/server with real NVMe — see Hardware notes
for recommended specs and a known incompatibility with Raspberry Pi 5.
Delta/poll mode's much lighter, occasional I/O pattern isn't subject to
this — see that section for details.
The status line's Q: field (way queue depth) is the primary feedback
mechanism for tuning -t (node threads) and -w (way threads). The
goal is to keep the queue neither constantly empty nor constantly full.
During the Nodes phase, the producer (single-threaded PBF parser) feeds
node entries into per-thread queues consumed by -t node threads. Each node
thread does Mercator projection, mmap writes, and COPY buffering.
| Queue depth | What it means | Action |
|---|---|---|
| Consistently 0 | Producer is the bottleneck — node threads are faster than the PBF parser can feed them. Adding more node threads won't help. | Reduce -t to free up resources, or leave as-is |
| Consistently at max (10000) | Node threads are the bottleneck — they can't keep up with the producer. | Increase -t if the system can handle the extra I/O load |
| Fluctuates 0–max | Roughly balanced — this is ideal | Leave as-is |
The default -t 1 is conservative — a single node thread typically leaves
Q: sitting at 0 (producer-bound), meaning the PBF parser is the ceiling
and adding threads won't improve throughput. On capable desktop/server
hardware, try -t 3 or higher and watch whether Q: rises.
During the Ways and Relations phases, the producer feeds way/relation
entries into a single shared queue consumed by -w way threads. Each way
thread does random node-coordinate lookups from nodes.dat (mmap), geometry
building, and PostgreSQL COPY.
| Queue depth | What it means | Action |
|---|---|---|
| Consistently 0 | Way threads are faster than the producer can feed them. More threads won't help and will only increase I/O/DB pressure. | Reduce -w |
| Consistently at max (10000) | Way threads are the bottleneck — they can't process fast enough. | Increase -w if the system is stable |
| Fluctuates 0–max | Balanced — ideal | Leave as-is |
The default -w 6 is a conservative starting point. Higher values (-w 12)
increase random I/O concurrency against nodes.dat — on desktop hardware
with a fast NVMe, -w 12 or higher should be stable and beneficial; watch
Q: to confirm the increase is actually helping.
- Q: always 0 → you have more threads than the bottleneck can feed; reduce thread count or accept that you're at the throughput ceiling
- Q: always at max → threads are the bottleneck; increase thread count if the system is stable under the load
- System instability (lockups/crashes) → reduce thread count regardless of queue depth; the I/O pressure from concurrent threads, not just throughput, is what triggers hardware/driver instability on constrained platforms — see Hardware notes
- Place
nodes.daton your fastest storage (NVMe preferred) - Use
-S <dir>to place shard files on a separate drive fromnodes.datif available - PostgreSQL should be on a separate drive from
nodes.datif possible - On desktop hardware, try
-t 3 -w 12and scale from there
A kernel regression in 7.0.0-1011-raspi (Ubuntu 26.04) causes full system lockups under sustained heavy mmap write workloads during bulk import. These mitigations are in the codebase and are harmless/beneficial in general, but — see Hardware notes — did not fully resolve the issue, which is why bulk import on Raspberry Pi 5 remains unsupported:
CHUNK_SIZEreduced from 16MB to 4MB to limit write burst size- Periodic
fsync/msyncevery 64 chunks to bound dirty page accumulation MADV_RANDOMhint on the merged node file is applied after merge completes rather than at construction time (applying it early on a large sparse mapping triggers the lockup)- Conservative default thread counts (
-t 1 -w 6)
If you experience lockups during the node phase on other platforms, try reducing CHUNK_SIZE further in OSMMMap.cpp.
Bulk import (-m import) is not supported on Raspberry Pi 5. It was
tried there — dual NVMe drives attached via PCIe 2.0 — and found to be a
poor fit for this specific workload: not because of anything in the import
logic itself, but because sustained heavy I/O (large concurrent writes
during the node phase, large concurrent random reads during ways/relations)
reliably triggered full system lockups requiring a hard power cycle. This
happened across many different phases and code paths, which pointed at a
platform-level reliability issue (kernel/driver/PCIe-adapter related) rather
than an application bug. Use a proper x86_64 desktop/server (see
Recommended hardware) for bulk import.
Delta/poll mode (-m delta/-m poll) and regional_install are a
different story — their I/O pattern is occasional and small (one diff at
a time, or one region bundle load), nowhere near the sustained heavy
concurrent I/O that triggers the lockup above, and both are intended to run
on a deployed Raspberry Pi 5 unit (with an NVMe SSD attached) as part of a
regional install. Only the initial bulk import that produces a region's
data needs the hardware described below.
- Reducing
CHUNK_SIZEfor the LZ4 shard writer (16MB → 4MB) and adding periodicfsync/msynccalls bounded dirty-page accumulation and fixed lockups during the node phase. - Reducing thread counts (
-t 1,-w 6) reduced concurrent I/O pressure and allowed runs to progress significantly further before any instability. - None of this fully eliminated the lockups under all conditions — they still occurred at lower frequency, in different phases, as data volume increased.
If you're running this on anything other than a Pi — a normal x86_64 desktop or server with a proper PCIe 3.0/4.0 NVMe slot — none of the above should be necessary, and you can scale thread counts up substantially:
./build/osm_import -i planet-latest.osm.pbf -s <host> -d <db> -u <user> \
-t 3 -w 12 -n 20000000000 -f nodes.dat -v -l osm.logMinimum specs for a comfortable full-planet import:
- CPU: any modern 4+ core x86_64 CPU — this workload is I/O-bound, not CPU-bound, so raw core count/clock speed matters far less than storage
- RAM: 16GB minimum, 32GB+ recommended (helps PostgreSQL buffer/cache hit rates and OS page cache for the node mmap file)
- Storage: a real PCIe 3.0 x4 (or better) M.2 NVMe slot, directly on the motherboard — avoid PCIe 2.0 and avoid USB/adapter-based NVMe enclosures for the node store file, since random-access throughput there is the primary bottleneck during the ways/relations phases
- A second NVMe/SSD for PostgreSQL's data directory, separate from the node store file, is recommended if available
If you do hit instability on non-Pi hardware, the same mitigations apply:
reduce -t/-w, and consider shrinking CHUNK_SIZE in OSMMMap.cpp
further.
| Source | License | Notes |
|---|---|---|
| OSM planet/extracts — planet.openstreetmap.org / Geofabrik | ODbL 1.0 | Attribution + share-alike on the database itself — see note below |
| OurAirports — davidmegginson.github.io/ourairports-data | Public domain | Downloaded automatically at import time |
| FAA Digital Obstacle File — aeronav.faa.gov/Obst_Data/DDOF.zip | Public domain (US govt. work) | Daily-updated; downloaded automatically at import time |
| FAA Class/Special Use Airspace, MTR, TFR — FAA Aeronautical Information Services ArcGIS portal (adds-faa.opendata.arcgis.com) | Public domain (US govt. work) | Downloaded automatically |
| USGS 3DEP | Public domain (US govt. work) | 1 arc-second (~30m) elevation, US + territories only |
| Copernicus DEM GLO-30 — copernicus-dem-30m.s3.amazonaws.com | Copernicus Data Terms (free, attribution required) | ~30m elevation, worldwide, public AWS Open Data, no API key |
| NOAA/NCEI World Magnetic Model — ncei.noaa.gov/products/world-magnetic-model | Public domain (US govt. work) | WMM2025 coefficients, embedded in source rather than fetched at runtime |
| OpenAIP — openaip.net | CC BY-NC 4.0, with a bundling carve-out — see note below | Crowd-sourced international airspace, requires a free API key |
Everything this project imports carries its own license, separate from and unaffected by this repository's own (MIT) license on the code. Two sources need active attention before any data — bundled exports, a hosted install, anything derived from the database — is distributed to a third party, commercially or otherwise:
- OpenStreetMap data is ODbL-licensed. ODbL is commercial-use-friendly — it does not restrict charging for a product or service built on OSM data (this is how Mapbox and many others operate). It requires (1) attribution to OpenStreetMap and its contributors, and (2) that the database itself — or a substantial extract of it, which a regional bundle is — stay available under ODbL or a compatible license if you redistribute it. This applies regardless of what license covers this project's code; ODbL governs the data, not the software that produces or serves it. A short attribution notice ("© OpenStreetMap contributors, ODbL 1.0") should accompany any bundle or public-facing instance. See osmfoundation.org/wiki/Licence for the current attribution guidelines.
- OpenAIP data is CC BY-NC 4.0, but openAIP's own site terms (see openaip.net/legal) go further than bare CC BY-NC: they state third parties may ship openAIP data bundled with paid/commercial applications as long as they don't exclusively sell openAIP data, plus attribution. Since this project bundles openAIP's airspace layer alongside OSM/FAA/terrain/WMM data rather than reselling it standalone, that carve-out plausibly covers commercial regional bundles as-is — but verify the current wording at openaip.net/legal yourself before relying on it, since terms can change and this project hasn't independently confirmed it beyond what's summarized there. For zero ambiguity (or if openAIP data becomes a more central feature later), openAIP also offers a commercial license that drops the NC/share-alike restriction — contact
licensing@openaip.net.
Public-domain US government sources (FAA, USGS, NOAA/NCEI) carry no restriction and need no attribution, though citing them is good practice.
None of the above is legal advice — it's a summary of publicly stated license terms as of this writing. Get a real legal opinion before relying on it for a commercial launch.
MIT License — see LICENSE — covers this repository's code only. The data it imports and can export is licensed separately per source; see Data sources above before redistributing anything derived from the database.
Issues and pull requests welcome. The codebase targets C++20 and builds on both aarch64 (Raspberry Pi 5) and x86_64 Linux — see Hardware notes for which modes are actually supported on which platform.