Skip to content

README.md

Latest commit

 

History

History
78 lines (62 loc) · 2.93 KB

README.md

File metadata and controls

78 lines (62 loc) · 2.93 KB

ReadSB to JSONL Converter

This Rust program converts flight history data written to .bin.ttf files by readsb into JSON Lines format. These .bin.ttf files are usually consumed by the tar1090 flight-tracking web interface but are also suitable for storage and exchange due to their compact data format.

Build the program using Cargo by executing cargo build --release.

Assuming you have a binary flight history file at data/10.bin.ttf, you can now convert it to JSON Lines by executing target/release/readsb-to-jsonl < data/10.bin.ttf > data/10.jsonl.

Use xargs to parallelize the conversion of a larger set of files like this:

printf '%s\n' data/*.bin.ttf \
| xargs -n1 -P$(nproc) sh -c \
  'exec target/release/readsb-to-jsonl < "$0" > "${0%.bin.ttf}.jsonl"'

This can comfortably convert one day’s worth of data (about 1.2 GB) within 15 seconds on a reasonably modern machine.

Analysis in DuckDB

DuckDB is an embedded database optimized for analytical workloads. It is an excellent choice for locally querying flight data because of its ease of use, high performance, and support for storing and indexing spatial data. It can also be used to transform data and write it to formats like JSON, CSV, and Parquet. This repository contains a few example queries to help you get started.

Install the spatial extension by executing duckdb -c 'INSTALL spatial'.

DuckDB can import data directly from JSON (Lines) files. To import your data into a DuckDB database, execute the following command:

cat data/*.jsonl | duckdb history.duckdb -f duckdb/import.sql

This may take a minute or two. Thanks to the index on the point column, we can now efficiently filter by geographic area. Execute the following command to list all position updates in the area of Bermuda Island (these planes are now presumably inhabited by marine life), convert them to GeoJSON using jq, and copy the resulting data to your clipboard:

duckdb history.duckdb -init duckdb/in-bounds.sql -c '
  EXECUTE in_bounds(32.242, -64.958, 32.401, -64.622);
' | jq -sc '{
  "type": "FeatureCollection",
  "features": [.[] | {
    "type": "Feature",
    "properties": {},
    "geometry": { "type": "Point", "coordinates": .}
  }]
}' | wl-copy

This should take at most a few hundred milliseconds for one day’s worth of data. You can visualize the result by pasting it into the geojson.io web application.

Apache Parquet is another compact data format commonly used in analytical processing. Write all position updates from your DuckDB database to a Parquet file by executing this command:

duckdb history.duckdb -f duckdb/convert-parquet.sql > history.parquet