This repository was archived by the owner on Jul 30, 2026. It is now read-only.
feat: add IPv6, reduce size, update to GeoLite2-City-CSV_20260220 - #120
Merged
Conversation
…table new data format uses two separate structures under a root metadata node: 1. prolly tree index: maps 128-bit IP keys to integer location IDs. content-defined chunk boundaries mean unchanged IP ranges produce identical blocks across dataset versions, so updates only regenerate blocks for changed ranges (~5-10% of total). 2. sharded location table: stores deduplicated geo data in pages of 256. root node is an array of page CIDs, lookup computes page index from location ID. ~350K unique locations stored once instead of duplicated across ~5M index entries. key changes: - unified IPv4/IPv6 support via 128-bit keys (IPv4 mapped to ::ffff: prefix) - dataset size reduced from ~260MB to ~120MB (54% smaller) - lookup fetches: 3-4 steady state (roots cached), down from 4-5 - prolly-trees library used only during generation (devDependency) - lookup traverses prolly tree blocks directly, no runtime dependency - removed bluebird, ip, lodash-es, multihashes dependencies
index entries now store [locId, endKey] instead of bare locId. lookup validates that the queried IP falls within the matched CIDR range, returning "Unmapped range" for IPs in gaps between ranges. also adds Data Structures section to README documenting the v2 format, and updates stale b-tree references in the Maintenance section.
two separate CID-keyed LRU caches (512 entries each) deduplicate block fetches across concurrent lookups. upper tree levels are heavily shared (level 0-1: 3 blocks serve all lookups). caches store promises to prevent thundering herd on concurrent requests for the same CID.
reduce DAG from 207 MB to 92 MB (-56%) and per-lookup byte cost by 21% by applying two optimizations to the v2 index: 1. merge adjacent CIDR ranges sharing the same location_id (5.0M entries -> 3.1M, 38.2% reduction) 2. store endKey as variable-length offset from startKey instead of full 16-byte value (avg 1-4 bytes vs 16) tested four variants with different fanout values (f=32/64/128) and optimization combinations. chose merge+compact with f=64 as best balance for ipfs-webui/desktop where the Peers tab resolves hundreds of IPs on initial load and thousands over the session lifetime: n=100: 211 blocks / 1,328 KB (was 231 / 1,577 KB) n=500: 700 blocks / 4,029 KB (was 791 / 5,127 KB) n=1000: 1,164 blocks / 5,896 KB (was 1,295 / 7,827 KB) f=32 would save more bytes per lookup (3,319 KB at n=500) but at the cost of 26% more blocks (886), increasing HTTP round-trip overhead. f=128 has fewest blocks (606) but wastes bandwidth on large irrelevant index nodes. f=64 reduces both dimensions consistently. see docs/dag-layout-analysis.md for full benchmark data and design rationale.
- add missing delete() method to LRUCache so rejected promises are properly evicted from the cache instead of being served on retry - fix getRawBlock to try all gateways in sequence before throwing, previously the catch block threw immediately making the loop unreachable past the first gateway
downloads latest GeoLite2-City-CSV from MaxMind via their direct download API, extracts CSVs + license files, adds to IPFS with CIDv1 and 1 MiB chunks, exports a CAR, and updates DATA_HASH in source. requires MAXMIND_ACCOUNT_ID and MAXMIND_LICENSE_KEY env vars. also updates DATA_HASH to GeoLite2-City-CSV_20260220.
regenerated index from latest dataset, updated root CID and test fixtures to match.
# Conflicts: # bin/load-fixtures.sh
Contributor
Author
|
There is significant improvement in request/total data fetched when leaving Peers screen in ipfs-webui to runs for a long time (tested against the same kubo 0.40, from two separate google chrome instances, both opened peers screen at the same time and left open for some time:
We can refine the way data is represented in the future, but its already way better on many fronts + updates geoip database. |
- DEVELOPER-NOTES.md: replace outdated fixture loading instructions with current workflow using bin/load-fixtures.sh - docs/dag-layout-analysis.md: rename to 2025-02-18_ prefix, add date to title, tighten prose for scannability
aegir generates typedoc API docs into docs/, so .gitignore excludes it. move hand-written design notes to design/ instead.
|
🎉 This PR is included in version 9.3.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds IPv6 lookup support and automates the MaxMind dataset refresh. Since IPv6 required reworking the data format, modernized the index structure along the way.
IPv6 support
Both IPv4 and IPv6 addresses are now stored in a unified 128-bit key space (IPv4 mapped into IPv6 space internally). The lookup API accepts both formats.
New data format (v2)
The index is now a prolly tree (avg fanout 64) instead of a b-tree. Locations are deduplicated into a separate table chunked into 256-entry pages, so each location is stored once instead of being duplicated
across every leaf. IP ranges are stored as 128-bit keys with variable-length end offsets, and adjacent ranges pointing to the same location are merged.
For the same source data:
DATA_HASH)GEOIP_ROOT)Thats ~63% smaller, ~75% fewer blocks.
Lookup improvements
LRU caches for index and location page blocks (avoids re-fetching during repeated lookups)
Updated dataset
Updated to GeoLite2-City-CSV_20260220 in this PR.
npm run update-datasetNew script that downloads the latest GeoLite2-City-CSV from MaxMind's API will make future updates easier.
Test