Skip to content

core: add CustomVectorSource for binary tile data delivery#4377

Open
Vladyslav-Odobesku-Intellias wants to merge 8 commits into
maplibre:mainfrom
Vladyslav-Odobesku-Intellias:custom-vector-source-upstream
Open

core: add CustomVectorSource for binary tile data delivery#4377
Vladyslav-Odobesku-Intellias wants to merge 8 commits into
maplibre:mainfrom
Vladyslav-Odobesku-Intellias:custom-vector-source-upstream

Conversation

@Vladyslav-Odobesku-Intellias

@Vladyslav-Odobesku-Intellias Vladyslav-Odobesku-Intellias commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Introduces CustomVectorSource — a new source type that lets applications deliver pre-encoded binary tile data (MVT) directly to the rendering pipeline, bypassing the GeoJSON encode/decode round-trip of CustomGeometrySource.

  • Core C++ API with fetch/cancel callbacks and async setTileData/setTileError/invalidateTile
  • Android/Kotlin platform binding using coroutines for ergonomic async tile fetching
  • Actor-based concurrency with cooperative cancellation
  • Reuses existing well-tested MVT infrastructure instead of introducing yet one more binary tile data format

Design Proposal

Included in this PR: design-proposals/2025-07-05-custom-vector-source.md

Motivation

Existing options for programmatic tile delivery have limitations:

  • URL-based vector sources require HTTP
  • CustomGeometrySource operates on GeoJSON (encode/decode overhead), uses bounding-box requests (unclear tile semantics), and manages its own threadpool (vs. Kotlin suspendible aka coroutines)

CustomVectorSource fills the gap for applications that generate tiles on-device, decode from proprietary archives, or stream over non-HTTP transports.

Here is the description of what problem we are solving by introducing such data source. Newer Google Maps SDK provides tile data through GRPC service. That service is launched along the app as a separate process and simply serves tiles through GRPC in its` own format that resembles vector tiles. We cannot request it to service tiles through GRPC + in the vector tiles format. So we need to adapt it. Proposed interface declares a simple interface for the tile data provider to implement.

interface CustomVectorTileProvider {
    suspend fun fetchTile(z: Int, x: Int, y: Int): TileData
}

It everything. Only says that this function is suspendable (coroutine) and under the hood it can be anything - some exotic HTTP tile service or as in our case some GRPC service. For us to now adapt such GRPC service to MapLibre is just about implementing CustomVectorTileProvider by doing a suspendable call to GRPC, which will naturally be suspended and then transform data from one binary format to vector tiles.

Check it yourself
This change also adds one entry to the test scenarios of the android demo app. It simply injects trivial CustomGeometrySource that provides cross tile data (2 lines that form a cross) and sets the style of those lines to be red.

AI Disclosure

AI assistance (Claude) was used for drafting the design proposal document and PR description. All code and architectural decisions are human-authored.

@github-actions github-actions Bot added build Related to build, configuration or CI/CD android core Changes that affect the C++ core of MapLibre Native labels Jul 6, 2026
Comment thread design-proposals/2025-07-05-custom-vector-source.md Outdated
Comment thread design-proposals/2025-07-05-custom-vector-source.md Outdated
Comment thread design-proposals/2025-07-05-custom-vector-source.md Outdated
@sjg-wdw

sjg-wdw commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Thoughts @TimSylvester @alexcristici @adrian-cojocaru ?

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Bloaty Results 🐋

Compared to main

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.5%  +876Ki  +0.4%  +145Ki    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results/pr-4377-compared-to-main.txt

Compared to d387090 (legacy)

    FILE SIZE        VM SIZE    
 --------------  -------------- 
   +55% +64.1Mi  +474% +28.3Mi    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results/pr-4377-compared-to-legacy.txt

@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Benchmark Results ⚡

Benchmark                                                          Time             CPU      Time Old      Time New       CPU Old       CPU New
-----------------------------------------------------------------------------------------------------------------------------------------------
OVERALL_GEOMEAN                                                 -0.0002         -0.0002             0             0             0             0

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/benchmark-results/pr-4377-compared-to-main.txt

@Vladyslav-Odobesku-Intellias

Copy link
Copy Markdown
Collaborator Author

@louwers I've removed mentions of GRPC.

  • I've made design proposal to sound more generic.
    As for the reasoning and description of how did we come up needing such custom source I've instead put it into this pull request description so that it does not pollute the design doc.

@louwers

louwers commented Jul 7, 2026

Copy link
Copy Markdown
Member

@Vladyslav-Odobesku-Intellias

We will discuss this during the Technical Steering Comittee Meeting tomorrow, 7:30PM CEST.

Do you want to join? If so, please e-mail team@maplibre.org and I will send you an invite.

Vladyslav Odobesku and others added 5 commits July 9, 2026 11:41
Introduces a new vector tile source type that accepts pre-encoded MVT
tile data via a Kotlin coroutine-based provider interface, bypassing
the internal HTTP server bridge.

C++ core: CustomVectorSource, CustomVectorTileLoader (actor-based),
CustomVectorTile (GeometryTile subclass), RenderCustomVectorSource.

Android JNI bridge: fetchTile/cancelTile callbacks via AttachEnv,
binary data transfer via GetArrayRegion.

Kotlin API: CustomVectorTileProvider (suspend interface), TileData
(sealed class), CustomVectorSource (coroutine-per-tile with
ConcurrentHashMap-based cancellation).

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Generates diagonal cross patterns per tile as MVT data using a hand-
rolled protobuf encoder. Toggle the source on/off with the FAB.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
- setTileError now erases tileCallbackMap + dataCache so errored tiles
  can be retried; previously the stale entry blocked re-fetches permanently
- cancelTile now erases tileCallbackMap so panning back to a cancelled
  tile correctly re-issues the fetch
- CustomVectorTile::setTileError delegates to GeometryTile::setError so
  loaded=true is set; previously TilePyramid::isLoaded() would stall
  permanently after any tile error
- fetchTile uses CoroutineStart.LAZY + stores job before start to close
  the race where onRemovedFromMap could miss an in-flight coroutine;
  finally block uses remove(key, value) to avoid evicting a newer job
- setTileData treats empty byte arrays as null (no data) instead of
  passing a 0-byte buffer to the MVT parser
- Add default: assert branch to TileDataFormat switch
Co-Authored-By: Claude <noreply@anthropic.com>
@Vladyslav-Odobesku-Intellias
Vladyslav-Odobesku-Intellias force-pushed the custom-vector-source-upstream branch from 06837e5 to 8352845 Compare July 9, 2026 08:41
@Vladyslav-Odobesku-Intellias Vladyslav-Odobesku-Intellias changed the title feat: add CustomVectorSource for binary tile data delivery core: add CustomVectorSource for binary tile data delivery Jul 9, 2026
@louwers

louwers commented Jul 21, 2026

Copy link
Copy Markdown
Member

Hi @Vladyslav-Odobesku-Intellias could you merge in main? That should resolve the CI issue.

Also there is a merge conflict.

# Conflicts:
#	src/mbgl/renderer/render_source.cpp
@github-actions

Copy link
Copy Markdown
Contributor

Bloaty Results (iOS) 🐋

Compared to main

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +0.3% +51.1Ki  +0.3% +48.0Ki    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results-ios/pr-4377-compared-to-main.txt

@Vladyslav-Odobesku-Intellias

Copy link
Copy Markdown
Collaborator Author

Hi @Vladyslav-Odobesku-Intellias could you merge in main? That should resolve the CI issue.

Also there is a merge conflict.

done, all green

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

android build Related to build, configuration or CI/CD core Changes that affect the C++ core of MapLibre Native

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants