Add protocol worker

[HarelM]

Hi @msbarry

I've finished the implementation of this approach.
I thought about creating a different repo and different npm package for this, but due to how maplibre-conrour package is bundled (shared worker code etc), it's a bit problematic.
Also I think having it here would make sense as this use some internal utilities methods.

The idea here is similar to how RTL plugin works, where you host a js file and import it in a worker.
The code around how I use it in my website is the following, it's fairly simple I think:

So I added a target in the rollup to take the code from add-protocol-worker and bundle it separately.

        const workerCode = await fetch("./add-protocol-worker.js"); // get the file (can be from unpkg CDN for example if this is merged)
        const workerCodeText = await workerCode.text();
        const workerUrl = URL.createObjectURL(new Blob([workerCodeText], { type: "application/javascript" }));
        getGlobalDispatcher().registerMessageHandler("contour-worker" as any, async () => { // after the worker code initializes it sends a message to the main thread.
            getGlobalDispatcher().broadcast("contour-worker" as any, {
                demUrlPattern: "another-protocol://global.israelhikingmap.workers.dev/jaxa_terrarium0-11_v2/{z}/{x}/{y}.webp",
                encoding: "terrarium",
                maxzoom: 11
            });
        });
        importScriptInWorkers(workerUrl); // this will run the code in the worker, I think that using it from CDN will even make this shorter and simpler.

This requires the latest additions I added to maplibre-gl-js in version 5.23.

I've also moved the parseUrl to the utils file.

I can add instructions on how to use it to readme with the above code to help people use it.

Do let me know what you want to do.

[HarelM]

I now see that my branch was not in an ideal state, let me know if I need to do some git magic to solve that.

[msbarry]

I think this looks good! Since it's just repackaging the code in a different distribution format and doesn't change bundle size much I think it makes sense to just keep in this repo. Would you mind adding instructions for how to use this to the readme?

[msbarry]

If this integrating more directly with maplibre, any chance we could let it return the raw vector tile without having to encode it first (since it gets immediately decoded right afterwards by maplibre) ?

[HarelM]

I don't think there's an easy way to do that right now. But I can look into it as a future improvement.

[HarelM]

Sure, I'll add those, wanted a quick look before I invest time in the docs.

[msbarry]

Do you think this should be the recommended way to integrate with mapibre and replace the setupMaplibre utility?

[HarelM]

That's a good question. I'm not sure to be honest.
The setupMaplibre doesn't require any complicated worker setup which is nice, especially when it comes to packaging, which is the more common case nowadays.
I'll write it up and let you decide where to place it.
I think both instructions are valuable.

[HarelM]

I've added the instructions.
I'm not sure what to do about the way to expose the url encoding method.
The current package can't be treeshaked, similar issue to maplibre, so I currently hard coded the dem-contour url address in the example, but it's not pretty...

Also let me know if you think I should change the output file name, maybe it should be maplibre-controur-worker.js instead of add-protocol-worker? I don't know...

In any case, changing the name is fairly simple...

I think it's a good addition, even if it's a bit rough, I'll be happy to support any issues related to it obviously, just tag me in the future.

It's now live in production in Mapeak.com and the Mapeak app, it has great performance, thanks to you!

[HarelM]

Nice! Thanks! Let me know when a new version is available 😎

[lhapaipai]

Hi and thank you for this great feature,
I'm wondering why the dem-shared protocol is not implemented for this feature? We could benefit from the LRU Cache? Or maybe I missed something?

// add-protocol-worker.ts
(self as any).addProtocol(
  "dem-shared",
  async (request: any, abortController: AbortController) => {
    const [z, x, y] = parseUrl(request.url);
    const data = await localDemManager.fetchTile(z, x, y, abortController);

    return { data: data.data.arrayBuffer() };
  },
);

[HarelM]

I'm not too familiar with all the details of this library, but dem-shared should be working with this worker.
This worker code is substituting getTile with maplibre's makeRequest here:

maplibre-contour/src/add-protocol-worker.ts

Line 17 in dc18c0f

getTile: async (url: string, abortController: AbortController) => {

So if you register a protocol for shared DEM, and you point your contour source and other sources to use it, I think it should be working as expected.
I'm not using this shared-dem as I think the browser can do most of the caching work, but I think it should be working.
Have you tested it?

[lhapaipai]

Hi and thank you for your answer,
Indeed, I've tested with browser caching and the worker requests are fetched from the cache.

Unfortunately, I haven't been able to register a "dem-shared" in the add-protocol-worker.ts

(self as any).worker.actor.registerMessageHandler(
  "contour-worker" as any,
  async (
    mapId: string | number,
    params: DemManagerRequiredInitializationParameters,
  ) => {
    const localDemManager = new LocalDemManager({
      demUrlPattern: params.demUrlPattern,
      cacheSize: params.cacheSize ?? 100,
      timeoutMs: params.timeoutMs ?? 10_000,
      encoding: params.encoding,
      maxzoom: params.maxzoom,
      getTile: async (url: string, abortController: AbortController) => {
        const request = {
          url,
          type: "arrayBuffer",
        };
        const { data } = await (self as any).makeRequest(
          request,
          abortController,
        );
        return {
          data: new Blob([data]),
        };
      },
    });

    (self as any).addProtocol(
      "dem-contour",
      async (request: any, abortController: AbortController) => {
        const [z, x, y] = parseUrl(request.url);
        const options = decodeOptions(request.url);
        const data = await localDemManager.fetchContourTile(
          z,
          x,
          y,
          getOptionsForZoom(options, z),
          abortController,
        );
        return { data: data.arrayBuffer };
      },
    );
+
+    (self as any).addProtocol(
+      "dem-shared",
+      async (request: any, abortController: AbortController) => {
+        const [z, x, y] = parseUrl(request.url);
+        const data = await localDemManager.fetchTile(z, x, y, abortController);
+
+        return { data: data.data.arrayBuffer() };
+      },
+    );
  },
);

in my reproduction in the main thread

const dispatcher = maplibregl.getGlobalDispatcher();

dispatcher.registerMessageHandler("contour-worker" as any, async () => {
  await dispatcher.broadcast("contour-worker" as any, {
    demUrlPattern: "https://tiles.mapterhorn.com/{z}/{x}/{y}.webp",
    encoding: "terrarium",
    maxzoom: 12,
  });

  map.addSource("demSource", {
    type: "raster-dem",
    encoding: "terrarium",
    tiles: ["dem-shared://{z}/{x}/{y}"],
    tileSize: 512,
    maxzoom: 12,
  });
  map.addLayer({
    id: "hills",
    type: "hillshade",
    source: "demSource",
  });
  map.addSource("contourSource", {
    type: "vector",
    tiles: [
      "dem-contour://{z}/{x}/{y}?contourLayer=contours&elevationKey=ele&levelKey=level&multiplier=3.28084&overzoom=1&thresholds=11*200*1000~12*100*500~13*100*500~14*50*200~15*20*100",
    ],
  });
  map.addLayer({
    id: "contours",
    type: "line",
    source: "contourSource",
    "source-layer": "contours",
  });
});

maplibregl.importScriptInWorkers(
  "http://localhost:5173/add-protocol-worker.js",
);

const map = new maplibregl.Map({
  container: "map",
  zoom: 13,
  center: [4.9, 44.9],
  hash: true,
  style: { version: 8, sources: {}, layers: [] },
});

It's not enough because maplibre requires that the dem-shared is registered on the main thread side. In this case there is no benefit because the LocalDemManager instance and its cache are declared on the worker side.

So I will follow your advice and not implement the dem-shared protocol, relying only on the browser cache

[HarelM]

I don't think the code above is using dem-shared correctly:
The following code will always fetch it from the network:

await dispatcher.broadcast("contour-worker" as any, {
    demUrlPattern: "https://tiles.mapterhorn.com/{z}/{x}/{y}.webp",
    encoding: "terrarium",
    maxzoom: 12,
  });

In order to utilize dem-shared you need to configure demUrlPattern to use it.
But I think the problem with this approach is that if you use raster-dem source it might not send the request to the worker to fetch the tile as I think raster sources fetch the tiles on the main thread.

So you can configure a localdemsource on the main thread for caching, but again, I think it might be an overkill, IDK...

[lhapaipai]

as I think raster sources fetch the tiles on the main thread

Completely agree, I thought they were fetched from the worker but in the test above there is an unresolved request made on the main thread with the dem-shared protocol. My example above was wrong and using 2 localDemSource removes all the benefit we could have gained from it.

[HarelM]

@msbarry note that maplibre-gl-js 6 is now in pre-release and has dropped support for commonjs in favor of esm, so it might be worth waiting a bit with the release of this and make it esm (if it's not already).

[msbarry]

Re: shared DEM source, I think to make that work we'd need the request to bounce back to the main thread so they could be cached there, and we'd also need to install maplibre contour code on both the main and worker threads...

@msbarry note that maplibre-gl-js 6 is now in pre-release and has dropped support for commonjs in favor of esm, so it might be worth waiting a bit with the release of this and make it esm (if it's not already).

Sounds good, I might end up making the current version (0.1.7) compatible with maplibre <6 and a new 0.2.0 version compatible >= 6, unless I'm able to get 0.2.0 to be compabile with both ...

[HarelM]

I've updated the code in maplibre to avoid the error in case of unknown worker-main-thread message type here:

• Remove error on unregistered actor message type maplibre/maplibre-gl-js#7589

I haven't tested it, but I hope it would work. I need to see if I can try out version 6 with my code somehow.
Once I have a working version and there are things in this package that require changes I'll open a PR.

Note that maplibre-gl 6 uses rolldown and the build is almost instantaneous. We have dropped the wiring of the worker code inside the package and now there's a two step requirement to initialize maplirbre (we'll see how much heat we'll get on that once we release this as non pre-release).
So maybe for version 6 it might be OK to only have the code here assuming it can do the same as the previous code and avoid all the packaging complexity and all the actor infrastructure. IDK... there's time to experiment while version 6 is still in pre-release...