Skip to content

Locally hosted tiles for benchmarks and debug pages #1552

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from

Conversation

xabbu42
Copy link
Contributor

@xabbu42 xabbu42 commented Aug 26, 2022

Locally serve tiles from mbtiles file in a included dev server.

This is a first step to fix #198. I implemented a very minimalistic tileserver to reduce additional dependencies. This seems fine to me as long as the code remains short and the number of features low. If we want a more feature complete tileserver I would prefer to use an existing implementation as a dependency instead of reimplementing our own. For now I just added a copy of the demotiles, this should be replaced by a download script for more complex free tiles.

Todos

  • script to fetch better suited free tiles instead of including a copy of the demotiles
  • change all demo pages to use locally hosted tiles
  • change all benchmarks to use locally hosted tiles

Launch Checklist

  • Confirm your changes do not include backports from Mapbox projects (unless with compliant license) - if you are not sure about this, please ask!
  • Briefly describe the changes in this PR.
  • Manually test the debug page.

@xabbu42
Copy link
Contributor Author

xabbu42 commented Aug 26, 2022

@HarelM @birkskyum @wipfli do you agree with this approach? should I continue with it?

@github-actions
Copy link
Contributor

Bundle size report:

Size Change: 0 B
Total Size Before: 204 kB
Total Size After: 204 kB

Output file Before After Change
maplibre-gl.js 195 kB 195 kB 0 B
maplibre-gl.css 9.65 kB 9.65 kB 0 B
ℹ️ View Details No major changes

@wipfli
Copy link
Contributor

wipfli commented Aug 26, 2022

I would prefer if we could avoid introducing new dependencies. Two options which come to mind is extract tiles into a repository (maybe have them in a zipfile) or use PMTiles...

@wipfli
Copy link
Contributor

wipfli commented Aug 26, 2022

But other than that I think this is going in the right direction, thanks for working on this @xabbu42!

@nyurik
Copy link
Member

nyurik commented Aug 26, 2022

I agree that this is probably not the preferred path forward. The fewer deps we have, the better, especially because we want to share testing as much as possible with the native project, as well as be able to use it in CI and on dev machines.

  • PMTiles might be a good solution once stabilized
  • Martin might be a good solution for dogfooding reasons (once it supports mbtiles/pmtiles/static files) - especially because eventually we may need to test new types of data/formats/..., and Martin is much more likely to support them from the start.
  • we may want to standardize on Just -- one of the best language-independent Makefile replacements I have seen. It still looks very much like a Makefile, except cleaner. BTW, we already use Just in maplibre-rs and it has been very successful.

@xabbu42
Copy link
Contributor Author

xabbu42 commented Aug 26, 2022

@wipfli Just so I understand: You would want to get rid of the tileserver part and the sqlite3 dependency by unpacking the tiles localy in the filesystem and just let the st handle serving the tiles? pmtiles or zip are only possible formats to download and unpack the tiles?

@wipfli
Copy link
Contributor

wipfli commented Aug 26, 2022

Yes. We could store tiles in a normal folder structure a bit like in https://github.com/maplibre/demotiles. To make git stuff faster, we could store the tiles in a zip file and unpack that in the npm pre-install hook or something. But maybe we do not even need the zip, we might get away with just putting tiles in the directory... Cloning time is probably anyway dominated by the long commit history and not by the number of file...

@xabbu42
Copy link
Contributor Author

xabbu42 commented Aug 26, 2022

@nyurik I believe this changes are separate from automated testing, where this topic is already solved. Also, as far as I can tell, native has its own benchmarks completely separate from maplibre-gl-js. Are there any plans to integrate manual testing with debug pages and benchmarks between native and js?

Also I'm not quite sure where Just comes into the picture. Can you elaborate?

@xabbu42
Copy link
Contributor Author

xabbu42 commented Aug 26, 2022

@wipfli I see two (possible) downsides to that approach.

  1. it makes it harder for an individual developer to use larger test files or a different region. for example I might want to look at a map of my own hometown when testing something for familiarity. Ideally I could just download a file from the region I want for example from https://data.maptiler.com/downloads/europe/switzerland/ and use that directly.

  2. mbtiles are compressed and the current code just passes the compressed data on to be decompressed by the browser. I'm not sure we could achieve that with just files and st, but maybe I'm wrong?

@wipfli
Copy link
Contributor

wipfli commented Aug 26, 2022

I am a bit tempted to just run the benchmarks and use the debug pages a bit with some sort of network request logger and then put all the tilea that were requested into a folder. Somewhow that might not be that many tiles is my guess...

@wipfli
Copy link
Contributor

wipfli commented Aug 26, 2022

Question about attribution: If we store OpenMapTiles data in MapLibre GL JS, do we have to mention it somewhere in the license or so?

@HarelM
Copy link
Collaborator

HarelM commented Aug 26, 2022

I would like to avoid adding more dependencies as well.
st and static files should be fine in theory.
If you can share where static file fails we can try and think how to solve this.

@xabbu42
Copy link
Contributor Author

xabbu42 commented Aug 27, 2022

@HarelM I stand by the downsides of static files I already mentioned. Probably there is a way to solve the second one though?

@wipfli how would the commited tiles be updated? or would we just live with old maps?

I generally would prefer not to include a copy of the data itself in the repo but to have a script to download it. This avoids the problem with attribution completely, allows us to use larger test data without bloating the repo and automatically ensures we have current data.

@HarelM
Copy link
Collaborator

HarelM commented Aug 27, 2022

I'm not sure I understand this thread anymore.
The proposed PR is problematic as was stated by some people here.
I don't want us to implement a tile server to serve MBtiles, which is what the current PR is suggesting.
There's no problem to use a different tile server hosted wherever you want, as long as it's hosted all the time and the data doesn't change (which I think isn't realistic).
The concept of using a different source isn't what we need in terms of testing, and thus produces a requirement that is not relevant (from my point of view).
Bottom line, let's keep things simple as they were, and better define the problem before we go and implement anything...

@xabbu42
Copy link
Contributor Author

xabbu42 commented Sep 2, 2022

Perhaps we are mixing too many different orthogonal questions here. As far as I see @klokan (tileserver-gl-lite) and @nyurik (Martin) suggested external tileservers as a dependency, where as @wipfli and @HarelM are in favor of a solution using static tile files.

As nobody beside me considers implementing a minimalistic tileserver in maplibre-gl-js to reduce dependencies worthwhile, I will close this pull request.

@nyurik
Copy link
Member

nyurik commented Sep 2, 2022

@xabbu42 thx, let me try to parse through those:

  • locally stored static files still require some "server", even as simple as python -m http.server. This approach is very simple, but it does limit to the most basic testing (no custom headers or any other extra features we may want to test)
  • tileserver-gl-lite is a good peace of software, but it hasn't been maintained in a long time. Its functionality is essentially it is near-identical to the static files approach except that the data is inside mbtiles
  • Martin is part of MapLibre ecosystem, but it does not yet support anything except PostgreSQL, making it unsuitable at least in the near term. I am working on it, but cannot give any timeline.

Given the above, I would recommend the python -m http.server as the quickest to get started with, and eventually switch to Martin for the purposes of integration testing with our other project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

more complex demotiles for benchmarks and debuging
4 participants