Optimise `tagLocations`

[daemon1024]

When we first open up the map endpoint, first and default markers that load on maps are tag locations.

But when we start interacting with the map, it starts sending loads of requests to the API.

pl-tagloc.mp4

Which causes throttles eventually

We could debounce queries on these events. ( cc @jywarren )

Potential source for these event triggers

plots2/app/assets/javascripts/leafletHelper.js

Line 154 in f3ec8ff

map.on('load viewreset resize zoomend moveend', contentMap);

plots2/app/assets/javascripts/leafletHelper.js

Line 54 in f3ec8ff

function contentLayerParser(map, markers_hash, map_tagname) {

Ref #9698

[daemon1024]

Well, it may help somewhat but I don't think debounce is helping as much as I thought it would.

On a wider map, the endpoint itself is slow.

But unlike our findings in #7556 ,

I don't see any particular SQL bottlenecks. More like I don't see anything useful in Skylight.

On switching to allocation insights,

I see a lot of new events which are hogging up a lot of time, will check it out.

Link for skylight info of the screenshots - https://oss.skylight.io/app/applications/GZDPChmcfm1Q/1627225620/5m/endpoints/Srch::Search%20%5BGET%5D%20srch%2Ftaglocations?mode=memory

[jywarren]

One clue to figure out is, when you change the limit does it vastly affect the speed, as @17sushmita noted in the call today? That might make sense, but if it doesn't much help, a question is, does limit actually reduce query time or are we just adding a limit near the end of a chain of queries and we aren't getting much benefit from it?

If a low limit does help a lot, we could think about hitting the endpoint with a low limit, but either randomizing it or maybe better yet paginating it by recency or some other sorting function? Then as time goes on we hit the same endpoint but with page=1, page=2 that kind of thing, so we slowly build up with shorter queries. That could look a bit weird but would be like debouncing - spacing out queries giving the DB time to recover memory? Testing would tell if this actually helps overall.

I would like to see @17sushmita and you both try out this idea of truncating longitude/latitude queries to see if this improves our load time. The window for data we fetch would be a bit different then the viewed region. You can learn a little about this kind of truncation for a different purpose (privacy) here: https://github.com/publiclab/leaflet-blurred-location but it also results in better database optimization with some caveats. We could discuss this approach in a new issue if you both prefer?

[daemon1024]

One clue to figure out is, when you change the limit does it vastly affect the speed, That might make sense, but if it doesn't much help, a question is, does limit actually reduce query time or are we just adding a limit near the end of a chain of queries and we aren't getting much benefit from it?

Limits are helping for larger boundary boxes, It's somewhere between 2x-4x slower when I removed the limit.

One thing that intrigues me is
https://unstable.publiclab.org/api/srch/taglocations?nwlat=52&selat=-52&nwlng=-150&selng=150

I tried this query, which took around 47s to load but since there's no limit this should be somewhere near the max amount of taglocations we can get?

and we have a fully loaded map now which is more loaded than I have ever seen :P

I am not sure how to put it, but there's a big time tradeoff initially but this is the most content we have ever seen. What if we can cache this specific content and send it out initially and later on update the cache in the background, that should eliminate the time tradeoff

[jywarren]

Comparison: https://publiclab.org/unearthing-pvd

[daemon1024]

Provided we cache things, this much of data won't be easy to navigate and might as well cause some client side slowness ( like you noted about for the link you referred which needed webgl? I am not sure if I understood that correctly), to mitigate that we can try map clustering. An Example for that.

[jywarren]

Handling 15k responses wasn't too bad with the gl based layer. I don't honestly think we'll get that high given the optimizations possible, but you can see how it's implemented in the code here - not too bad!

https://github.com/publiclab/leaflet-environmental-layers/blob/7782b57a8ae0a08134d32b7c29a09f0e5923f518/src/unearthing.js#L35-L61

[daemon1024]

Here's what the current unstable looks like which loads the markers under 20ms with 0b5080e

simplescreenrecorder-2021-08-06_02.20.57.mp4

What I did is, I am storing the json data based on response from https://publiclab.org/api/srch/taglocations?nwlat=52&selat=-52&nwlng=-150&selng=150&limit=1000 under static/tagLocStore.json and referencing the raw github content in our map. ( We can update the box boundary to cover max of the map and maybe increase our limit if needed )

But we can't really keep the data once and forget it, so I made a github scheduled action in #9997 which will get the data from our endpoint every hour and commit the changes if any directly onto our main branch. We can choose to keep it on a seperate branch if it's not suitable to commit onto main branch, since we can fetch the raw json content anyway.

What do you think about this?

[jywarren]

This looks pretty cool! Could we achieve something similar with action caching? https://guides.rubyonrails.org/caching_with_rails.html

I know it's precaching though so I guess we could have a script that prefetches the cache daily...

Also, would there be scaling issues as the number of markers increases?