Do we ever cache the panos?

[misaugstad]

Do we ever cache the panos, or do we download them anew each time? Although neither of these are a concern right now, in the future caching could help with

1. Speed
2. Potential for rate limiting

If we aren't already, our use case seems like a great candidate for caching: users in PS are often adding many labels on the same pano in quick succession, particularly at intersections.

[misaugstad]

And we could also make use of the panos that we already have saved that are stored on makelab2!

[jsomeara]

I think this is a great idea for situations where we need response times to be fast (e.g. Label AId 2.0).

However, would it not make more sense to use a separate crop-based model there in the first place? We've done comparisons before and found that the crop-based model and depth-based system perform essentially the same.
I think the reasons we don't use the crop model for async AI validation is because:

• Technically, the depth-based system performed ever so slightly better, so might as well use that for situations where speed doesn't matter
• In the future, we might want to include whole-pano context for these async AI validations

Going back to the LabelAId stuff:

I do wonder if it's actually slower to have the user send an image of the crop over a network connection than it would be for them to just send pano id and we download it ourselves. Google's servers are probably faster than their network connection, right? Ideally, we'd use on-device AI, and that option is still very much on the table. I guess we just kinda forgot about that.

And we could also make use of the panos that we already have saved that are stored on makelab2!

This is something that I agree we should do. Though, it wouldn't help with new, unseen panos, right?

@jonfroehlich your thoughts wanted here as well.

[misaugstad]

Though, it wouldn't help with new, unseen panos, right?

Correct! Might only help for like 1/4 of data or something (I'd need to look up stats on how many labels are on repeated panos)

And remind me, our panos that we have saved on makelab2 aren't sufficient? Is that because of resolution, or because we need depth data as well? If we started recording depth, would we be able to use those?

[jsomeara]

And remind me, our panos that we have saved on makelab2 aren't sufficient?

They definitely are sufficient luckily :)

[misaugstad]

In 98f18fd I added the ability to grab our locally stored panos as a default, then to download the image as a backup. Up till now, we had been downloading the image fresh every time, and failing if an image was no longer available.

However, there are some very minor differences between the two pano download methods. sidewalk-panorama-tools uses a different endpoint, which returns 256x256 pixel tiles along the top and bottom third of the image (with 512x512 in the center), whereas this repo uses a different endpoint that returns 512x512 pixel tiles everywhere. Google must be using a different interpolation method to resize those tiles. You can't actually see any difference between the two images in practice, but below I used ImageMagick's compare tool to find differences. You can see that the middle band is identical, with changes along the top and bottom of the image.

ImageMagick Diff

RGB Images
But the images are functionally identical (pano-tools method first, this repo second):

I ran an A/B test on 993 labels to see what impact these minor differences might make on the model predictions. Of those 993 labels,

1. 935 (94.2%) had identical predictions in terms of both validation (agree/disagree/unsure) and included tags
2. 36 (3.6%) had a change in validation; though these only changed agree <-> unsure or disagree <-> unsure, none changed agree <-> disagree. The predictions are all very close, these just happened to be near the cutoff point
3. 23 (2.3%) had a change in tags the model thinks should be applied

I went through each of those labels to determine which was correct. When using the scraped panos, 15 (42%) of the 36 validation changes were right, and 20 (87%) of the 23 changes in tags were right.

It looks to me like differences in how we download the images are pretty negligible. One method isn't giving clearly better results than the other. So it seems safe enough to me to start using this in production, as it will allow us to run the AI validator/tagger on expired imagery, and it will speed up the AI API when we do hit the cache.

Any issues with that @jonfroehlich?

[jonfroehlich]

Sounds good. And great analysis. Are the images you shared representative? I assume you looked at more than one visually? (In addition to the empircal analysis you computed).

[misaugstad]

I at least looked at a handful, though it was primarily the empirical analysis!

[misaugstad]

Context from Slack:

This is the code for the AI server. When someone adds a label in PS, we send a request to the server with metadata about the label, it downloads the pano, then runs the AI validator and tagger on it, and responds with the predictions. We then also do this for a batch of older labels every night. The update makes it so that we first check our cache of panos downloaded with the scraper and read from there (taking ~2 seconds rather than ~6 seconds). And if it doesn't find a cached pano (which would be the case for new incoming data), it downloads the pano like it used to