Right now, we're just using the panos that the GSV JS API says are linked to the current pano, but there are better ones to use in some cases. @jonfroehlich has mentioned using other methods in his StreetViewAi paper. From a Slack message from him:
We did a brute force grid search around each pano and added in GSV neighbors along unique directions if they weren’t in the network
This should help to fill in some places where navigation arrows are missing and users are forced to use the compass messages or click the Stuck button right now.