Add a configurable overlap window to the page cursor

[VillePihlava]

Is your feature request related to a problem? Please describe.

When using paging, because the next search is done right after the current search window (or before depending on the direction), some useful itinerary results can be left out. This happens because real time updates can change departure and arrival times. I created a visualization of this:

Goal / high level use-case

The goal is to use a best effort approach by adding a configurable overlap window (see the above picture) to try to include itineraries that might be left out otherwise.

Describe the solution you'd like

There has been discussion about how to achieve this, but an approach has not been agreed on. Adding an overlap field to the page cursor should be trivial, but the main problem is the deduplication of itineraries. After thinking about this I came up with two approaches to deduplication, but I'm not fully convinced by either one:

• Using information from the already existing itineraryPageCut with the overlap window taken into account to determine what to deduplicate.
  • This runs into the problem that an itinerary existing on the previous page and inside the overlap window might get shown again on the next page
  • This should be easy to implement
• Deduplication by hashing itineraries from the previous page
  • How many itineraries should be hashed is debatable
  • Figuring out unique hashes can be hard
    • Times should ideally not be used because real time data can change them
    • Using only e.g. generalizedCost, numOfTransfers, etc. would not be enough because an itinerary that uses a later departure of the same route might have the same hash
    • Creating a hash based on the trip ID would be useful, however, the trip ID can change when real time data is used
      • Can trip IDs generated from real time data be used for hashing?

cc @optionsome @t2gran @tkalvas

[t2gran]

We talked about this on todays developer meeting:

• The best way to hash transit itineraries is to hash TripOnServiceDate (DatedServiceJourney) ids, if it does not exist for a leg, then the trip (ServiceJourney) id can be used. Including the service-date will probably fix a problem we have in the itinerary filter chain today - for long distance travel the search window may span more then 24 hours and we filter out trips with different service-dates (same trip id). PS! The search-window can be more than 24 hours.
• We can include a hash for all itineraries in the search-window independent of the overlap of search-windows. There is probably no reason not to. This will still cause potensial misses/duplicates, but we agree on not including the "whole" search history in the token.
• The overlapping window is needed to prevent misses.
• No matter what we do, there will be cases we will not catch and the only way to be 100% sure to remove duplicates is to do it client side. A overlapping-search-window is needed to prevent misses.