OBA's way to deal with schedule_relationship: NO_DATA or SKIPPED at a stop doesn't comply to GTFS-RT spec

[kalon33]

When a stop schedule_relationship is set to NO_DATA, current OBA (1.1.12 to 1.1.18 were tested) output an error and no data from the feed is applied:

2018-05-25 23:59:42,683 WARN  [GtfsRealtimeTripLibrary.java:235] : unknown/total trips= 102/346
2018-05-25 23:59:42,685 WARN  [GtfsRealtimeSource.java:401] : Error updating from GTFS-realtime data sources
java.lang.IllegalStateException: expected at least an arrival or departure time or delay for update: stop_id: "StopPoint:40:41"
schedule_relationship: NO_DATA

        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeTripLibrary.getTimeForStopTimeUpdate(GtfsRealtimeTripLibrary.java:534)
        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeTripLibrary.getBlockStopTimeForStopTimeUpdate(GtfsRealtimeTripLibrary.java:492)
        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeTripLibrary.applyTripUpdatesToRecord(GtfsRealtimeTripLibrary.java:411)
        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeTripLibrary.createVehicleLocationRecordForUpdate(GtfsRealtimeTripLibrary.java:158)
        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeSource.handleCombinedUpdates(GtfsRealtimeSource.java:275)
        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeSource.handeUpdates(GtfsRealtimeSource.java:265)
        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeSource.refresh(GtfsRealtimeSource.java:244)
        at org.onebusaway.transit_data_federation.impl.realtime.gtfs_realtime.GtfsRealtimeSource$RefreshTask.run(GtfsRealtimeSource.java:399)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Here is the corresponding GTFS-RT feed snippet:

entity { id: "34" trip_update { trip { trip_id: "93063548-1_299483" start_date: "20180525" schedule_relationship: SCHEDULED route_id: "059440034:34" } stop_time_update { arrival { time: 1527287100 } departure { time: 1527287100 } stop_id: "StopPoint:40:547" schedule_relationship: SCHEDULED } stop_time_update { stop_id: "StopPoint:40:41" schedule_relationship: NO_DATA } stop_time_update { arrival { time: 1527287296 } departure { time: 1527287296 } stop_id: "StopPoint:40:134" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287377 } departure { time: 1527287377 } stop_id: "StopPoint:40:548" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287431 } departure { time: 1527287431 } stop_id: "StopPoint:40:549" schedule_relationship: SCHEDULED } stop_time_update { stop_id: "StopPoint:40:129" schedule_relationship: NO_DATA } stop_time_update { arrival { time: 1527287623 } departure { time: 1527287623 } stop_id: "StopPoint:40:550" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287682 } departure { time: 1527287682 } stop_id: "StopPoint:40:551" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287810 } departure { time: 1527287810 } stop_id: "StopPoint:40:301" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287861 } departure { time: 1527287861 } stop_id: "StopPoint:40:552" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287842 } departure { time: 1527287842 } stop_id: "StopPoint:40:299" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287889 } departure { time: 1527287889 } stop_id: "StopPoint:40:300" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287941 } departure { time: 1527287941 } stop_id: "StopPoint:40:306" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527287980 } departure { time: 1527287980 } stop_id: "StopPoint:40:307" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527288005 } departure { time: 1527288005 } stop_id: "StopPoint:40:308" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527288093 } departure { time: 1527288093 } stop_id: "StopPoint:40:553" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527288131 } departure { time: 1527288131 } stop_id: "StopPoint:40:554" schedule_relationship: SCHEDULED } stop_time_update { arrival { time: 1527288214 } departure { time: 1527288214 } stop_id: "StopPoint:40:555" schedule_relationship: SCHEDULED } vehicle { id: "STIVO:VehicleJourney::268436840:LOC" } timestamp: 1527284691 } }

OBA outputs an error and is complaining that the stop with schedule_relationship: NO_DATA should have at least an arrival or departure time or delay for update, which doesn't comply to GTFS-RT spec. Indeed, GTFS-realtime v2.0 docs for StopTimeUpdate state that:

If schedule_relationship is NO_DATA, arrival and departure must be empty.

Please fix OBA behavior to comply to the spec.

[kalon33]

That also happen with schedule_relationship: SKIPPED:

java.lang.IllegalStateException: expected at least an arrival or departure time or delay for update: stop_id: "StopPoint:8727147:810:B"
schedule_relationship: SKIPPED

GTFS-RT feed snippet:
entity { id: "4306" trip_update { trip { trip_id: "93112979-1_314879" start_date: "20180526" schedule_relationship: SCHEDULED route_id: "810:B" } stop_time_update { arrival { time: 1527325380 } departure { time: 1527325380 } stop_id: "StopPoint:8727144:810:B" schedule_relationship: SCHEDULED } stop_time_update { stop_id: "StopPoint:8727147:810:B" schedule_relationship: SKIPPED } stop_time_update { stop_id: "StopPoint:8727140:810:B" schedule_relationship: SKIPPED } stop_time_update { stop_id: "StopPoint:8727139:810:B" schedule_relationship: SKIPPED } stop_time_update { stop_id: "StopPoint:8727130:810:B" schedule_relationship: SKIPPED } vehicle { id: "Fuzzy_SNCF-ACCES:VehicleJourney::KARI86_20180526:LOC" } timestamp: 1527323593 } }

[barbeau]

@kalon33 Could you please try running this with the master branch? Instructions are here.

It looks like behavior is different in the master branch - in GtfsRealtimeTripLibrary.getTimeForStopTimeUpdate() the comment above the return statement says:

    // instead of illegal state exception we return -1 to not corrupt the read
    _log.debug("expected at least an arrival or departure time or delay for update: "
            + stopTimeUpdate);
    return -1;
  }

It's been a while since I touched this code, but I assume that's a change from v1.x, meaning that the master branch (v2.0.0-SNAPSHOT) should process this without throwing an exception.

[tesobota]

@barbeau Can you confirm how/where OBA (v1.x, or 2.x) may be reading in GTFS-RT schedulerelationship values of 1 (skipped) from stopid+tripid entity trip updates - and outputting this status (in API, etc)?

[barbeau]

@tesobota camsys/onebusaway-application-modules#19 is probably the best place to look, which is where @NJBL and @sheldonabrown just implemented better support for CANCELED and SKIPPED in CS's fork. That PR description reads:

Added functionality: Handling StopTimeUpdates that have Skipped Schedule Relationships.
Also, handling TripUpdates that have Cancelled Scheduled Relationships more effectively.

AFAIK that hasn't been merged back into the main OBA project yet.

Note that definition of how SKIPPED should be handled by consumers was fuzzy in the GTFS-rt spec until a week or two ago when this proposal was accepted, and @NJBL or @sheldonabrown would be the best ones to comment on if OBA follows the new SKIPPED spec.