Deprecate TripUpdate.schedule_relationship = ADDED, add TripUpdate.schedule_relationship = NEW / REPLACEMENT to specify new / replaced trips which do not run on a schedule from the GTFS static.

[miklcct]

The use of TripUpdate.schedule_relationship = ADDED was unspecified and different producers / consumers used it in different ways. For example, it is sometimes used to specify additional departures on an existing route, but it is also used to specify departures which can't be matched to any existing trips.

This PR attempts to fully deprecate ADDED, and introduce NEW and REPLACEMENT, based on the implementation of OpenTripPlanner which specifies the whole journey to be added or replaced. Additional fields, such as headsigns, and pickup / drop-off types, are introduced as required to support the full specification of completely new trips.

NEW

In this proposal, TripUpdate.schedule_relationship = NEW should be used to add trips which do not duplicate an existing trip. Such trips are considered to be unrelated to any existing trips in the GTFS Static and can serve an arbitrary pattern, including completely new patterns not found in the GTFS Static.

A typical use case is for relief trips for extra demand, typically after big events.

NEW trips are intended as a migration target for existing feeds which use ADDED to specify new trips unrelated to the static GTFS, including OpenTripPlanner. trip_id in the TripDescriptor for new trips must be completely new (not found in GTFS static) and unique, and a start_date should also be specified as well (I am not using the word "must" here because it is permitted not to specify start_date to match scheduled trips, in this case the trip is assumed to run today).

The whole journey of the added trip must be specified, in stop order, as StopTimeUpdates inside the TripUpdate without any omission. Fields are added to TripProperties and StopTimeProperties for esssential information such as names, headsigns, pickup / drop off types.

REPLACEMENT

I propose to un-deprecate TripUpdate.schedule_relationship = REPLACEMENT as well. It works in the same way as NEW, apart from that the TripDescriptor must match one instance of a scheduled trip (like other values of TripUpdate.schedule_relationship), and that instance is replaced with the complete replacement trip specified in form of StopTimeUpdates like an added trip. The original stop times in the GTFS static are not considered by the replacement trip in any form to avoid confusion. The replacement trip can serve an arbitrary pattern with an arbitrary schedule, the only expectation is that the passenger should associate the replacement trip to actually be a replacement of the original trip.

A typical use case is for short-term timetable change, or short-term (near real-time) diversion, where the fact that the trip_id remains the same can be used by journey planners to notify the user that the booked service has been changed. (In particular, I have successfully used this feature to handle real-time train diversions in GB in OpenTripPlanner and route users to alight at diverted stops, which is something neither Google Maps nor Citymapper can do now)

This is the behaviour implemented in OpenTripPlanner, which is equivalent to deleting the matched trip, and processing the replacement TripUpdate as an ADDED trip mentioned above.

Relationship to TripModification

TripModification provides a way to modify trips en-masse by specifying a list of trip IDs where the same detour can be applied. However, it is not suited to change the schedule on a per-trip basis, replacing the trip with a completely different schedule after any diversions with different running times (common due to pathing constraints on railways).

It should be forbidden to modify the same trip via a REPLACEMENT trip update and also via a TripModification.

[gcamp]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

[leonardehrenfried]

Thanks for opening this PR!

OTP has had an implementation of ADDED for a long time but its behaviour is severely underspecified. I'd love to formalise it.

Yes, OTP allows you to create completely new free form trips that have no relation to an existing pattern or trip. It tries match the given route id to an existing one but if none is in the message a dummy one is created. For once, OTP is really following the "just give us what you have, and we will try to work it out" strategy.

The only requirement we have is that the stop ids must match the static GTFS. The question is what should happen when they don't. Should the entire update be dropped or individual stops? Does that even need to be specified?

I agree with what @gcamp said about the markdown tables and the issue description.

Lastly, you might find it easier to get this through review if you split it into two PRs: one for ADDED and one for REPLACEMENT. That's just a guess though.

[miklcct]

I think that the requirement for the whole trip to be specified is written in the code. Let me know if it is not clear enough.

I'll fix the formatting later today.

[miklcct]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

"The whole journey of the added trip must be specified" is a fact in the core of this PR, noted in the updated definition of StopTimeUpdate:

Updates to StopTimes for the trip (both future, i.e., predictions, and in some cases, past ones, i.e., those that already happened). The updates must be sorted by stop_sequence, and apply for all the following stops of the trip up to the next specified stop_time_update.
If trip.schedule_relationship is SCHEDULED, at least one stop_time_update must be provided for the trip.
If trip.schedule_relationship is ADDED or REPLACEMENT, stop_time_updates must be provided for all stops in the added or replacement trip, and the stop times in the static GTFS are not used.
If the trip is canceled or deleted, no stop_time_updates need to be provided. If stop_time_updates are provided for a canceled or deleted trip then the trip.schedule_relationship takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip.

The route and direction of the ADDED trip is specified in TripDescriptor.route_id. Sorry I didn't make it clear and I'll going to refine the PR. It should not be possible to replace a trip to work on a different route in a REPLACEMENT trip as it may confuse consumers.

[miklcct]

I do not want to specify the behaviour of missing stops at this moment because it may depend on the client's capability for dynamically adding stops via Stop messages. Theoretically the stop_id must refer to a stop in GTFS static, or a stop added via Stop messages.

[leonardehrenfried]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

MBTA has also indicated that they use ADDED as a producer.

@optionsome @jfabi @sam-hickey-ibigroup

[skinkie]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

Producing it for over 12 years too.

[optionsome]

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

HSL doesn't produce or consume ADDED or REPLACEMENT updates currently, if that was what you were referring to.

[leonardehrenfried]

What happens when you ADD a trip and then CANCEL it again? Should it be become invisible in the system (DELETED?) or show up as a CANCELLED?

[miklcct]

That's a good question. I still need to think about how things will work.

My producer implementation cancels an added trip using TripUpdate.schedule_relationship = ADDED with all StopTimeUpdate having a SKIPPED relationship.

The questions are that:

1. What if a later version of the full dataset real time feed doesn't contain the ADDED feed? (My intention is that it no longer exists and should be considered as DELETED. A GTFS-RT full dataset should only be applied into the original static data.)
2. How do I cancel an ADDED trip? I think that a TripUpdate with a trip id not found in static GTFS, schedule_relationship = CANCELED and the original planned stops marked with StopTimeUpdate.schedule_relationship = 'SKIPPED' makes it clear that the trip was added then cancelled. (Use DELETED instead of CANCELED to hide it from the board)

[skinkie]

As we are considering FULL_DATASET shouldn't that just replace?

[leonardehrenfried]

Actually, @skinkie is right. If you use FULL_DATASET then the moment you fetch the new version of the RT feed the old ADDED trip will completely vanish and it neither exists as DELETED nor CANCELLED. It's like it never existed.

However, once there is movement towards specifying INCREMENTAL we will have to revisit this.

[leonardehrenfried]

Does anyone know how Google and Apple handle ADDED?

@bdferris-v2

@eliasmbd I don't know who the relevant person from Apple would be. Could you tag them?

[skinkie]

@leonardehrenfried at Google the thing was limited to stop sequences previously observed. Hence if the ADDED trip was an instance of a stop sequence that is part of the database, it could be processed. I don't know if it is already capable of processing a partial instance of a stop sequence.

https://support.google.com/transitpartners/answer/10106497?hl=en#zippy=%2Cadd-with-tripupdates

[miklcct]

Here's a real world example that I experienced today on our commuter rail service (MARC)- funny enough with our real life Train 536 😄

Train 536 was terminated at stop 7 of 13 (Halethorpe) due to mechanical problems. Passengers were to be transferred to and accommodated by Train 440 which operates roughly 30 minutes after Train 536.

However, Train 440 ends its trip at station 9 (Penn Station) that Train 536 would have served and continued past if it was physically capable of doing so. At that point, Train 440 became Train 536 again and continued to the last 4 stations (10 to 13) per Train 536's schedule (with a significant delay).

In effect, Train 440 was operating as Train 440 and Train 536 at the same time before becoming Train 536 again.

Would REPLACEMENT have worked in this situation? If so, how do you think riders waiting to be picked up would see and understand the replacement in their apps/maps? And if not, what would you recommend the proper solution to be?

I ask because it seems like REPLACEMENT is the proposed way to handle this since another trip (Train 440) is sorta kinda "replacing" Train 536 due to a breakdown, but our agency knows the train will continue further than Train 440's scheduled last stop and our riders care about Train 536 because they know Train 440 does not serve those other stops. And given this proposal, I don't know what the best way to set all that up such that trip planners and realtime applications would be able to follow it using the REPLACEMENT enum. Hence asking!

It is complicated. It should be modelled as a through service of 440 and the second 536 (i.e. two trips with in-seat transfer), where the latter would be a NEW TripUpdate (because there were effectively two 536s, and they can't share the same trip_id) and the final stops running late.

However, this requires real-time support of transfer types, which is out of scope of this proposal. Otherwise, journey planners would show that the stranded passengers need to transfer to the new 536 at the final stop of the 440.

If you, instead, choose to extend the 440 with additional stops at the end, it will simply be a REPLACEMENT trip of the original 440 (not the 536).

[sam-hickey-arcadis]

Sorry for adding these comments so late in the game. Overall these changes look really good. Most of my comments are minor and formalizing things that are in the discussion comments.

Like @SteveGladstone, I can't find where "REPLACEMENT is considered an atomic "DELETE-then-NEW" but with the same trip id" is stated in this set of changes. I also haven't found any statement in these changes that a REPLACEMENT trip needs to use a trip_id from the static GTFS, so I've added proposed changes to state these things in a few places.

There needs to be more guidance on how to publish a trip cancellation for a NEW or DUPLICATED trip. The updated best practices says this:

If no stops in a trip will be visited, the trip should be CANCELED instead of having all stop_time_updates being marked as SKIPPED, unless the trip was a NEW or DUPLICATED trip and was subsequently canceled.

But this doesn't provide enough guidance. Should a NEW or DUPLICATED trip that is canceled after it is first published maintain schedule_relationship of NEW or DUPLICATED with all stops having schedule_relationship = SKIPPED and scheduled arrival and departure times must be defined?

[miklcct]

Sorry for adding these comments so late in the game. Overall these changes look really good. Most of my comments are minor and formalizing things that are in the discussion comments.

Like @SteveGladstone, I can't find where "REPLACEMENT is considered an atomic "DELETE-then-NEW" but with the same trip id" is stated in this set of changes. I also haven't found any statement in these changes that a REPLACEMENT trip needs to use a trip_id from the static GTFS, so I've added proposed changes to state these things in a few places.

I have decided not to write it into the specification because.

1. The proposal is now in the voting phase and I am afraid this may be considered a material change which may invalidate the voting.
2. The atomic DELETE-then-NEW behaviour was first mentioned in some previous issues on short-term changes on consumers failing to process one of the two TripUpdates, causing confusion to riders. The use of a REPLACEMENT solves this issue perfectly. as it is only one TripUpdate.
3. The atomic DELETE-then-NEW behaviour is the implementation in OpenTripPlanner, but I am open to other consumers using a different implementation as long as it achieves the same effect.

Sorry for the confusion. I have updated the original comment which now reads "REPLACEMENT can be implemented with" rather than "considered as".

There needs to be more guidance on how to publish a trip cancellation for a NEW or DUPLICATED trip. The updated best practices says this:

If no stops in a trip will be visited, the trip should be CANCELED instead of having all stop_time_updates being marked as SKIPPED, unless the trip was a NEW or DUPLICATED trip and was subsequently canceled.

But this doesn't provide enough guidance. Should a NEW or DUPLICATED trip that is canceled after it is first published maintain schedule_relationship of NEW or DUPLICATED with all stops having schedule_relationship = SKIPPED and scheduled arrival and departure times must be defined?

In this case, yes, because otherwise it is impossible to know when and where the cancelled trip runs.

[miklcct]

@sam-hickey-arcadis The CLA check now fails because I have applied some of your review suggestions.

[sam-hickey-arcadis]

3. The atomic DELETE-then-NEW behaviour is the implementation in OpenTripPlanner, but I am open to other consumers using a different implementation as long as it achieves the same effect.

That's a fair and good point.

There needs to be more guidance on how to publish a trip cancellation for a NEW or DUPLICATED trip. The updated best practices says this:

If no stops in a trip will be visited, the trip should be CANCELED instead of having all stop_time_updates being marked as SKIPPED, unless the trip was a NEW or DUPLICATED trip and was subsequently canceled.

But this doesn't provide enough guidance. Should a NEW or DUPLICATED trip that is canceled after it is first published maintain schedule_relationship of NEW or DUPLICATED with all stops having schedule_relationship = SKIPPED and scheduled arrival and departure times must be defined?

In this case, yes, because otherwise it is impossible to know when and where the cancelled trip runs.

What about using schedule_relationship = CANCELED but still publishing StopTimeUpdates with scheduled arrival and departure times and schedule_relationship = SKIPPED? This would allow consumers to more easily conclude that the entire trip is canceled without having to check that all stops on the trip are SKIPPED, and it would still maintain the scheduled times.

I think this needs to be formalized in the specification, but that could happen in the future (at a minimum, before removing the experimental tags). I don't think that needs to happen now and doesn't need to hold up this vote.

[sam-hickey-arcadis]

@sam-hickey-arcadis The CLA check now fails because I have applied some of your review suggestions.

I'm working on this.

[sam-hickey-arcadis]

+1 Arcadis

[bdferris-v2]

+1 Google

I will say that I share some of the same concerns raised by @SteveGladstone and @sam-hickey-arcadis about REPLACEMENT but since the field is still marked as experimental, I guess that's a discussion for another day. In particular, reconciling TripModifications and REPLACEMENT is something to keep an eye on sooner rather than later as these two experimental fields start to gain adoption.

[SteveGladstone]

+1 Maryland Transit Administration. Looking forward to leveraging NEW trip functionality 😄

[jfabi]

+1 from the @mbta—as heavy users of ADDED trips, especially on our subway system, we're very excited for this new standardization of added and modified service, as well as for potential applications such as defining bus bridges.

[miklcct]

The voting is now closed. There are 9 votes supporting this proposal and 1 abstention. Therefore the proposal is passed.

There are a few grammatic changes needed as mentioned by @bdferris-v2 before merging. I'll handle them tomorrow.

[miklcct]

The voting is now closed. There are 9 votes supporting this proposal and 1 abstention. Therefore the proposal is passed.

There are a few grammatic changes needed as mentioned by @bdferris-v2 before merging. I'll handle them tomorrow.

The editorial changes have been done, waiting for @sam-hickey-arcadis to complete his CLA such that this PR can be merged.