discuss & define data format

[derhuerst]

No description provided.

[derhuerst]

My 2 cents about the format the we specify an API endpoint with:

As the specific parameters of different backends (e.g. OpenTripPlanner, Navitia, HAFAS, EFA) are quite specific, let's keep the specified JSON fields specific to the backend type (i.e. different fields for OTP than for HAFAS) and define them only roughly. I'm of course fine with general properties of the API, such as a description of the data contained or the provider, to be specified in a consistent way.

Personally, I like the format used by kpublictransport a lot, but I propose to

• rename the filter field (coarse bounding box of the covered area), possibly even split it into "largest area this API is known to return any data for" and "area this API is known to return canonical/detailed/exact data for".
• change lineModeMap not to specify the semantics of the modes of transport used by the API, but rather just descriptions & localisations. Attempting to standardise modes of transport in a semantic way is very hard, and many others have tried and failed before.

In general, I'd like us to rapidly iterate on this format. If something doesn't fit, let's open a PR to change it and make a new version!

[derhuerst]

Currently, at least in the German & European area, we have several open source projects that already specify API endpoints in a somewhat generalised way:

• @vkrause's kpublictransport src/lib/networks
• @derf's EFA and HAFAS endpoints list
• @schildbach's public-transport-enabler list of classes & @alexander-albers TripKit list of classes
• @marudor's marudor.de HAFAS endpoint list
• @derhuerst's hafas-client's list of "profiles" and, building on top, pan-european-public-transport's list of endpoints; @em0lar's pyhafas list of profiles
• GNOME Map's list of transit plugins
• transit.land's list of GTFS-RT feeds
• openmobilitydata.org's list of feeds & corresponding API

Similar resources:

• a list of European public transport operators with links to API endpoints & API clients, made by @juliuste

(edited to include the projects mentioned in #1 (comment))

[vkrause]

Regarding the KPublicTransport format:

• it does encode the ISO 3166-1/2 region codes in the file name, it might be worth to list that explicitly in the JSON file (for one this is somewhat specific to our implementation, and more importantly, this doesn't work for international/interregional providers). We use this information to group endpoints in a sensible way in the UI (Öffi/Transportr use a similar approach IIRC).
• the "KPlugin" wrapping for name/description is an artifact of our translation system, that is probably also not something that should leak into a shared format
• anything under "options" is backend-specific, everything else is generic - keeping such a separation probably makes sense
• the filter property naming is indeed not precise enough for what it does
• lineModeMap: that is indeed somewhat specific to our implementation, only defining the mapping to Hafas line types here is indeed more generic. We can then still map that to our own types separately (PTE also does such a mapping IIRC).
• "type" is a bit of a bad example in the de_db.json one you linked, as we special-case DB. See e.g. for something slightly more "normal": https://invent.kde.org/libraries/kpublictransport/-/blob/master/src/lib/networks/de_be_bvg.json - that would contain which Hafas/Efa/Otp/Navitia/etc variant this endpoint actually uses. There is potentially a bit of a blurry line between what is a different type, and what is just parameters for the same type, from what I have seen all our implementations more or less agree on this though (ie. things that use the same general interface and thus more or less the same client code are the same type).
• the location identifier entries are also very specific to our implementation, although I could imagine @derhuerst having similar things for matching data between different backends.

[derf]

For geocoordinates, I propose:

• reliableArea: [[lon, lat], [lon, lat], ...] is the polygon in which the service returns data with the maximum known amount of detail and accuracy. It should be set for each entry.
• usableArea: [[lon, lat], [lon, lat], ...] is the polygon in which the service returns any kind of useful data. In case of HAFAS: For locations contained in usableArea, but not contained in reliableArea, data such as line numbers or train attributes may be missing, but core functionality (e.g. routing with real-time data) remains available. usableArea is optional; if unset, it is assumed to be identical with reliableArea.

[vkrause]

Additional projects with similar setups would be:

• pyhafas: https://github.com/FahrplanDatenGarten/pyhafas
• Gnome Maps: https://gitlab.gnome.org/GNOME/gnome-maps/-/tree/master/src/transitplugins

[derhuerst]

• reliableArea [...] with the maximum known amount of detail and accuracy
• usableArea [...] any kind of useful data. [...] such as line numbers or train attributes may be missing, but core functionality (e.g. routing with real-time data) remains available.

Your definition of "usable" is what I'd consider to be "reliable". 😬 The phrasing aside, I'd say there are several nuances/levels of data coverage:

1. Incomplete and/or shallow data about areas outside of their operating area, e.g. long-distance trains & buses but not local modes of transport, or planned data but not realtime data. You could say that this is the extent to which, in a region, the API provides any data.
2. Reasonably complete data, but realtime data from other operators is missing or inaccurate quite often, e.g. DB & SNCF
3. Data about their own vehicles, with a high level of detail and the most up-to-date realtime data.

Of course, we could make this distinction arbitrarily precise, which wouldn't help all of these projects.

[vkrause]

Attribution information would probably be also a good idea for proper Open Data backends, even if those are still rare. Example: https://invent.kde.org/libraries/kpublictransport/-/blob/master/src/lib/networks/no_entur.json#L38

[derf]

There are backends with more than one endpoints. For instance, most XML EFA backends provide both XSLT_DM_REQUEST (departure monitor) and XSLT_TRIP_REQUEST2 (routing). Similarly, HAFAS installations don't just have mgate.exe (with "crypto"), but also less capable, easier to use endpoints such as ajax-getstop.exe, trainsearch.exe or stboard.exe/bhftafel.exe.

As different andpoints have different requirements and configuration variables, we shouldn't just have one JSON file per endpoint, but also one type definition. E.g. efa_dmrequest, efa_triprequest, hafas_mgate and hafas_stationboard.

[derhuerst]

Attribution information would probably be also a good idea for proper Open Data [...].

Do you think it makes sense to use the datapackage.json spec (or just the field names) or some linked open data vocabulary for that? It is somewhat specific to files/blobs of data (vs. API endpoints), but we wouldn't add yet another ad-hoc standard to the ecosystem.

[derhuerst]

As different andpoints have different requirements and configuration variables, we shouldn't just have one JSON file per endpoint, but also one type definition. E.g. efa_dmrequest, efa_triprequest, hafas_mgate and hafas_stationboard.

I'm not sure if such an "enum of types of APIs" will scale well. As an example, if you consider HAFAS endpoints, there are those with "crypto", without "crypto", rest.exe APIs, stboard.exe APIs, ajax-getstop.exe APIs, extxml.exe APIs, query.exe APIs, and probably more that I don't know of.

[vkrause]

There are backends with more than one endpoints. For instance, most XML EFA backends provide both XSLT_DM_REQUEST (departure monitor) and XSLT_TRIP_REQUEST2 (routing). Similarly, HAFAS installations don't just have mgate.exe (with "crypto"), but also less capable, easier to use endpoints such as ajax-getstop.exe, trainsearch.exe or stboard.exe/bhftafel.exe.

As different andpoints have different requirements and configuration variables, we shouldn't just have one JSON file per endpoint, but also one type definition. E.g. efa_dmrequest, efa_triprequest, hafas_mgate and hafas_stationboard.

For Hafas that's the two types we have implemented indeed, mgate.exe or the (old?) query.exe/ajax-getstop.exe/stbboard.exe variant, modeled as different types as they both need different requests and different result parsing. We currently have only one endpoint for the latter (ie. query.exe/ajax-getstop.exe/stbboard.exe combined, not each of them individually) - example: https://invent.kde.org/libraries/kpublictransport/-/blob/master/src/lib/networks/ch_sbb.json

For EFA we have 1.5 variants: only a single request path, but two separate parsers depending on whether the result is the full XML or the mobile/compact variant. Our current config files model this as one type, with different parameters. This is also how we implement the small variations in the request parameters. We could also handle that as different types though, the impact on our implementation would be quite small.

[derf]

I'm not sure if such an "enum of types of APIs" will scale well. As an example, if you consider HAFAS endpoints, there are those with "crypto", without "crypto", rest.exe APIs, stboard.exe APIs, ajax-getstop.exe APIs, extxml.exe APIs, query.exe APIs, and probably more that I don't know of.

You're right. In fact, when it comes to the HAFAS query variant, some endpoints are mostly useless when viewed in isolation. For example, traininfo.exe is only usable with the trainLink obtained by using trainsearch.exe, so those should belong to the same JSON file.

I think it's time to start tinkering with JSON files (at least for me, having an example endpoint definition in a JSON file works much better than just reading a discussion thread). To this end, I have created two DB HAFAS definitions (one for mgate, one for query) and an EFA (VRR) definition. They're suggestions based on the discussion so far; feel free to edit them as you see fit.

For me, the following open questions remain:

• how should we perform localization? The kpublictransport definitions look sensible to me, but I don't have experience in that area, so I'll leave that decision to you.
• I can't think of a sensible distinction between usable/reliable/... areas, which is why I left out the coordinates in the example files. @derhuerst I suggest you just go ahead with the solution you prefer :)
• Personally, I'd like an endpoint repository to document both sophisticated and simple API variants (e.g. both hafas-mgate and hafas-query). As db-hafas-mgate and db-hafas-query have the same provider, client software should be able to decide by itself whether it prefers the mgate or query API, so we don't need to specify a preference or otherwise indicate that they're identical. What do you think?
• I'm not familiar with the DB HAFAS mgate endpoint, so I left the "type": "hafas_mgate_deutschebahn" special case nearly as-is. Feel free to change it it.

[vkrause]

For me, the following open questions remain:

* how should we perform localization? The kpublictransport definitions look sensible to me, but I don't have experience in that area, so I'll leave that decision to you.

For KPublicTransport this is connected to KDE's translation infrastructure, so they get translated automatically by just being there. No idea how we best handle that here.

* Personally, I'd like an endpoint repository to document both sophisticated and simple API variants (e.g. both hafas-mgate and hafas-query). As db-hafas-mgate and db-hafas-query have the same provider, client software should be able to decide by itself whether it prefers the mgate or query API, so we don't need to specify a preference or otherwise indicate that they're identical. What do you think?

Agreed. As long as there is a way to detect multiple endpoints for the same provider in client code I'd indeed let the client code decide on the priority. For single protocol clients this is simple anyway, multi-protocol clients should get good results by picking the better implemented or more powerful protocol first.

* I'm not familiar with the DB HAFAS mgate endpoint, so I left the `"type": "hafas_mgate_deutschebahn"` special case nearly as-is. Feel free to change it it.

I'd go with "hafas_mgate" here, the "deutschebahn" special case in KPublicTransport is for coach layout support, which is a bit out of scope here I guess.

[derhuerst]

As I said before, I don't think an "enum of types of APIs" will scale. I'd rather prefer something like "hafasMgate": true, because it can be combined with other flags describing the endpoint.

[vkrause]

ah, I see what you meant there now. That would work for us too. Making type an array could be an alternative?

[derf]

Fair enough. I prefer boolean flags (e.g. "hafasMgate": true) over a "type":["hafas_mgate", ...] array – checking whether a dict key exists is more straightforward than iterating over an array.

[derhuerst]

I don't care if its several boolean flags, or an array of flags. Both are a lot more future-proof than a single type enum.

[derhuerst]

What is the idea behind this? @vkrause That the IDs returned by the endpoint are DB-style IBNRs?

[vkrause]

KPublictTransport has multiple id "namespaces" per location. This can be endpoint-specific ones that have no meaning outside (the default), proprietary ones that are shared between two or more endpoints (BVG/VBB are such an example), or standard ones (IBNR, UIC, IFOPT, etc). This is useful for merging data from different sources (different backends, OSM, Wikidata, etc).

To support this we have the following settings:

• locationIdentifierType defines the id namespace. This is optional for proprietary id spaces not used anywhere else.
• For many Hafas-based endpoints there is the problem that they use an IBNR or UIC code for stations having one of those, but a proprietary numeric scheme for everything else. The standardLocationIdentifierType and standardLocationIdentifierCountries options address this, the list of covered UIC country codes is needed to reliably distinguish IBNR/UIC codes from other numeric values.

This is obviously very specific to what KPublicTransport does, not particularly elegant or generic, and for most existing users probably irrelevant. I could imagine something like this to be relevant for your merging work though?

[derhuerst]

This is obviously very specific to what KPublicTransport does, not particularly elegant or generic, and for most existing users probably irrelevant. I could imagine something like this to be relevant for your merging work though?

Yes, highly relevant for my merging work! In fact, there are several projects (in the European community) that try to cross-reference public transport "things" in some way.

[vkrause]

The options {} vs top-level keys split is another implementation detail of KPublicTransport worth reconsidering here, anything in options is protocol-specific, anything top-level is handled by generic infrastructure there. For single-protocol clients that separation is completely arbitrary though, and even for multi-protocol clients that split might be different.

[derhuerst]

The options {} vs top-level keys split is another implementation detail of KPublicTransport worth reconsidering here, anything in options is protocol-specific, anything top-level is handled by generic infrastructure there. For single-protocol clients that separation is completely arbitrary though, and even for multi-protocol clients that split might be different.

Having those entries that are unspecified by this spec in a nested object probably makes maintaining this spec easier, having them directly at the root level increases the usability. I don't really care about this though, I'd rather try in practice what we have.

[derhuerst]

LGTM for now!

[derf]

+1, let's make this v1 and see how it turns out in practice.

I moved the documentation to the main readme file and specified the language codes (I presume we're going to use ISO 639-1), so we should be good to go.

[derhuerst]

@vkrause Please merge if you think this looks good.

[vkrause]

Agreed, let's get this in, and continue in smaller/more focused PRs/issues to keep the discussion easier to follow.