Mode and purpose codes differ across survey cycles and regions

[FlaviaTsang]

Description

Mode and purpose codes are currently inconsistent across survey cycles (e.g., between the San Francisco Bay Area Travel Study (BATS) 2019 and 2023) and across regions. The current trip linking and tour building scripts are not flexible enough to handle these inconsistencies.

For example, the same purpose (e.g., change mode) was coded as 10 in BATS 2019 and 11 in BATS 2023; likewise, the same mode (e.g., car) was coded as 3 in BATS 2019 and 8 in BATS 2023.

Proposed Solution

Switch to working with mode and purpose strings instead of numeric codes (as originally proposed by PSRC).

[lmz]

I've been thinking about this and I don't think this is a particularly robust solution.
Just relying on strings alone could be brittle and subject to nuanced differences.

We've been talking about data models and using pydantic, which I think might be more straight forward and robust: each individual survey would have their mode code definitions and then we'd have a standardized version and it would be quick/readable to translate from individual to standard.

[sdrewc]

I share the concern that simply using strings could come with it's own problems (like trailing spaces, or inconsistent usage of separators "-", "_", and " "). I would like to explore a data model approach to better understand the capabilities.

Whatever solution needs to be able to handle a workflow that starts with some arbitrary variable coding that may change survey to survey, convert it to some standardized schema, and export to schemas that may differ based on the user's needs. It's my limited understanding that data model packages like pydantic allow you to build in functions for this type of conversion.

On the other hand, are there trade-offs like higher maintenance costs, or the need to identify all possible values up front?

[FlaviaTsang]

I can see how string-based processing can have pitfalls. For example, in the mode_type list, we have “Car” and “Carshare.” Someone might write a regex like “if the string includes ‘car’, then it’s the car mode,” but that would incorrectly capture “streetcar” in another region. So string processing would require some vigilance.

I’m not very familiar with the data model approach (yet), but if it’s primarily about defining a standardized (canonical) mode and purpose code and mapping each individual survey’s codes to that standard, I can see how it can be done, and that it can have the benefit of keeping the data processing code stable. The mappings would need to be created manually (?), but we can keep them in a .yaml they could be easily reviewed and checked.

I’m just wondering if there’s any additional “magic” in the data model approach that I might be missing. One potential challenge is that the standardized definitions may need to evolve over time.