ABM3 v15.2.1 Production Model Specifications

[i-am-sijia]

Note: This PR is a duplicate of PR #29, which was auto-closed by GitHub and could not be reopened. The original PR #29 has been reviewed by SANDAG.

This PR brings in model specifications from SANDAG's ABM3 production model, version v15.2.1.

The model specifications changes have been manually reviewed to ensure they do not overwrite any existing Sharrow-related modifications implemented in this repository but not present in SANDAG's production model v15.2.1.

Specifications changes from v15.2.1 include:

• Updated model constants values.
• Updated coefficients and constants in various models.
• Corrected a mixed party spec error in joint_tour_frequency_composition.csv.
• Introduced 3 segments in parking_location_choice.csv.
• Segmented auto cost factors by occupancy.

[jpn--]

So obviously the tests for this model are failing, because they are integration tests which rely on the model being the same, and in this PR we are changing the model. We can re-generate the test targets, which will make the tests pass as long as the reproducible random is working, but that will not verify anything is “correct”. @JoeJimFlood have you or anyone at SANDAG reviewed these changes to check they are actually consistent with your production model?

[jpn--]

I found a few oddities.

[jpn--]

What is up with this? It looks like we are creating 3 segments in this model by dividing tours almost arbitrarily, but looking at the edits to the coefficients file it appears all the segments are identical.

[bhargavasana]

Segmentation was added to overcome a memory issue - SANDAG/ABM#132. I recall we temporarily overcame this but started seeing crashes again for future year runs (that had thousands of parking constrained locations). Segmentation was increased to 4 recently to be reliably avoid the memory error - https://github.com/SANDAG/ABM/commits/main/src/asim/configs/resident/parking_location_choice.csv.

[jpn--]

Ok, I will award points to the house of whomever did this as a clever hack to reduce memory usage. But we do have an (explicit) chunking mechanism for exactly this. Why not use that?

[dhensle]

We had to do this because the explicit chunking was not yet implemented and SANDAG needed to be able to run their model. I think explicit chunking is the correct long-term solution.

[i-am-sijia]

Since explicit chunking is the superior long-term solution, our options are:

1. Revert this change, as memory is not an issue in the example model.
2. Add the explicit chunking setting to the example model, even if it diverges from SANDAG’s v15.2.1 production version.
3. Accept this change and be consistent with v15.2.1. (sounds the least favored?)

[bhargavasana]

I am ok with option 1 here. I feel that in general, it would be reasonable for these kinds of differences to exist between SANDAG-production and SANDAG-example models.

[jpn--]

This seems counterintuitive, to flip the meaning of the variable name. Also, in the YAML file for this component, we are filtering choosers based on transit_subsidy_available so this should probably never be triggered?

[bhargavasana]

Correct, this is probably never triggered. Not exactly sure why this was added but looking at the history (https://github.com/SANDAG/ABM/commits/main/src/asim/configs/resident/transit_pass_subsidy.csv), it may have been a routine addition for controlling availability.

[JoeJimFlood]

I checked the source code and it is indeed being filtered so that line in the spec file is redundant.

[bhargavasana]

we went ahead and dropped this expression SANDAG/ABM@a1969da

[i-am-sijia]

So obviously the tests for this model are failing, because they are integration tests which rely on the model being the same, and in this PR we are changing the model. We can re-generate the test targets, which will make the tests pass as long as the reproducible random is working, but that will not verify anything is “correct”. @JoeJimFlood have you or anyone at SANDAG reviewed these changes to check they are actually consistent with your production model?

Some v15.2.1 changes seem to be from calibration adjustments. e.g.,[1][2]

This raises a question: Should we continue carrying over calibration-related (or other minor) changes into the example model, knowing they likely cause integration test failures due to result shifts?

Reasons to skip:

• Our tests run on a subset of zones and don’t validate against targets, so calibration changes may not be meaningful.
• Local calibration tweaks may not generalize across regions.
• Avoiding them saves effort and reduces test failures.

Reasons to include:

• It keeps the example model aligned with SANDAG’s production specs, which seems important.
• But it introduces result changes. How do we ensure correctness without validation?

Open question:
If we do carry them over, what validation strategy (besides manual review) could ensure correctness given the lack of target-based testing?
Let me know if I’m overthinking this.

[JoeJimFlood]

So obviously the tests for this model are failing, because they are integration tests which rely on the model being the same, and in this PR we are changing the model. We can re-generate the test targets, which will make the tests pass as long as the reproducible random is working, but that will not verify anything is “correct”. @JoeJimFlood have you or anyone at SANDAG reviewed these changes to check they are actually consistent with your production model?

It's been a few months since I did my review of this, but if I recall correctly I did compare what was in it to the state of our config files. By the time I did my review v15.2.1 was already outdated, which I noted in my review (I added a fourth parking location segment). We've actually had a couple releases since v15.2.1 back in December.

[bhargavasana]

So obviously the tests for this model are failing, because they are integration tests which rely on the model being the same, and in this PR we are changing the model. We can re-generate the test targets, which will make the tests pass as long as the reproducible random is working, but that will not verify anything is “correct”. @JoeJimFlood have you or anyone at SANDAG reviewed these changes to check they are actually consistent with your production model?

Some v15.2.1 changes seem to be from calibration adjustments. e.g.,[1][2]

This raises a question: Should we continue carrying over calibration-related (or other minor) changes into the example model, knowing they likely cause integration test failures due to result shifts?

Reasons to skip:

• Our tests run on a subset of zones and don’t validate against targets, so calibration changes may not be meaningful.
• Local calibration tweaks may not generalize across regions.
• Avoiding them saves effort and reduces test failures.

Reasons to include:

• It keeps the example model aligned with SANDAG’s production specs, which seems important.
• But it introduces result changes. How do we ensure correctness without validation?

Open question: If we do carry them over, what validation strategy (besides manual review) could ensure correctness given the lack of target-based testing? Let me know if I’m overthinking this.

@i-am-sijia I think you raise an important question. I would vote for keeping the integration tests. In general, maybe we do not update the example model with every little calibration update in the production model. In future, maybe we do a "calibration changes only" PR to the example model which would also include updating test targets. The review of this PR needs to ensure that only calibration-related commits are included. We could do this at some set frequency like once per year.

[bhargavasana]

I would like to request this PR be updated to match ABM3 v15.3.1 specs. SANDAG ABM3 v15.3.1 will be released next week and we will essentially be freezing that version for our 2025 rtp (to be adopted in Dec). This would be a good version for the example model to match via this PR. The marginal effort to match v15.3.1 should be minimal and SANDAG can take that on if needed.