Skip to content

Commit 52efeaf

Browse files
janzillJan ZillJan Zilljpn--tomstephen
authored
Explicit error terms production code (ActivitySim#1064)
* bug fix * add utility based choice option for transit virtual pathbuilder * custom chooser * mem tracing in interaction_sample * bug fix * interaction_sample memory saving implementation * memory work * spelling * delete after last ref, not before * set index * to numpy once * remove comment * interaction sample dev * remove scale and location params from ev1 quartile fct * use numpy's gumbel to draw from ev1 * clean up * gumbel by hand by inverse cdf * memory logging * merge bug * more merge bugs * more merge fixes, example runs through * log eet setting on start up * some variation checks, dev notebooks, to delete * seeding corr test * corr * sign on logsum * removes notebook folder * compute_setting overrides for eet, currently only for interaction_sample to test loc sampling with MC * Adds subcomponent-specific eet setting * default compute settings set before, no need to check if none * comment clean up * identifies eet todos * test SOA MC for disagg access * lint * comments * more comments * jtp custom chooser for EET * lint * move interaction sample without sampling out of MC loop to enable for both MC and eet * update comments * fix no sampling in interaction_sample * fix _interaction_sample return doc * fix bool conversion of None for eet compute settings in sub-components * compute settings in disaggreagte accessibility w/o hack * lint * fix nest_spec is None bug * remove comment * clean up comments * comment clean up * avoid choosing alternatives corresponding to padded utilities in interaction_sample_simulate * lint * patch other instance of padded_utility for eet * adds validate utility method to align with unavailable choices in MC simulation * consistently treat utilities that would lead to zero choices in MC simulation * lint * fix allow_zero_probs for prob calculation in EET interaction_sample * fix unbound variable in estimation mode * blacken * update larch pin to 6.0.46 * np array instead of for loop * memory reduction * no duplicate arrays * bug fix: order of chooser_idx in interaction_simulate * add tests, docstrings for logit * Add basic with/without EET test for interaction simulate * Fix, complete tests for interaction sample, simulate * Linting * Normalise number of choosers, alternatives and minimum tolerance for EET comparison tests * reshape, do not flatten for potential performance * undo stray comment * unify mc and eet reporting during choice making * series not array * reinstate test, up number of draws for comparison * numpy not loop * interaction_sample test to catch index order bug * make test clearer * reset rng offset on iterate_location_choice (shadow pricing) * Add tests for logit NL, ordering, and all models using EET * Add basic docs for EET * Add tests checking that choices made using eet and from probabilities are the same for eval_mnl and eval_nl * Linting, minor changes to test_simulate.py * roll back changes to core tests to minimize noise * Implement Jan's suggestion of how to calculate household ids for testing cdap eet parity * Add test for compute_nested_utilities. Computes nested utilities on a toy example and compares the result * Linting * Move nest_spec to a fixture in simulate.py tests * Finish removing nest_spec definition in simulate.py tests * lint * clean up * reset rng only for eet for now. discuss in engineering meeting * Close out remaining TODO-EETs * Shuffle check to logit.py * Linting * Fix test * Roll back validate_utils * eet doco start * doco part two * add scale comment * Revert linting changes * . * Fix failing tests * doc fix * docs * doco clean up * doco re memory * doco clean up * memory usage doco * clean up doco * undo changes in unrelated code to reduce noise * testing for semcog model, update to shadow pricing reset * lint * disable test unitl further investigation * mp rng reset, disabling one new test for no * semcog test with different seed for eet mp to work around non-determinism in trip_scheduling (probabilistic, no EET branch) * no need to restrict to specific choosers for reset, each process has its own state * Add eet multiple zone test for zone=2 * Change formatting from using ruff to black * add back accidentally removed import * re-add 3-zone tests * base seed same as non-eet, update regress eet data * progressive 2 zone test * Copy changes from m-richards:matt/poisson_eet_2026_04 * Tidy, complete implementation of alts_context, add tests, rebuild regress for multi zone tests * Commenting/fix alts context constructions, update semcog test * Deterministic randoms when adding/removing tours for trip scheduling choice * PR noise * Roll back final_trips.csv for production_semcog * PR noise, make sure alts_context built on un-modified alternatives in trip_dest * linting * updates outdated comment * re-add arc test, remove parking choice column from regress trips * re-instate arc tests * Revert "Deterministic randoms when adding/removing tours for trip scheduling choice" This reverts commit 2d181b2. * no alts_context for trip_scheduling * updates arc regress files * removes outdated comment * trip scheduling comment for EET * disable arc sharrow test until regress trips are updated * doco update * Matt's Poisson sampling changes from PR 1065 * bug fix * add second return arg for zero probs as well * eet for nested logit * towards a unified choice interface * test and doco updates * jtp fix Co-authored-by: Copilot <copilot@github.com> * fused location sampling Co-authored-by: Copilot <copilot@github.com> * c order to map the first n_alt rands to all alternatives for the first sample, etc Co-authored-by: Copilot <copilot@github.com> * fused mnl ev1 argmax with alt_info Co-authored-by: Copilot <copilot@github.com> * stable sampling Co-authored-by: Copilot <copilot@github.com> * some doco updates * doc clean up * poisson sample tests and runtime improvements * tests for random sampling as fallback, some doc strings and removal of superfluous probs * avoid unneccessary dataframe construction, better doco * update interaction sample tests for poisson * integration of sampling methods with clear separation from eet as simulation method * poisson sampling optimization * poisson sampling with stable rands across scenarios where attractions flip from 0 to non-zero, other clean ups * do not throw on poisson fallback sampling with sample_size > alternatives * debug logging * clean up * sampling doco * clean up * doco update * linting * razzmatazz. I mean maz for taz. * test fixes and stable sorting for tour/trip loc choice * no stable alts for MC * lint * stable two-zone maz_from_taz for Poisson * more doco * no op if no alts_context * do not throw on interaction_sample_simulate w/o alts_context * eet special casing * tour_od with maz_taz stable, clean up * arc eet golden trips * switch base seed to avoid trip_scheduling (probabilistic) to come up with edge cases for small sample test. note this also happens for non-eet, e.g. with base seed 1 * forgot the corresponding golden trips * test multiple_zone golden * removes requirement of interaction_sample_simulate to have alts_context * fix test by using eet as intended, remove stable_indexing in tour_dest for mc * clean up * conditional stable sample indexes * stable alts only for eet with poisson sampling for two-zone * trip maz-for-taz stable alts for eet with poisson sampling * clean up * stable two-zone alts for tour_od * tour_od stable alt cond * resolve_smapling_method * decouple sample and simulation methods * separate RNG for shadow pricing to enable loc choiec rng reset for simulation method * lint * logging and comments * debug logging for sample method, add warning for disagg acc and poisson * doco * info, nto warning * compute settings, not model settings * compute settings, not model settings * variable name * log log on subset of rands only * optionally bias poisson disagg acc * bias also for non-mand * avoid copy in favour of log log for gumbel choice * switch to bias all location choice logsums for poisson sampling * add setting * adds missing import in trip dest * adds missing import in trip dest * ling * addresses claude code review * consistent spelling * mal-formed html comment * doc ref * addresses copilot code review * remove try block * clean up * logit clean up * clean up * lint * estimation safe guard only for poisson * deletes arc test as discussed in engineering meeting in June * adding skip_failed_choices into test, removing dead function parameter * fix eet trip_dest presample test * removes duplicate call * addresses pr review comments * changes poisson sampling to be deterministic when no alternative sampled * do not allow zero probs for nl leafs * handle non-available alternatives in make_choices_utility_based like MC in make_choices * harmonize sample choice maker naming --------- Co-authored-by: Jan Zill <jan.zill@veitchlister.com.au> Co-authored-by: Jan Zill <jzill@anl.gov> Co-authored-by: Jeff Newman <jeff@driftless.xyz> Co-authored-by: Tom Stephen <tom.stephen@icloud.com> Co-authored-by: Tyler Pearn <tyler@outerloop.io> Co-authored-by: Copilot <copilot@github.com>
1 parent da4533e commit 52efeaf

64 files changed

Lines changed: 9521 additions & 893 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

activitysim/abm/models/disaggregate_accessibility.py

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,11 @@
1515
from activitysim.abm.models.util import tour_destination
1616
from activitysim.abm.tables import shadow_pricing
1717
from activitysim.core import estimation, los, tracing, util, workflow
18-
from activitysim.core.configuration.base import PreprocessorSettings, PydanticReadable
18+
from activitysim.core.configuration.base import (
19+
ComputeSettings,
20+
PreprocessorSettings,
21+
PydanticReadable,
22+
)
1923
from activitysim.core.configuration.logit import TourLocationComponentSettings
2024
from activitysim.core.expressions import assign_columns
2125

@@ -184,6 +188,8 @@ class DisaggregateAccessibilitySettings(PydanticReadable, extra="forbid"):
184188
If not supplied or None, will default to the chunk size in the location choice model settings.
185189
"""
186190

191+
compute_settings: ComputeSettings | None = None
192+
187193

188194
def read_disaggregate_accessibility_yaml(
189195
state: workflow.State, file_name
@@ -795,6 +801,11 @@ def get_disaggregate_logsums(
795801
if disagg_model_settings.explicit_chunk is not None:
796802
model_settings.explicit_chunk = disagg_model_settings.explicit_chunk
797803

804+
# Can set compute settings for disaggregate accessibility
805+
# Otherwise this will be set to whatever is in the location model settings
806+
if disagg_model_settings.compute_settings is not None:
807+
model_settings.compute_settings = disagg_model_settings.compute_settings
808+
798809
# Include the suffix tags to pass onto downstream logsum models (e.g., tour mode choice)
799810
if model_settings.LOGSUM_SETTINGS:
800811
suffixes = util.concat_suffix_dict(disagg_model_settings.suffixes)

activitysim/abm/models/joint_tour_participation.py

Lines changed: 53 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -19,9 +19,9 @@
1919
workflow,
2020
)
2121
from activitysim.core.configuration.base import ComputeSettings, PreprocessorSettings
22-
from activitysim.core.configuration.logit import LogitComponentSettings
23-
from activitysim.core.util import assign_in_place, reindex
22+
from activitysim.core.configuration.logit import LogitComponentSettings, LogitNestSpec
2423
from activitysim.core.exceptions import InvalidTravelError
24+
from activitysim.core.util import assign_in_place, reindex
2525

2626
logger = logging.getLogger(__name__)
2727

@@ -127,10 +127,11 @@ def get_tour_satisfaction(candidates, participate):
127127

128128
def participants_chooser(
129129
state: workflow.State,
130-
probs: pd.DataFrame,
130+
probs_or_utils: pd.DataFrame,
131131
choosers: pd.DataFrame,
132132
spec: pd.DataFrame,
133133
trace_label: str,
134+
nest_spec: dict | LogitNestSpec | None = None,
134135
) -> tuple[pd.Series, pd.Series]:
135136
"""
136137
custom alternative to logit.make_choices for simulate.simple_simulate
@@ -147,9 +148,10 @@ def participants_chooser(
147148
148149
Parameters
149150
----------
150-
probs : pandas.DataFrame
151+
probs_or_utils : pandas.DataFrame
151152
Rows for choosers and columns for the alternatives from which they
152-
are choosing. Values are expected to be valid probabilities across
153+
are choosing. If running with explicit_error_terms, these are utilities.
154+
Otherwise, values are expected to be valid probabilities across
153155
each row, e.g. they should sum to 1.
154156
choosers : pandas.dataframe
155157
simple_simulate choosers df
@@ -166,7 +168,7 @@ def participants_chooser(
166168
167169
"""
168170

169-
assert probs.index.equals(choosers.index)
171+
assert probs_or_utils.index.equals(choosers.index)
170172

171173
# choice is boolean (participate or not)
172174
model_settings = JointTourParticipationSettings.read_settings_file(
@@ -202,7 +204,7 @@ def participants_chooser(
202204
"%s max iterations exceeded (%s).", trace_label, MAX_ITERATIONS
203205
)
204206
diagnostic_cols = ["tour_id", "household_id", "composition", "adult"]
205-
unsatisfied_candidates = candidates[diagnostic_cols].join(probs)
207+
unsatisfied_candidates = candidates[diagnostic_cols].join(probs_or_utils)
206208
state.tracing.write_csv(
207209
unsatisfied_candidates,
208210
file_name="%s.UNSATISFIED" % trace_label,
@@ -215,9 +217,31 @@ def participants_chooser(
215217
f"Forcing joint tour participation for {num_tours_remaining} tours."
216218
)
217219
# anybody with probability > 0 is forced to join the joint tour
218-
probs[choice_col] = np.where(probs[choice_col] > 0, 1, 0)
219-
non_choice_col = [col for col in probs.columns if col != choice_col][0]
220-
probs[non_choice_col] = 1 - probs[choice_col]
220+
if state.settings.use_explicit_error_terms:
221+
# need "is valid choice" such that we certainly choose those with non-zero values,
222+
# and do not choose others.
223+
probs_or_utils[choice_col] = np.where(
224+
probs_or_utils[choice_col] > logit.UTIL_MIN,
225+
logit.UTIL_LARGE_ENOUGH,
226+
logit.UTIL_UNAVAILABLE,
227+
)
228+
non_choice_col = [
229+
col for col in probs_or_utils.columns if col != choice_col
230+
][0]
231+
probs_or_utils[non_choice_col] = np.where(
232+
probs_or_utils[choice_col] <= logit.UTIL_MIN,
233+
logit.UTIL_LARGE_ENOUGH,
234+
logit.UTIL_UNAVAILABLE,
235+
)
236+
else:
237+
probs_or_utils[choice_col] = np.where(
238+
probs_or_utils[choice_col] > 0, 1, 0
239+
)
240+
non_choice_col = [
241+
col for col in probs_or_utils.columns if col != choice_col
242+
][0]
243+
probs_or_utils[non_choice_col] = 1 - probs_or_utils[choice_col]
244+
221245
if iter > MAX_ITERATIONS + 1:
222246
raise InvalidTravelError(
223247
f"{num_tours_remaining} tours could not be satisfied even with forcing participation"
@@ -227,9 +251,21 @@ def participants_chooser(
227251
f"{num_tours_remaining} tours could not be satisfied after {iter} iterations"
228252
)
229253

230-
choices, rands = logit.make_choices(
231-
state, probs, trace_label=trace_label, trace_choosers=choosers
232-
)
254+
if state.settings.use_explicit_error_terms:
255+
choices, rands = logit.make_choices_utility_based(
256+
state,
257+
probs_or_utils,
258+
trace_label=trace_label,
259+
trace_choosers=choosers,
260+
nest_spec=nest_spec,
261+
)
262+
else:
263+
choices, rands = logit.make_choices(
264+
state,
265+
probs_or_utils,
266+
trace_label=trace_label,
267+
trace_choosers=choosers,
268+
)
233269
participate = choices == PARTICIPATE_CHOICE
234270

235271
# satisfaction indexed by tour_id
@@ -252,7 +288,7 @@ def participants_chooser(
252288
rands_list.append(rands[satisfied])
253289

254290
# remove candidates of satisfied tours
255-
probs = probs[~satisfied]
291+
probs_or_utils = probs_or_utils[~satisfied]
256292
candidates = candidates[~satisfied]
257293

258294
logger.debug(
@@ -401,6 +437,8 @@ def joint_tour_participation(
401437
if i not in model_settings.compute_settings.protect_columns:
402438
model_settings.compute_settings.protect_columns.append(i)
403439

440+
custom_chooser = participants_chooser
441+
404442
choices = simulate.simple_simulate_by_chunk_id(
405443
state,
406444
choosers=candidates,
@@ -409,7 +447,7 @@ def joint_tour_participation(
409447
locals_d=constants,
410448
trace_label=trace_label,
411449
trace_choice_name="participation",
412-
custom_chooser=participants_chooser,
450+
custom_chooser=custom_chooser,
413451
estimator=estimator,
414452
compute_settings=model_settings.compute_settings,
415453
)

0 commit comments

Comments
 (0)