Update for 2025 as baseline year

[jeancochrane]

Now that it's time to test 2025 North Tri PINs, let's do two updates.

First, let's update pinval.qmd to have these baseline params:

run_id: "2025-02-11-charming-eric"
year: "2025"
pin: "10123130170000

Second, based on generating the report with the updated params, it looks like there are 13 characteristic names that are new (for example, prox_nearest_road_arterial_dist_ft) that are shown in their original snake case form and not in a "pretty" format. Let's fix, so that when the quarto doc is rendered, all characteristics names in the "Characteristics" table have a pretty format.

[jeancochrane]

One question about this effort @ccao-jardine: 2025-02-11-charming-eric is the final model run for 2025, but its comps for 2-3 card parcels were not accurate due to the bug we fixed in ccao-data/model-res-avm#360. 2025-04-25-fancy-free-billy is a run using the same hyperparameters as the 2025 final model, but with fixed comps. However, per our investigation in EI issue 253, model results are not fully reproducible between runs, even with identical hyperparameters; in this case, there are 80 cards with prediction differences ranging from ~$20 to ~$2,000, and 3 of these cards are on small (2) multicard parcels.

How do we want to handle this issue? Some options:

1. Use 2025-04-25-fancy-free-billy for everything except the 80 cards that are different; for those cards, use either 2025-02-11-charming-eric (for single-cards) or don't support PINVAL (for the 3 multicards)
   • Low lift, but won't transfer well to the production version of PINVAL due to hard-to-automate edge case handling, and requires we don't support 3 PINs
   • Alternatively: Use 2025-02-11-charming-eric for all single-card PINs, use 2025-04-25-fancy-free-billy for all 2-3 card PINs, and don't support the 3 multicards that mismatch
   • Another alternative with similar tradeoffs: Use 2025-04-25-fancy-free-billy for comps, but use 2025-02-11-charming-eric for values
2. Update the comps pipeline to allow us to run comps using a pre-trained model object, so that we can generate new comps for 2025-02-11-charming-eric
   • Heavy lift, but will transfer well tot he production version of PINVAL, and is something that will probably be useful to us again in the future
3. Investigate and fix model training reproducibility issues (Make model training fully deterministic model-res-avm#373) so that we can rerun the 2025 model for new comps but have confidence that values will be exactly the same
   • Unsure how heavy this lift will be, since I don't know where the reproducibility problems are coming from, but requires fewer changes to the comps pipeline and the comps data model than option 2

[ccao-jardine]

I like option 2; how much of a lift, and what are the risks?

[jeancochrane]

High level steps for option 2 include:

• Figure out some way of tweaking the comps data model to allow for multiple runs using the same model run, since model.comp is currently unique by (run_id, pin, card) but the new approach will allow multiple runs per run ID
  • I think the easiest solution is probably adding an incrementing version column, such that subsequent runs using the same run ID increment the version field, and the row with the highest version is the published comp
• Update the interpret pipeline stage to save comps using the new data model
• Update code that consumes comps (like PINVAL and the 2-PIN comparison doc) to handle the new data model, which we could do two ways:
  1. Tweak the consuming code to query the most recent version for all comps
  2. Switch consuming code to point to a new view called something like model.vw_comp that performs the version filtering logic and thereby maintains uniqueness by (run_id, pin, card)
• Make a new GitHub workflow to run comps based on an existing model run ID
  • Overall workflow logic: Pull model artifacts for the supplied run ID, then run dvc repro interpret to run the interpret stage with comp_enable=True and write a set of comps with an incremented version

All in all I think this would take about a week for me to finish, assuming it's my main focus but that I also have to do PR reviews and other managerial tasks in the background. I think a junior could do it in a couple weeks but they'd need a lot of guidance, since I expect there are unknown unknowns lurking here.

I think the main risk is that it's a major data model change, which might have unexpected effects on consuming code if we aren't super careful. We could mitigate this risk if we named the new view model.comp and changed the model code to push comps to a different raw table, in which case consuming code wouldn't have to change at all. There's also a risk insofar as this approach will take a lot of work, which might be better spent somewhere else; but I do think it's clearly the most sustainable fix, so I would bet that it's worth the effort.

[jeancochrane]

PINVAL now uses 2025 as the baseline year, but we still face the issue that it's not possible to run comps using an existing model object. I spun that work out into a separate issue, which doesn't need to block PINVAL development: ccao-data/model-res-avm#383