Skip to content

Alzheimers: V&V updates for models 8.0-9.0 #1869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
NathanielBlairStahn merged 8 commits into from
Dec 29, 2025
Merged

Alzheimers: V&V updates for models 8.0-9.0 #1869

Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
3fb657d
add model run descriptions through 9.0
NathanielBlairStahn Dec 10, 2025
b974848
add links to V&V notebooks for models 8.0-8.2
NathanielBlairStahn Dec 11, 2025
7e33cbd
fill out remainder of model runs table through 9.0
NathanielBlairStahn Dec 13, 2025
45a7d05
add links to V&V notebooks for models 8.3-8.6
NathanielBlairStahn Dec 13, 2025
b9572c9
fill in more cells in V&V table
NathanielBlairStahn Dec 16, 2025
49a9f63
add details of models 8.5 and 8.6 V&V
NathanielBlairStahn Dec 17, 2025
4274eb8
add details for 8.0 and 8.1 V&V
NathanielBlairStahn Dec 17, 2025
5019343
add reminders to add links once links exist
NathanielBlairStahn Dec 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
314 changes: 237 additions & 77 deletions docs/source/models/concept_models/vivarium_alzheimers/concept_model.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@
https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#sections
And then add it to the list of decorators above.

.. Define a substitution to create manual line breaks using HTML:

.. |br| raw:: html

<br />

.. _2025_concept_model_vivarium_alzheimers:

===============================================
Expand Down Expand Up @@ -578,59 +584,80 @@ scenario, and input draw.
status
- Default
* - 8.0
-
-
-
-
-
- `Abie's consistent rates model
<https://github.com/ihmeuw/vivarium_csu_alzheimers/pull/44>`_
- Baseline, Alternative Scenario 1, Alternative Scenario 2
- * Locations: Sweden, USA
- Stratify disease state transitions and person-time by treatment
- Default
* - 8.1
-
-
-
-
-
- Consistent rates model with AD prevalence bug fix from model 7.4
- Baseline, Alternative Scenario 1, Alternative Scenario 2
- * Locations: Sweden, USA
- Stratify disease state transitions and person-time by treatment
- Default
* - 8.2
-
-
-
-
-
- Model 8.1, but one draw with 500 seeds, for estimating population
size for final run
- Baseline, Alternative Scenario 2
- * Locations: USA
* Simulation end date: 2060-12-31
* Number of draws: 1
* Population size per draw: 10 million (500 seeds of 20,000
simulants each)
- Stratify disease state transitions and person-time by treatment
- Default
* - 8.3
-
-
-
-
-
- Model 8.1 + use EMR from DisMod
- Baseline, Alternative Scenario 1, Alternative Scenario 2
- * Locations: All
* Simulation end date: 2060-12-31
* Number of draws: 5
- Stratify disease state transitions and person-time by treatment
- Default
* - 8.4
-
-
-
-
-
- Final runs for 10/31 deadline: Model 8.3 + merge in changes
through model 7.6
- Baseline, Alternative Scenario 1, Alternative Scenario 2
- * Locations: All
* Population size per draw: 2 million (100 seeds of 20,000
simulants each)
- Stratify disease state transitions and person-time by treatment
- Default
* - 8.5
-
-
-
-
-
- `Bugfix: Ensure all new simulants enter the sim in the BBBM state
after time zero
<https://github.com/ihmeuw/vivarium_csu_alzheimers/pull/62>`_
(previously, new simulants had been continuing to enter all 3
states according to the initial prevalences)
- Baseline
- * Locations: USA, Sweden
- Stratify disease state transitions and person-time by treatment
- Default
* - 8.6
-
-
-
-
-
- Run model 8.5 for all three scenarios
- Baseline, Alternative Scenario 1, Alternative Scenario 2
- * Locations: USA, Sweden
- Stratify disease state transitions and person-time by treatment
- Default
* - 8.7
-
-
-
-
-
- New final runs, same as models 8.5/8.6, but with more random seeds
- Baseline, Alternative Scenario 1, Alternative Scenario 2
- * Locations: All
* Population size per draw: 2 million (100 seeds of 20,000
simulants each)
- Stratify disease state transitions and person-time by treatment
- Default
* - 9.0
-
-
-
-
-
- `Split AD dementia into mild, moderate, severe
<https://github.com/ihmeuw/vivarium_csu_alzheimers/pull/59>`_
- Baseline, Alternative Scenario 1, Alternative Scenario 2
- * Locations: China
* Number of draws: 3
* Population size per draw: 2 million (100 seeds of 20,000
simulants each)
- Stratify disease state transitions and person-time by treatment
- Default

5.2 V & V Tracking
------------------------
Expand Down Expand Up @@ -850,7 +877,14 @@ scenario, and input draw.
* Year-stratified CSF/PET test counts per CSF/PET eligible person-year match location and time-specific rates
* Year-stratified BBBM test count per newly eligible person count match time-specific rates
* CSF/PET tests initialized properly - no testing spike for first time step
- * CSF and PET testing rates in baseline scenario match artifact
- `V&V summary in PR 14
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/14>`_

.. add manual line break for extra vertical space:

|br|

* CSF and PET testing rates in baseline scenario match artifact
values
* Baseline CSF and PET testing rates match between concept model
and artifact
Expand All @@ -876,7 +910,12 @@ scenario, and input draw.
person-time)
* Compute fraction of simulants who have had BBBM tests as
(person-time ever tested) / (person-time ever eligible)
- * The means of CSF and PET testing rates in baseline still look
- `V&V summary in PR 15
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/15>`_

|br|

* The means of CSF and PET testing rates in baseline still look
good, but the uncertainty intervals look off (I didn't check the
uncertainty in model 6.0)
* Plots of (BBBM tests) / (eligible person-time) look similar to
Expand Down Expand Up @@ -905,7 +944,12 @@ scenario, and input draw.
* Interactive sim verification spot checking a simulant's durations in treatment statuses as they move through
`BBBM test negative`, `Full treatment effect`, `Waning treatment effect`, `No treatment effect` statuses (for both completed and discontinued treatments)
* Check hazard ratios for simulants who begin treatment and those who transition to `No treatment effect`
- Things that look good:
- `V&V summary in PR 19
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/19>`_

|br|

Things that look good:

* Treatment coverage ramps up as expected
* 10% of simulants discontinue treatment as expected
Expand Down Expand Up @@ -943,10 +987,9 @@ scenario, and input draw.
an off-by-one error due to a misinterpretation of the docs, but
we decided to leave it as is to err on the side of less
effective treatment)

- * `Treatment
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_10_14_model7.0_vv_treatment.ipynb>`__
* `Interactive sim
* `Interactive sim (hazard rates)
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_10_20_model7.0_interactive_sim.ipynb>`__
* - 7.1
- Same as model 7.0, but add:
Expand Down Expand Up @@ -1045,41 +1088,158 @@ scenario, and input draw.
- * `Testing
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_10_29_model7.6_vv_testing.ipynb>`__
* - 8.0
-
-
-
- * Re-run model 5.0 V&V notebook (incidence, prevalence, mortality,
etc.)
- * EMR appears to be correct
* AD dementia incidence by age no longer looks obviously shifted
to the left like in model 5.0; it starts off a bit too high,
then gets closer to its target as tim goes on
* AD dementia prevalence starts off too low, then becomes a bit
too high as the sim progresses
* The relative prevalence of the three disease states over time
still looks weird: AD dementia prevalence starts off too low,
then increases much more quickly than the other two states,
which doesn't seem right
- * `Disease transition rates, mortality, incidence, prevalence
(Sweden) <https://github.com/ihmeuw/vivarium_research_alzheimers/blob/88fd461cb6c040964d3b6aaf7fadfff5b9ec2078/verification_and_validation/2025_10_23a_consistent_model_prototype_vv.ipynb>`__
* `Disease transition rates, mortality, incidence, prevalence
(USA) <https://github.com/ihmeuw/vivarium_research_alzheimers/blob/88fd461cb6c040964d3b6aaf7fadfff5b9ec2078/verification_and_validation/2025_10_27_model8.0_vv.ipynb>`__
* - 8.1
-
-
-
- * Re-run model 5.0 V&V notebook (incidence, prevalence, mortality,
etc.)
* See how initial prevalence looks after bugfix from model 7.4
- * EMR appears to be correct
* AD dementia incidence now looks slightly underestimated,
particularly in older ages
* AD dementia prevalence now matches its target at the beginning
of the sim, but still becomes too high as the sim progresses
* The relative prevalence of the three disease states over time
still looks weird: Even though AD dementia prevalence now starts
off correctly, it still increases much more quickly than the
other two states and becomes too high
- * `Disease transition rates, mortality, incidence, prevalence
(Sweden)
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/88fd461cb6c040964d3b6aaf7fadfff5b9ec2078/verification_and_validation/2025_10_24b_consistent_model_prototype_vv-3-M8.1.ipynb>`__
* `Disease transition rates, mortality, incidence, prevalence
(USA) <https://github.com/ihmeuw/vivarium_research_alzheimers/blob/88fd461cb6c040964d3b6aaf7fadfff5b9ec2078/verification_and_validation/2025_10_27b_model8.1_vv.ipynb>`__
* - 8.2
-
-
-
- * See how variance and uncertainty in averted deaths scale with
population size
- See linked notebooks →
- * `Disease transition rates, mortality, incidence, prevalence
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/88fd461cb6c040964d3b6aaf7fadfff5b9ec2078/verification_and_validation/2025_10_27b_model8.2_vv.ipynb>`__
* `Treatment
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_10_28_model8.2_vv_treatment.ipynb>`__
* Abie's notebook estimating variance when scaling up [[Add link
once uploaded]]
* - 8.3
-
-
-
- * Re-run model 5.0 V&V notebook (incidence, prevalence, mortality,
etc.)
* Make sure EMR matches new EMR from DisMod
* Re-run testing and treatment V&V notebooks
- `V&V summary in PR 27
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/27>`_

|br|

* Testing results have the same bugs as model 7.3, as expected
* Treatment results look like those in model 7.2, which is good
- * `Testing
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_10_29_model8.3_vv_testing.ipynb>`__
* `Treatment
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_10_29_model8.3_vv_treatment.ipynb>`__
* - 8.4
-
-
-
- No V&V (due to lack of time), just generate results tables

* Model 5.0 V&V notebook (incidence, prevalence, mortality, etc.)
should look like model 8.3 in baseline scenario
* Testing notebook should look like model 7.6
* Treatment notebook should look similar to models 8.3 and 7.2
- N/A
- * `Results tables in PR 28
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/28>`_
* - 8.5
-
-
-
- * Re-run model 5.0 V&V notebook (incidence, prevalence, mortality,
etc.)
* Check whether prevalence of AD dementia still increases faster
than the other two states, or if it looks more reasonable
* In the interactive sim, check that new simulants are only
entering in the BBBM state
- `V&V summary in PR 31
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/31>`_

|br|

* The plot of AD dementia incidence by age is no longer shifted to
the left like in model 5.0 and seems to be pretty closely
matching its target, but it is now slightly low at the beginning
of the sim rather than slightly high
* Prevalence of AD dementia by age is also a bit low at the start
of the sim, but it now looks much closer to its target at later
times rather than becoming too high
* Prevalence of the three states over time now looks much better:
Namely, the relative proportions of the 3 states remains pretty
constant as opposed to the AD dementia prevalence increasing
much faster than preclinical and MCI
* I confirmed in the interactive sim that the new simulants added
on each time step are always only in the BBBM state (at least
for the four time steps I checked)
* I confirmed in the interactive sim that the displacement of PET
and CSF tests by BBBM tests seems to be working as expected
* The interactive sim showed that we still have the same
off-by-one error in the waning effect of treatment as in model
7.0, but we decided to leave this as is (and update the docs
instead) to err on the side of making a conservative estimate
- * `Disease transition rates, mortality, incidence, prevalence
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_11_04_model8.5_vv.ipynb>`__
* `Interactive sim (hazard rates)
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_11_04_model8.5_interactive_sim_hazard_rates.ipynb>`__
* `Interactive sim (testing)
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_11_05_model8.5_interactive_sim_testing.ipynb>`__
* - 8.6
-
-
-
- * Re-run testing and treatment V&V notebooks
- `V&V summary in PR 31`_

|br|

* The number of tests, treatments, and averted deaths all
increased, by approximately a factor of 2
- * `Testing
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_11_04_model8.6_vv_testing.ipynb>`__
* `Treatment
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/85e167993e790ca561657a62c3d713630f89bc7a/verification_and_validation/2025_11_04_model8.6_vv_treatment.ipynb>`__
* - 8.7
-
-
-
- No V&V (model is identical to 8.5/8.6), just generate results
tables
- N/A
- * `Preliminary results tables in PR 32
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/32>`_
* Final results tables in PR XXX [[Add link once it exists]]
* - 9.0
-
-
-
- * Re-run model 5.0 V&V notebook (incidence, prevalence, mortality,
etc.)
* Make sure incidence, prevalence, etc. still look good
* Check that the prevalence of the different dementia severities
looks reasonable
- `V&V summary in PR 33
<https://github.com/ihmeuw/vivarium_research_alzheimers/pull/33>`_

|br|

* AD dementia incidence and prevalence are looking generally good,
except they're a bit too high in the younger ages, likely
because Abie didn't solve the differential equations for age <
45.
* AD dementia incidence and prevalence are a bit low at the
beginning of the sim. The same problem appears in model
8.5/8.6/8.7, but it looks slightly worse here.
* The prevalence of all disease states (including severities)
follows the expected pattern, and we can see slight changes in
prevalence between the baseline and treatment scenarios
* These results are only for 3 draws, so they could be affected by
parameter uncertainty.
- * `Disease transition rates, mortality, incidence, prevalence
<https://github.com/ihmeuw/vivarium_research_alzheimers/blob/8d0339e3ac748c5eed213a3a98e40b11a3aabffa/verification_and_validation/2025_11_19_model9.0_vv.ipynb>`__

.. list-table:: Outstanding model verification and validation issues
:header-rows: 1
Expand Down