Skip to content

Commit 3f72727

Browse files
michaeldeistlermichaeldeistler
authored
Add citations to how-to guide (#1579)
1 parent 9ca4291 commit 3f72727

File tree

6 files changed

+192
-39
lines changed

6 files changed

+192
-39
lines changed

‎docs/how_to_guide/08_permutation_invariant_embeddings.ipynb‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,26 @@
7777
"source": [
7878
"For a detailed explanation and example, see [this tutorial](https://sbi.readthedocs.io/en/latest/advanced_tutorials/12_iid_data_and_permutation_invariant_embeddings.html)."
7979
]
80+
},
81+
{
82+
"cell_type": "markdown",
83+
"metadata": {},
84+
"source": [
85+
"## Citation\n",
86+
"\n",
87+
"```\n",
88+
"@article{radev2020bayesflow,\n",
89+
" title={BayesFlow: Learning complex stochastic models with invertible neural networks},\n",
90+
" author={Radev, Stefan T and Mertens, Ulf K and Voss, Andreas and Ardizzone, Lynton and K{\\\"o}the, Ullrich},\n",
91+
" journal={IEEE transactions on neural networks and learning systems},\n",
92+
" volume={33},\n",
93+
" number={4},\n",
94+
" pages={1452--1466},\n",
95+
" year={2020},\n",
96+
" publisher={IEEE}\n",
97+
"}\n",
98+
"```"
99+
]
80100
}
81101
],
82102
"metadata": {

‎docs/how_to_guide/13_diagnostics_lc2st.ipynb‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,24 @@
9494
"source": [
9595
"For a detailed example and additional explanation, see [this tutorial](https://sbi.readthedocs.io/en/latest/advanced_tutorials/13_diagnostics_lc2st.html)."
9696
]
97+
},
98+
{
99+
"cell_type": "markdown",
100+
"metadata": {},
101+
"source": [
102+
"## Citation\n",
103+
"\n",
104+
"```\n",
105+
"@article{linhart2023c2st,\n",
106+
" title={L-c2st: Local diagnostics for posterior approximations in simulation-based inference},\n",
107+
" author={Linhart, Julia and Gramfort, Alexandre and Rodrigues, Pedro},\n",
108+
" journal={Advances in Neural Information Processing Systems},\n",
109+
" volume={36},\n",
110+
" pages={56384--56410},\n",
111+
" year={2023}\n",
112+
"}\n",
113+
"```"
114+
]
97115
}
98116
],
99117
"metadata": {

‎docs/how_to_guide/15_expected_coverage.ipynb‎

Lines changed: 53 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,15 @@
22
"cells": [
33
{
44
"cell_type": "markdown",
5-
"id": "7de8bf27",
5+
"id": "e26566da",
66
"metadata": {},
77
"source": [
88
"# How to run expected coverage"
99
]
1010
},
1111
{
1212
"cell_type": "markdown",
13-
"id": "e431d6e1",
13+
"id": "9eedd737",
1414
"metadata": {},
1515
"source": [
1616
"Expected coverage provides a simple and interpretable tool to diagnose issues in the posterior. In comparison to other diagnostic tools such as L-C2ST, it requires relatively few additional simulations (~200) and it does not rely on any additional hyperparameters (as TARP would) or additional neural network training.\n",
@@ -22,15 +22,15 @@
2222
},
2323
{
2424
"cell_type": "markdown",
25-
"id": "81d072a6",
25+
"id": "1ca1862e",
2626
"metadata": {},
2727
"source": [
2828
"## Main syntax"
2929
]
3030
},
3131
{
3232
"cell_type": "markdown",
33-
"id": "a1e890c2",
33+
"id": "bdd96080",
3434
"metadata": {},
3535
"source": [
3636
"```python\n",
@@ -66,23 +66,23 @@
6666
},
6767
{
6868
"cell_type": "markdown",
69-
"id": "dd374b82",
69+
"id": "4a02be94",
7070
"metadata": {},
7171
"source": [
7272
"This will return a figure such as the following:"
7373
]
7474
},
7575
{
7676
"cell_type": "markdown",
77-
"id": "5db59c6e",
77+
"id": "a84f920d",
7878
"metadata": {},
7979
"source": [
8080
"<img src=\"data/sbc_rank_plot.png\" width=\"500\">"
8181
]
8282
},
8383
{
8484
"cell_type": "markdown",
85-
"id": "1072f685",
85+
"id": "7bc986ee",
8686
"metadata": {},
8787
"source": [
8888
"You can interpret this plots as follows:\n",
@@ -91,6 +91,52 @@
9191
"- If the line is above the gray region, then the `posterior_esitmator` is, on average, under-confident.\n",
9292
"- If the line is within the gray region, then we cannot reject the null hypothesis that the posterior is well-calibrated."
9393
]
94+
},
95+
{
96+
"cell_type": "markdown",
97+
"id": "c9a19771",
98+
"metadata": {},
99+
"source": [
100+
"## Citation\n",
101+
"\n",
102+
"The sample-based implementation of expected coverage used in `sbi` is described in:\n",
103+
"\n",
104+
"```\n",
105+
"@article{\n",
106+
" deistler2022truncated,\n",
107+
" title={Truncated proposals for scalable and hassle-free simulation-based inference},\n",
108+
" author={Deistler, Michael and Goncalves, Pedro J and Macke, Jakob H},\n",
109+
" journal={Advances in neural information processing systems},\n",
110+
" volume={35},\n",
111+
" pages={23135--23149},\n",
112+
" year={2022}\n",
113+
"}\n",
114+
"```\n",
115+
"\n",
116+
"Expected coverage had previously been introduced for simulation-based inference here:\n",
117+
"```\n",
118+
"@article{ \n",
119+
" hermans2022crisis,\n",
120+
" title={A crisis in simulation-based inference? beware, your posterior approximations can be unfaithful},\n",
121+
" author={Hermans, Joeri and Delaunoy, Arnaud and Rozet, Fran{\\c{c}}ois and Wehenkel, Antoine and Louppe, Gilles},\n",
122+
" journal={Transactions on Machine Learning Research},\n",
123+
" year={2022},\n",
124+
" publisher={OpenReview, Amherst, United States-Massachusetts}\n",
125+
"}\n",
126+
"```\n",
127+
"\n",
128+
"```\n",
129+
"@article{\n",
130+
" miller2021truncated,\n",
131+
" title={Truncated marginal neural ratio estimation},\n",
132+
" author={Miller, Benjamin K and Cole, Alex and Forr{\\'e}, Patrick and Louppe, Gilles and Weniger, Christoph},\n",
133+
" journal={Advances in Neural Information Processing Systems},\n",
134+
" volume={34},\n",
135+
" pages={129--143},\n",
136+
" year={2021}\n",
137+
"}\n",
138+
"```"
139+
]
94140
}
95141
],
96142
"metadata": {

‎docs/how_to_guide/16_sbc.ipynb‎

Lines changed: 40 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,15 @@
22
"cells": [
33
{
44
"cell_type": "markdown",
5-
"id": "7741d8ac",
5+
"id": "b917881c",
66
"metadata": {},
77
"source": [
88
"# How to run simulation-based calibration (SBC)"
99
]
1010
},
1111
{
1212
"cell_type": "markdown",
13-
"id": "aceee7d6",
13+
"id": "4e4f7f81",
1414
"metadata": {},
1515
"source": [
1616
"Similar to expected coverage, simulation-based calibration (SBC) provides a simple and interpretable tool to diagnose issues in the posterior. It also requires relatively few additional simulations (~200) and it does not rely on any additional hyperparameters (as TARP would) or additional neural network training.\n",
@@ -22,15 +22,15 @@
2222
},
2323
{
2424
"cell_type": "markdown",
25-
"id": "48d5f324",
25+
"id": "8e21e1ce",
2626
"metadata": {},
2727
"source": [
2828
"## Main syntax"
2929
]
3030
},
3131
{
3232
"cell_type": "markdown",
33-
"id": "41f152b0",
33+
"id": "abeefc9d",
3434
"metadata": {},
3535
"source": [
3636
"```python\n",
@@ -63,31 +63,31 @@
6363
},
6464
{
6565
"cell_type": "markdown",
66-
"id": "0d2ec9af",
66+
"id": "345cb5a1",
6767
"metadata": {},
6868
"source": [
6969
"The only difference to running expected coverage is that we did not pass `run_sbc(..., reduce_fns=...)` and we visualize it differently by not passing `sbc_rank_plot(..., plot_type=\"cdf\")`"
7070
]
7171
},
7272
{
7373
"cell_type": "markdown",
74-
"id": "2cdd5ca1",
74+
"id": "705887d9",
7575
"metadata": {},
7676
"source": [
7777
"This will return a figure such as the following:"
7878
]
7979
},
8080
{
8181
"cell_type": "markdown",
82-
"id": "cb91ea03",
82+
"id": "ddaa4e01",
8383
"metadata": {},
8484
"source": [
8585
"<img src=\"data/sbc_plot.png\" width=\"500\">"
8686
]
8787
},
8888
{
8989
"cell_type": "markdown",
90-
"id": "3467dcb5",
90+
"id": "51f22670",
9191
"metadata": {},
9292
"source": [
9393
"This plots as many plots as there are parameters. For each of the parameters you can interpret the the shape of the red bars as follows:\n",
@@ -100,19 +100,49 @@
100100
},
101101
{
102102
"cell_type": "markdown",
103-
"id": "e758d940",
103+
"id": "27b8f89e",
104104
"metadata": {},
105105
"source": [
106106
"## Example"
107107
]
108108
},
109109
{
110110
"cell_type": "markdown",
111-
"id": "67c19086",
111+
"id": "5560ee0f",
112112
"metadata": {},
113113
"source": [
114114
"For a detailed example and further explanation, see [this tutorial](https://sbi.readthedocs.io/en/latest/advanced_tutorials/11_diagnostics_simulation_based_calibration.html)."
115115
]
116+
},
117+
{
118+
"cell_type": "markdown",
119+
"id": "f26b06c0",
120+
"metadata": {},
121+
"source": [
122+
"## Citation\n",
123+
"\n",
124+
"```\n",
125+
"@article{cook2006validation,\n",
126+
" title={Validation of software for Bayesian models using posterior quantiles},\n",
127+
" author={Cook, Samantha R and Gelman, Andrew and Rubin, Donald B},\n",
128+
" journal={Journal of Computational and Graphical Statistics},\n",
129+
" volume={15},\n",
130+
" number={3},\n",
131+
" pages={675--692},\n",
132+
" year={2006},\n",
133+
" publisher={Taylor \\& Francis}\n",
134+
"}\n",
135+
"```\n",
136+
"\n",
137+
"```\n",
138+
"@article{talts2018validating,\n",
139+
" title={Validating Bayesian inference algorithms with simulation-based calibration},\n",
140+
" author={Talts, Sean and Betancourt, Michael and Simpson, Daniel and Vehtari, Aki and Gelman, Andrew},\n",
141+
" journal={arXiv preprint arXiv:1804.06788},\n",
142+
" year={2018}\n",
143+
"}\n",
144+
"```"
145+
]
116146
}
117147
],
118148
"metadata": {

‎docs/how_to_guide/17_tarp.ipynb‎

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,15 @@
22
"cells": [
33
{
44
"cell_type": "markdown",
5-
"id": "8cd307dd",
5+
"id": "7bf18a3b",
66
"metadata": {},
77
"source": [
88
"# How to run TARP"
99
]
1010
},
1111
{
1212
"cell_type": "markdown",
13-
"id": "59d719f3",
13+
"id": "0922ab20",
1414
"metadata": {},
1515
"source": [
1616
"TARP is an alternative calibration check proposed recently in https://arxiv.org/abs/2302.03026. \n",
@@ -30,7 +30,7 @@
3030
},
3131
{
3232
"cell_type": "markdown",
33-
"id": "5db62ea3",
33+
"id": "4f8d303e",
3434
"metadata": {},
3535
"source": [
3636
"```python\n",
@@ -62,39 +62,39 @@
6262
},
6363
{
6464
"cell_type": "markdown",
65-
"id": "c73833f8",
65+
"id": "dbdfcac7",
6666
"metadata": {},
6767
"source": [
6868
"This generates a plot like the following:"
6969
]
7070
},
7171
{
7272
"cell_type": "markdown",
73-
"id": "a3589a9f",
73+
"id": "d84c3063",
7474
"metadata": {},
7575
"source": [
7676
"<img src=\"data/tarp_plot.png\" width=\"400\">"
7777
]
7878
},
7979
{
8080
"cell_type": "markdown",
81-
"id": "ee39a40d",
81+
"id": "63c590f8",
8282
"metadata": {},
8383
"source": [
8484
"If the blue curve is above the diagonal, then the posterior estimate is under-confident. If it is under the diagonal, then the posterior estimate is over confident."
8585
]
8686
},
8787
{
8888
"cell_type": "markdown",
89-
"id": "2bf07d41",
89+
"id": "e4bdf2d0",
9090
"metadata": {},
9191
"source": [
9292
"## Explanation"
9393
]
9494
},
9595
{
9696
"cell_type": "markdown",
97-
"id": "45c402a1",
97+
"id": "300a7ca2",
9898
"metadata": {},
9999
"source": [
100100
"Given a test set $(\\theta^*, x^*)$ and a set of reference points $\\theta_r$, TARP calculates \n",
@@ -108,6 +108,25 @@
108108
"For each given coverage level $\\alpha$, one can then calculate the corresponding average counts\n",
109109
"and check, whether they correspond to the given $\\alpha$. "
110110
]
111+
},
112+
{
113+
"cell_type": "markdown",
114+
"id": "e37a97fe",
115+
"metadata": {},
116+
"source": [
117+
"## Citation\n",
118+
"\n",
119+
"```\n",
120+
"@inproceedings{lemos2023sampling,\n",
121+
" title={Sampling-based accuracy testing of posterior estimators for general inference},\n",
122+
" author={Lemos, Pablo and Coogan, Adam and Hezaveh, Yashar and Perreault-Levasseur, Laurence},\n",
123+
" booktitle={International Conference on Machine Learning},\n",
124+
" pages={19256--19273},\n",
125+
" year={2023},\n",
126+
" organization={PMLR}\n",
127+
"}\n",
128+
"```"
129+
]
111130
}
112131
],
113132
"metadata": {

0 commit comments

Comments
 (0)