Skip to content

Commit f97deff

Browse files
dgergeldgergel
authored
Merge pull request #609 from ClimateImpactLab/add_analysis_notebooks
Add analysis notebooks
2 parents cc6263d + 3b49be4 commit f97deff

16 files changed

+14761
-0
lines changed

‎notebooks/downscaling_pipeline/pipeline_testing/check_bias_correction_two_methods.ipynb‎

Lines changed: 302 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,302 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "1a1c2a62-0daa-490c-8c27-d38b5082fa06",
6+
"metadata": {
7+
"tags": []
8+
},
9+
"source": [
10+
"##### Short bias correction method validation exercise. where we check that we properly bias corrected the CMIP6 model output. #####\n",
11+
"\n",
12+
"##### Check (1) For each cell in a small selection we calculate a range of quantiles (across the temporal distribution of a given cell) of tasmax within the whole historical period and we compare that to the reference ERA-5 data. ######\n",
13+
"\n",
14+
"##### Check (2) Similarly, we calculate quantiles for the non-bias-corrected and bias-corrected future model output, compute the absolute change relative to historical and verify the changes are preserved after correction. ######"
15+
]
16+
},
17+
{
18+
"cell_type": "markdown",
19+
"id": "4b99f421-d6e0-423a-ab9f-5cc633247189",
20+
"metadata": {
21+
"jp-MarkdownHeadingCollapsed": true,
22+
"tags": []
23+
},
24+
"source": [
25+
"#### Set up "
26+
]
27+
},
28+
{
29+
"cell_type": "code",
30+
"execution_count": 1,
31+
"id": "d4be1bb5-2c55-40ae-9c9c-803efaf697f1",
32+
"metadata": {},
33+
"outputs": [],
34+
"source": [
35+
"%%capture\n",
36+
"! pip install xclim\n",
37+
"import numpy as np\n",
38+
"from urbanspoon import core\n",
39+
"import json\n",
40+
"import gcsfs\n",
41+
"import xarray as xr"
42+
]
43+
},
44+
{
45+
"cell_type": "code",
46+
"execution_count": 4,
47+
"id": "db87076f-dd75-49ef-8f19-2af23b5dadcf",
48+
"metadata": {},
49+
"outputs": [],
50+
"source": [
51+
"gcm = 'GFDL-ESM4'\n",
52+
"data_paths_file = '/home/jovyan/output/tasmax_gcms_data_paths.json'\n",
53+
"with open(data_paths_file) as json_file:\n",
54+
" data_dict = json.load(json_file)[gcm]"
55+
]
56+
},
57+
{
58+
"cell_type": "code",
59+
"execution_count": 5,
60+
"id": "a0e3d0f5-a8c4-4a87-81a4-1d4ef0e8818a",
61+
"metadata": {},
62+
"outputs": [],
63+
"source": [
64+
"def read_gcs_zarr(zarr_url, token='/opt/gcsfuse_tokens/impactlab-data.json', check=False, consolidated=True):\n",
65+
" \"\"\"\n",
66+
" takes in a GCSFS zarr url, bucket token, and returns a dataset \n",
67+
" Note that you will need to have the proper bucket authentication. \n",
68+
" \"\"\"\n",
69+
" fs = gcsfs.GCSFileSystem(token=token)\n",
70+
" store_path = fs.get_mapper(zarr_url, check=check) \n",
71+
" ds = xr.open_zarr(store_path, consolidated=consolidated) \n",
72+
" ds.close() \n",
73+
" return ds "
74+
]
75+
},
76+
{
77+
"cell_type": "markdown",
78+
"id": "aeaacdc6-b6c8-49cb-92b4-33cf951c7540",
79+
"metadata": {
80+
"jp-MarkdownHeadingCollapsed": true,
81+
"tags": []
82+
},
83+
"source": [
84+
"#### Select a few grid cells and quantiles to look at ###"
85+
]
86+
},
87+
{
88+
"cell_type": "code",
89+
"execution_count": 6,
90+
"id": "a44600a0-4870-4a79-b93e-07231040d278",
91+
"metadata": {},
92+
"outputs": [],
93+
"source": [
94+
"# Andorra, Ohio, N Brazil\n",
95+
"cells = [(41.5, 1.5),(39.5, -83.5),(-5.5, -49.5)]\n",
96+
"myquants = [0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.99]\n",
97+
"quants_results = {}"
98+
]
99+
},
100+
{
101+
"cell_type": "markdown",
102+
"id": "e8d58846-7dc5-4183-9789-eabe1a56c366",
103+
"metadata": {
104+
"jp-MarkdownHeadingCollapsed": true,
105+
"tags": []
106+
},
107+
"source": [
108+
"#### Compute quantiles for each dataset ###"
109+
]
110+
},
111+
{
112+
"cell_type": "code",
113+
"execution_count": 11,
114+
"id": "ccd7cce4-8a39-474b-83e4-ff241c1a6e36",
115+
"metadata": {
116+
"tags": []
117+
},
118+
"outputs": [],
119+
"source": [
120+
"data_files = [\n",
121+
" data_dict['coarse']['cmip6']['historical'],\n",
122+
" data_dict['coarse']['cmip6']['ssp370'],\n",
123+
" data_dict['coarse']['bias_corrected']['historical'],\n",
124+
" data_dict['coarse']['bias_corrected']['ssp370'],\n",
125+
" data_dict['coarse']['ERA-5']\n",
126+
"]\n",
127+
"for data_file in data_files:\n",
128+
" da = read_gcs_zarr(data_file)['tasmax'].chunk(dict(time=-1))\n",
129+
" if data_file == data_files[1] or data_file == data_files[3]:\n",
130+
" da = da.sel(time=slice('2080', '2100'))\n",
131+
" result = core.xr_quantiles_across_time_by_cell(da=da, q=myquants, cells=cells)\n",
132+
" for r,k in result.items():\n",
133+
" result[r] = k.compute()\n",
134+
" quants_results[data_file] = result"
135+
]
136+
},
137+
{
138+
"cell_type": "markdown",
139+
"id": "a0385778-08c7-4127-9316-8eaeaf1212e7",
140+
"metadata": {
141+
"jp-MarkdownHeadingCollapsed": true,
142+
"tags": []
143+
},
144+
"source": [
145+
"### Bias correction method check (1) ###"
146+
]
147+
},
148+
{
149+
"cell_type": "markdown",
150+
"id": "cb69b7c6-6dba-4807-8f8d-0115e7c1eeb7",
151+
"metadata": {
152+
"tags": []
153+
},
154+
"source": [
155+
"##### Verify bias-corrected historical CMIP6 is consistent with ERA-5, while non-bias-corrected isn't. Correction should reduce the bias (so quantiles diffs should be closer to zero)"
156+
]
157+
},
158+
{
159+
"cell_type": "code",
160+
"execution_count": 67,
161+
"id": "11080d9c-539a-4a08-bf7a-d7dd0ec9dbac",
162+
"metadata": {},
163+
"outputs": [
164+
{
165+
"name": "stdout",
166+
"output_type": "stream",
167+
"text": [
168+
"cell : (41.5, 1.5)\n",
169+
"non corrected\n",
170+
"[2.56773102 2.15558777 1.97580566 2.34085083 3.32855225 3.06008911\n",
171+
" 2.49491577 2.06820374 1.76759369]\n",
172+
"corrected\n",
173+
"[1.12341064 0.98111115 0.84924927 0.58791351 0.47463989 0.6499939\n",
174+
" 0.42756042 0.24011841 0.12395844]\n",
175+
"cell : (39.5, -83.5)\n",
176+
"non corrected\n",
177+
"[0.24749176 0.93431091 2.05977478 4.80963898 4.96957397 3.92437744\n",
178+
" 3.11152039 2.97287292 3.2295816 ]\n",
179+
"corrected\n",
180+
"[-0.3089743 -0.37416229 -0.40345154 0.03394318 0.18711853 0.39482117\n",
181+
" 0.30922241 0.26907196 0.19752075]\n",
182+
"cell : (-5.5, -49.5)\n",
183+
"non corrected\n",
184+
"[ 2.81440765 2.70875092 2.26695557 2.23838806 2.25622559 -0.85774231\n",
185+
" -2.69372253 -3.00125885 -3.31266205]\n",
186+
"corrected\n",
187+
"[0.79361389 0.74092407 0.57475586 0.46287537 0.47612 0.40182495\n",
188+
" 0.38529968 0.40956726 0.41493103]\n"
189+
]
190+
}
191+
],
192+
"source": [
193+
"for c in cells:\n",
194+
" print(f'cell : {c}')\n",
195+
" print('non corrected')\n",
196+
" print(quants_results[data_dict['coarse']['ERA-5']][c].values-quants_results[data_dict['coarse']['cmip6']['historical']][c].values)\n",
197+
" print('corrected')\n",
198+
" print(quants_results[data_dict['coarse']['ERA-5']][c].values-quants_results[data_dict['coarse']['bias_corrected']['historical']][c].values)"
199+
]
200+
},
201+
{
202+
"cell_type": "markdown",
203+
"id": "a1692629-b8b3-4bba-9b49-4697b968a643",
204+
"metadata": {
205+
"tags": []
206+
},
207+
"source": [
208+
"### Bias correction method check (2) ###"
209+
]
210+
},
211+
{
212+
"cell_type": "markdown",
213+
"id": "73d77fb5-e9f1-402b-9666-77cac00b3d0e",
214+
"metadata": {
215+
"jp-MarkdownHeadingCollapsed": true,
216+
"tags": []
217+
},
218+
"source": [
219+
"##### Verify CMIP6 absolute changes in quantiles across time are preserved after QDM bias correction."
220+
]
221+
},
222+
{
223+
"cell_type": "code",
224+
"execution_count": 14,
225+
"id": "7b8e3b54-1cf2-45d5-b030-75eefcee81f6",
226+
"metadata": {},
227+
"outputs": [
228+
{
229+
"name": "stdout",
230+
"output_type": "stream",
231+
"text": [
232+
"cell : (41.5, 1.5)\n",
233+
"non corrected\n",
234+
"[3.68578522 3.20266876 3.14243774 3.14105988 3.70657349 4.42950439\n",
235+
" 4.95620422 4.98266907 4.68912598]\n",
236+
"corrected\n",
237+
"[3.87143188 3.08726807 3.14093018 3.51397705 3.96615601 4.8757019\n",
238+
" 5.29766846 5.19889526 4.94866333]\n",
239+
"cell : (39.5, -83.5)\n",
240+
"non corrected\n",
241+
"[4.91748657 2.47857208 2.54364014 3.39523315 3.83833313 4.31138611\n",
242+
" 4.07737732 3.99814301 3.65487091]\n",
243+
"corrected\n",
244+
"[5.15460327 2.65206299 2.66172485 3.64028931 4.08895874 4.36862183\n",
245+
" 4.30161133 4.03612671 3.95448853]\n",
246+
"cell : (-5.5, -49.5)\n",
247+
"non corrected\n",
248+
"[2.24838043 2.55540619 2.65269165 3.25076294 3.73539734 4.23408508\n",
249+
" 3.90587769 3.66032257 3.33886932]\n",
250+
"corrected\n",
251+
"[2.55775635 2.8618042 3.00776978 3.49560547 4.03863525 4.00012207\n",
252+
" 3.64265137 3.59101563 3.56733765]\n"
253+
]
254+
}
255+
],
256+
"source": [
257+
"for c in cells:\n",
258+
" print(f'cell : {c}')\n",
259+
" print('non corrected')\n",
260+
" print(quants_results[data_dict['coarse']['cmip6']['ssp370']][c].values-quants_results[data_dict['coarse']['cmip6']['historical']][c].values)\n",
261+
" print('corrected')\n",
262+
" print(quants_results[data_dict['coarse']['bias_corrected']['ssp370']][c].values-quants_results[data_dict['coarse']['bias_corrected']['historical']][c].values)"
263+
]
264+
},
265+
{
266+
"cell_type": "code",
267+
"execution_count": null,
268+
"id": "23668f48-76d6-4180-9eec-a850cd6cce4c",
269+
"metadata": {},
270+
"outputs": [],
271+
"source": []
272+
}
273+
],
274+
"metadata": {
275+
"kernelspec": {
276+
"display_name": "Python 3",
277+
"language": "python",
278+
"name": "python3"
279+
},
280+
"language_info": {
281+
"codemirror_mode": {
282+
"name": "ipython",
283+
"version": 3
284+
},
285+
"file_extension": ".py",
286+
"mimetype": "text/x-python",
287+
"name": "python",
288+
"nbconvert_exporter": "python",
289+
"pygments_lexer": "ipython3",
290+
"version": "3.8.12"
291+
},
292+
"widgets": {
293+
"application/vnd.jupyter.widget-state+json": {
294+
"state": {},
295+
"version_major": 2,
296+
"version_minor": 0
297+
}
298+
}
299+
},
300+
"nbformat": 4,
301+
"nbformat_minor": 5
302+
}

0 commit comments

Comments
 (0)