Add NetCDF File Fixer Tutorial to main

NetCDF File Fixer.ipynb

@@ -0,0 +1,203 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# NetCDF File Reformat for ArcGIS Pro Tutorial Notebook"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Summary\n",
+    "The purpose of this notebooks is to show users of suborbital campaign data in NetCDF format how to reformat input files to make them useable in ArcGIS Pro as a Multidimensional Dataset. Previously, users have given feedback that NetCDF files do not plot properly as Multidimensional Datasets in ArcGIS Pro - this notebook highlights the root cause and demonstrates how to restructure the file to work seemlessly in ArcGIS Pro. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Prerequisites\n",
+    "**Please create a \"nc_files\" folder to house input NetCDF files, and create a \"new_nc_files\" folder to house converted NetCDF files.**\n",
+    "- netCDF4\n",
+    "- numpy\n",
+    "- pathlib\n",
+    "- datetime\n",
+    "- tqdm\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Notebook Author/Affiliation\n",
+    "Gabriel Mojica/Atmospheric Science Data Center (ASDC)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Steps\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"5\">Import required packages</font><p></p>\n",
+    "<font size=\"4\">For packages you don't have, run a <i>pip install</i> to add them to your machine</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from netCDF4 import Dataset\n",
+    "import numpy as np\n",
+    "from pathlib import Path\n",
+    "from datetime import datetime, time\n",
+    "import tqdm"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"5\">Define directories to iterate through and write to. Ensure you have the source and output directory folders created</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "source_dir = 'nc_files/'\n",
+    "output_dir = 'new_nc_files/'\n",
+    "input_path = Path(source_dir)\n",
+    "input_list = list(input_path.iterdir())\n",
+    "output_path = Path(output_dir)\n",
+    "output_list = list(output_path.iterdir())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"5\">Build function to copy dimensions, attributes, and variables from <i>src_group</i> to <i>dst_group</i></font><p></p>\n",
+    "<font size=\"4\">Copy group attributes with the setncatts() function</font><p></p>\n",
+    "<font size=\"4\">Copy dimensions with the createDimension() function</font><p></p>\n",
+    "<font size=\"4\">Copy variables with condition to change \"flag\" variables from <i>float64</i> to <i>int32</i> datatypes</font><p></p>\n",
+    "- <font size=\"3\">Check if variable name contains \"flag\" and is <i>float64</i>, change to <i>int32</i> then fix NaN values</font>\n",
+    "- <font size=\"3\">Create the variables in the newly built group</font>\n",
+    "- <font size=\"3\">Copy variable attributes to the new variables</font>\n",
+    "- <font size=\"3\">Write in the data</font>\n",
+    "<p><font size=\"4\">Create the subgroups with the createGroup() function</font></p>\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def copy_nc_structure(src_group, dst_group):\n",
+    "    \n",
+    "    dst_group.setncatts(src_group.__dict__)\n",
+    "\n",
+    "    for name, dimension in src_group.dimensions.items():\n",
+    "        dst_group.createDimension(\n",
+    "            name, (len(dimension) if not dimension.isunlimited() else None)\n",
+    "        )\n",
+    "\n",
+    "    for name, var in src_group.variables.items():\n",
+    "        if 'flag' in name.lower() and var.datatype == np.float64:\n",
+    "            datatype = np.int32\n",
+    "            data = np.nan_to_num(var[:], nan=-1).astype(np.int32)\n",
+    "        else:\n",
+    "            datatype = var.datatype\n",
+    "            data = var[:]\n",
+    "\n",
+    "        new_var = dst_group.createVariable(name, datatype, var.dimensions, zlib=True)\n",
+    "\n",
+    "        new_var.setncatts(var.__dict__)\n",
+    "\n",
+    "        new_var[:] = data\n",
+    "\n",
+    "    for group_name, sub_group in src_group.groups.items():\n",
+    "        new_sub_group = dst_group.createGroup(group_name)\n",
+    "        copy_nc_structure(sub_group, new_sub_group)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"5\">Once you create the function, you can now run it using an <i>if</i> loop</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "if len(input_list) >0:\n",
+    "    print(f\"Processing {len(input_list)} NetCDF Files.\")\n",
+    "    for file in tqdm.tqdm(input_list):\n",
+    "        with Dataset(str(input_path) + '/' + file.name, 'r') as src, Dataset(str(output_path) + '/' + file.name, 'w') as dst:\n",
+    "            copy_nc_structure(src, dst)\n",
+    "    print(f\"Your cleaned NetCDF files have been created in {output_path}.\")\n",
+    "else:\n",
+    "    print(f\"There are no NetCDF files in {input_list}.\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font size=\"5\">Now that you've created the new NetCDF files, you can delete the files in your <i>nc_files</i> directory by running the following cell:</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "for item in tqdm.tqdm(input_list):\n",
+    "    if item.is_file():\n",
+    "        item.unlink()\n",
+    "print(\"Your old NetCDF files have been deleted.\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "ArcGISPro",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "version": "3.13.5"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}