added geospatial

Author: kosuri-indu

docs/make.jl

@@ -8,10 +8,13 @@ makedocs(;
         "Tutorials" => [
             "Patient-Level Prediction" => [
                 "Introduction" => "plp-intro.md",
-                "PLP Workflow" => "plp-workflow.md",
+                "Workflow" => "plp-workflow.md",
                 "Package Examples" => "plp-examples.md",
             ],
-            "Geospatial Health Informatics" => "geospatial.md",
+            "Geospatial Health Informatics" => [
+                "Introduction" => "geospatial-intro.md",
+                "Workflow" => "geospatial-workflow.md",
+            ],
             "MRI Simulation and Analysis" => "mri.md",
         ],
     ],

docs/src/geospatial-intro.md

@@ -0,0 +1,79 @@
+# Geospatial Health Informatics Using Census Microdata
+
+Geospatial health informatics studies health and social outcomes in relation to place.
+It helps answer questions such as: Which regions have stronger educational attainment?
+Where are social risk factors concentrated? Which areas should be prioritized for policy action?
+
+For public health research, this matters because many health patterns are not evenly distributed.
+Disease burden, healthcare access, education, income, and environmental exposure often vary by region.
+A map-based workflow makes those differences easier to see, compare, and communicate.
+
+In this project, we use census microdata together with administrative boundaries to move from
+individual records to region-level indicators. The example focuses on educational attainment,
+but the same approach can be reused for many public-health-relevant measures.
+
+## Why Census Microdata + Geospatial Data?
+
+- Census microdata provides person-level or household-level variables, which makes it possible to build detailed indicators.
+- IPUMS data projects are designed for integrated population research across time and place, with harmonized documentation and metadata.
+- Geospatial datasets provide the boundary geometry (for example NUTS regions), which turns tabular indicators into maps.
+- When we aggregate microdata by region and join it to boundaries, we can compare population-level trends and detect spatial inequalities.
+
+In practical terms, microdata gives us the "what" and geospatial boundaries give us the "where".
+Combining both gives an interpretable view for public health planning and communication.
+
+## IPUMS in This Workflow
+
+IPUMS (Integrated Public Use Microdata Series) provides structured census and survey data with rich metadata.
+In this workflow, IPUMS.jl is used to load:
+
+- DDI metadata (XML)
+- microdata extract files (DAT)
+- NHGIS-style geospatial files for boundaries
+
+This allows a reproducible, script-first pipeline: load metadata, load records, inspect variables,
+and then aggregate to region-level outputs for mapping.
+
+## Package Stack Used
+
+- `GeoMakie.jl`
+	Geographic plotting utilities in the Makie ecosystem. Its `GeoAxis` supports map projections
+	and coordinate transformations, so data can be displayed in an appropriate destination projection.
+
+- `CairoMakie.jl`
+	A high-quality Makie backend for SVG/PDF/vector output. This is well-suited for publication-style
+	static figures and clean map exports.
+
+- `GeoInterfaceMakie.jl`
+	Adds Makie plotting support for geometries that implement GeoInterface traits.
+	In practice, this helps geometry objects from geospatial packages work smoothly with Makie plotting calls.
+
+- `GeoDataFrames.jl`
+	A DataFrame-oriented way to read and handle geospatial vector data in Julia,
+	inspired by GeoPandas-style workflows.
+
+- `IPUMS.jl`
+	The package used here to parse DDI metadata and load census and boundary extracts.
+	In this repository environment, IPUMS is pinned from the JuliaHealth IPUMS.jl source in the workflow Manifest.
+
+- `DataFrames.jl`, `Chain.jl`, `StatsBase.jl`
+	Support aggregation, labeling, grouping, and summary steps before visualization.
+
+## End-to-End Logic
+
+The workflow is intentionally explicit and reproducible:
+
+1. Acquire census microdata and metadata with IPUMS.jl.
+2. Load and clean region boundaries.
+3. Assign readable education categories from coded microdata values.
+4. Aggregate and normalize counts for fair visual comparison.
+5. Join regional summaries to geospatial geometries.
+6. Create choropleth maps with clear titles, color schemes, and colorbars.
+
+Each step is documented and implemented as runnable Julia code in the workflow scripts.
+
+## Reproducibility Note
+
+This example follows a reproducible Poland workflow with explicit file paths, clear transformations,
+and deterministic plotting steps. If restricted datasets are unavailable, the same structure can be
+reused with other compatible IPUMS extracts and boundary files.

docs/src/geospatial-workflow.md

@@ -0,0 +1,206 @@
+# Geospatial Workflow
+
+This workflow demonstrates:
+
+1. Data acquisition with IPUMS.jl
+2. Preprocessing and educational category labeling
+3. Integration of aggregated census metrics with geospatial geometries
+4. Choropleth visualization with colorbar and layout tuning
+
+The scripts in this page mirror the runnable workflow under `src/workflows/geospatial_census/src/`. The goal is to keep the code easy to read and easy to run: load data, summarize it by region, join to geometry, and plot maps.
+
+## Requirement Coverage
+
+This workflow explicitly covers all required tutorial goals:
+
+- Data acquisition: load census microdata and metadata through IPUMS.jl (`parse_ddi`, `load_ipums_extract`) and load geospatial boundaries (`load_ipums_nhgis`).
+- Preprocessing: filter and clean geometry rows, convert join keys, map coded education values to readable labels, and aggregate counts.
+- Normalization: compute min-max normalized counts across regions before choropleth coloring.
+- Integration: merge aggregated census summaries with geospatial geometries using key-based joins.
+- Visualization: produce choropleth maps with multiple color schemes, titles, layout tuning, and a final colorbar for interpretability.
+- Reproducibility: provide runnable Julia code, fixed script order, and config-driven local file paths.
+
+## 1. Load Required Packages
+
+```julia
+using CairoMakie, Chain, CSV, DataFrames, GeoDataFrames, GeoInterfaceMakie, GeoMakie, StatsBase
+import IPUMS: load_ipums_extract, load_ipums_nhgis, parse_ddi
+```
+
+## 2. Load IPUMS Census Data
+
+This step reads the metadata dictionary (`.xml`) and the fixed-width microdata extract (`.dat`).
+Both paths are configured in `config.toml` when you run the script pipeline.
+
+```julia
+ddi = parse_ddi("poland_data/ipumsi_00001.xml")
+df = load_ipums_extract(ddi, "poland_data/ipumsi_00001.dat")
+```
+
+## 3. Metadata Exploration
+
+This section documents data acquisition quality checks. You inspect dataset-level metadata first, then column-level descriptions to verify
+definitions before deriving indicators.
+
+```julia
+md_df = metadata(df)
+for md in keys(md_df)
+  println("$(md):\n----------------\n\n $(md_df[md])\n\n")
+end
+```
+
+Then inspect variable descriptions:
+
+```julia
+for colname in names(df)
+  println("$(colname):\n----------------\n")
+  try
+    println(" $(colmetadata(df, colname, "description"))\n")
+  catch
+    println(" description metadata not available\n")
+  end
+end
+```
+
+## 4. Load and Filter Shapefile for Poland
+
+The boundary file is filtered to Poland, rows without geometry are removed, and the regional key
+is cast to `Int64` so it can be joined to aggregated census results.
+
+```julia
+geodf = load_ipums_nhgis("shapefiles/ENUTS2_2013.shp").geodataframe
+filter!(x -> x.CNTRY_NAME == "Poland", geodf)
+dropmissing!(geodf, :geometry)
+geodf[!, :ENUTS2] = parse.(Int64, geodf[!, :ENUTS2])
+```
+
+## 5. Aggregate Educational Attainment by Region
+
+This step maps raw education codes into readable categories (`PRIMARY`, `SECONDARY`, `UNIVERSITY`),
+then computes regional counts for each category.
+
+It also prepares data for comparative mapping by creating grouped regional counts that are later normalized.
+
+```julia
+using Chain
+
+edu_df = @chain df begin
+  groupby(_[:, [:ENUTS2_2013, :EDUCPL]], [:ENUTS2_2013, :EDUCPL])
+  combine(nrow => :Count)
+end
+
+primary = [12, 20]
+secondary = [40, 41, 42, 43, 50]
+university = [70, 71, 72, 73]
+
+edu_df.EDUCPL = convert(Vector{Any}, edu_df.EDUCPL)
+replace!(x -> in(x, primary) ? "PRIMARY" : x, edu_df.EDUCPL)
+replace!(x -> in(x, secondary) ? "SECONDARY" : x, edu_df.EDUCPL)
+replace!(x -> in(x, university) ? "UNIVERSITY" : x, edu_df.EDUCPL)
+
+edu_counts = @chain edu_df begin
+  filter!(row -> !isa(row.EDUCPL, Real), _)
+  groupby([:ENUTS2_2013, :EDUCPL])
+  combine(:Count => sum => :Count)
+  groupby([:EDUCPL])
+end
+```
+
+## 6. Visualize Educational Attainment Across Regions
+
+The workflow keeps both map styles:
+
+- rough exploration with multiple color schemes (`:Purples`, `:Greens`, `:Blues`, `:Reds`)
+- final cleaned map with `:Wistia` and a colorbar
+
+Inside the plotting loop, counts are normalized to the 0-1 range. This improves comparability between
+education categories and across regions, independent of absolute category sizes.
+
+```julia
+# Base region map
+fig_regions = Figure(size = (1200, 1400), fontsize = 20)
+ax_regions = CairoMakie.Axis(fig_regions[1, 1])
+poly!(ax_regions, geodf.geometry, color = 1:16, colormap = :Reds, strokecolor = :black, strokewidth = 3)
+Label(fig_regions[:, :, Top()], "Voivodeships of Poland", fontsize = 50)
+hidedecorations!(ax_regions)
+
+# Multi-color comparison map
+fig_rough = Figure(size = (1200, 1400), fontsize = 20)
+axs_rough = [Axis(fig_rough[x, y]) for x in 1:2 for y in 1:2]
+colors = [:Purples, :Greens, :Blues, :Reds]
+poly!(axs_rough[1], geodf.geometry, color = :white, strokecolor = :black, strokewidth = 3)
+hidedecorations!(axs_rough[1])
+axs_rough[1].title = "Voivodeships of Poland"
+Label(fig_rough[:, :, Top()], "Normalized Education Counts across Poland", fontsize = 50, padding = (0, 0, 30, 0))
+
+for (idx, counts) in enumerate(edu_counts)
+  geo_counts = outerjoin(counts, geodf; on = [:ENUTS2_2013 => :ENUTS2])
+  norm_counts = (geo_counts.Count .- minimum(geo_counts.Count)) / (maximum(geo_counts.Count) .- minimum(geo_counts.Count))
+  cmap = cgrad(colors[idx + 1], norm_counts)
+  dropmissing!(geo_counts, :geometry)
+  ax = axs_rough[idx + 1]
+  poly!(ax, geo_counts.geometry, color = cmap[norm_counts], strokecolor = :black, strokewidth = 3)
+  ax.title = "$(counts.EDUCPL |> first)"
+  hidedecorations!(ax)
+end
+
+# Final cleaned visualization with a colorbar
+fig = Figure(size = (1200, 1400), fontsize = 20)
+axs = [Axis(fig[x, y]) for x in 1:2 for y in 1:2]
+
+poly!(axs[1], geodf.geometry, color = :white, strokecolor = :black, strokewidth = 2)
+axs[1].title = "Voivodeships of Poland"
+hidedecorations!(axs[1])
+Label(fig[:, :, Top()], "Normalized Educational Attainment across Poland", fontsize = 50, padding = (0,0,30,0))
+
+for (idx, counts) in enumerate(edu_counts)
+    geo_counts = outerjoin(counts, geodf; on = [:ENUTS2_2013 => :ENUTS2])
+    norm_counts = (geo_counts.Count .- minimum(geo_counts.Count)) / (maximum(geo_counts.Count) .- minimum(geo_counts.Count))
+
+    cmap = cgrad(:Wistia, norm_counts)
+    dropmissing!(geo_counts, :geometry)
+
+    ax = axs[idx + 1]
+    poly!(ax, geo_counts.geometry, color = cmap[norm_counts], strokecolor = :black, strokewidth = 2)
+    ax.title = "$(counts.EDUCPL |> first)"
+    hidedecorations!(ax)
+end
+
+Colorbar(fig[:, 3], limits = (0, 1), colormap = :Wistia)
+
+save(joinpath(OUTPUT_DIR, "voivodeships_poland.png"), fig_regions)
+save(joinpath(OUTPUT_DIR, "education_rough_multicolors.png"), fig_rough)
+save(joinpath(OUTPUT_DIR, OUTPUT_FIGURE), fig)
+```
+
+Additional outputs produced by the scripted workflow:
+
+- `output/voivodeships_poland.png`
+- `output/education_rough_multicolors.png`
+- `output/education_choropleth_poland.png`
+
+## Reproducible Scripted Run
+
+From the workflow directory:
+
+```bash
+julia --project=. -e "using Pkg; Pkg.instantiate()"
+copy config.toml.example config.toml
+julia --project=. run.jl
+```
+
+On macOS/Linux, replace `copy` with `cp`.
+
+No database is required for this workflow. Inputs are local files configured in `config.toml`:
+
+- IPUMS DDI metadata (`.xml`)
+- IPUMS extract data (`.dat`)
+- NHGIS shapefile (`.shp` and companion files)
+
+The scripted workflow is implemented in:
+
+- `src/workflows/geospatial_census/src/01_load_data.jl`
+- `src/workflows/geospatial_census/src/02_preprocess.jl`
+- `src/workflows/geospatial_census/src/03_visualize.jl`
+
+This keeps acquisition, preprocessing, integration, and visualization steps explicit and reproducible.

docs/src/geospatial.md

@@ -0,0 +1,2 @@
+output/
+config.toml

src/workflows/geospatial_census/.gitignore

@@ -0,0 +1,16 @@
+name = "GeospatialCensusWorkflow"
+authors = ["JuliaHealth contributors"]
+
+[deps]
+CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
+CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
+Chain = "8be319e6-bccf-4806-a6f7-6fae938471bc"
+DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
+GeoDataFrames = "62cb38b5-d8d2-4862-a48e-6a340996859f"
+GeoInterfaceMakie = "0edc0954-3250-4c18-859d-ec71c1660c08"
+GeoMakie = "db073c08-6b98-4ee5-b6a4-5efafb3259c6"
+IPUMS = "51d1f77e-d457-4c14-a89d-9ed71839f38d"
+StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
+
+[compat]
+julia = "1.10"

src/workflows/geospatial_census/Project.toml

@@ -0,0 +1,3 @@
+using Pkg
+Pkg.activate(@__DIR__)
+Pkg.add(url="https://github.com/JuliaHealth/IPUMS.jl.git")

src/workflows/geospatial_census/README.md

@@ -0,0 +1,31 @@
+# Geospatial Census Workflow Configuration
+# 
+# This file documents the expected data paths for the IPUMS Poland workflow.
+# Copy this to config.toml and update paths to match your local data.
+#
+# Data Sources:
+# - IPUMS International: https://international.ipums.org/
+# - IPUMS NHGIS: https://www.nhgis.org/
+#
+# To set up:
+# 1. Create a directory `poland_data/` in the workflow root
+# 2. Download IPUMS extract (XML and DAT files) 
+# 3. Create a directory `shapefiles/` in the workflow root
+# 4. Download ENUTS2_2013 shapefile from IPUMS NHGIS
+
+[data]
+# Path to IPUMS census data DDI metadata file (XML)
+ipums_ddi = "poland_data/ipumsi_00001.xml"
+
+# Path to IPUMS census data extract (DAT)
+ipums_dat = "poland_data/ipumsi_00001.dat"
+
+# Path to ENUTS2 shapefile for Poland regions
+shapefile = "shapefiles/ENUTS2_2013.shp"
+
+[output]
+# Directory where results will be saved
+dir = "output"
+
+# Figure filename
+figure = "education_choropleth_poland.png"