GDEX TDS — Deployment & Operations Guide

Overview

GDEX TDS (THREDDS Data Server) is deployed on NCAR's CIRRUS Kubernetes cluster. This repo contains:

rda-tds/ — TDS application code, catalog XML, and Dockerfiles
rda-tds-helm/ — Helm chart for Kubernetes deployment on CIRRUS
prefect-workflow/ — Prefect-based automation for daily dataset management
src/ — Helper scripts for generating XML catalogs and CTL control files

Repository Tree

gdex-tds/
├── .github/                    # GitHub Actions workflows
├── prefect-workflow/           # Prefect automation (dataset autoscan, control registration)
│   ├── auto_add_data_tds.py   # Daily autoscan: add new datasets to TDS catalog
│   ├── add_control_tds.py     # Post-merge: register TDS URL in GDEX data portal
│   ├── delete_control_tds.py  # Remove datasets from TDS
│   ├── modify_control_tds.py  # Modify existing control entries
│   └── auto_add_data_tds_<datetime>.log  # Output logs committed by autoscan
├── rda-tds-helm/              # Helm chart (k8s/CIRRUS deployment)
│   ├── Chart.yaml
│   ├── values.yaml            # All tunable parameters
│   ├── scripts/
│   │   └── log_stats.py       # Access log aggregation script (deployed via ConfigMap)
│   └── templates/
│       ├── deployment.yaml
│       ├── service.yaml
│       ├── ingress.yaml
│       ├── deny-backend.yaml
│       ├── ingress-dap4-block.yaml
│       ├── ingress-wmsold-block.yaml
│       ├── pvc-tds.yaml
│       ├── pvc-log.yaml
│       ├── heapdump-clean-cronjob.yaml
│       ├── log-clean-cronjob.yaml
│       ├── log-stats-configmap.yaml
│       ├── log-stats-cronjob.yaml
│       └── pv-s3-backup.yaml
├── rda-tds/                   # TDS application (catalog XML, Dockerfiles)
│   ├── content/               # catalog.xml, per-dataset XMLs, threddsConfig.xml
│   └── Dockerfiles/
├── src/                       # Helper scripts
│   ├── createXML.py
│   ├── createCTL.py
│   └── gen_stats_plot.py      # Generate interactive usage stats HTML from Boreas backup
└── tds_usage_stats.html       # Generated by gen_stats_plot.py

Helm Chart — `rda-tds-helm/`

Changes here redeploy the chart without rebuilding container images.

`values.yaml`

Central configuration file. Key sections:

Key	Description
`replicaCount`	Number of TDS pod replicas
`webapp.tds.image`	TDS Docker image
`webapp.logs.image`	Log sidecar image
`webapp.tdm.image`	TDM (Thredds Data Manager) image
`webapp.tdsPersist.fs`	PVC for TDS index files / cache (1 Ti CephFS)
`webapp.logPersist.fs`	PVC for Tomcat access logs (100 Gi CephFS)
`backup.enabled`	Toggle the Boreas S3 backup CronJob
`backup.logs.enabled`	Include Tomcat log volume in backup

Templates

`deployment.yaml`

Main pod spec. Runs three containers in each pod:

Container	Image	Mounts	Purpose
`rda-tds`	`rda-tds`	data (NFS), logs PVC, index PVC	THREDDS Data Server
`rda-logs`	`rda-logs`	logs PVC	Streams access logs to Grafana (Promtail sidecar)
`gdex-tdm`	`gdex-tdm`	index PVC, data (NFS)	THREDDS Data Manager — builds index files

TDS-specific PVC subPath mounts (all under tds-persist):

Container path	SubPath	Purpose
`/usr/local/tomcat/content/thredds/cache/cdm`	`tds-temp-cache-cdm`	CDM feature cache
`/usr/local/tomcat/content/thredds/cache/ncss`	`tds-temp-cache-ncss`	NCSS temp files
`/usr/local/tomcat/content/thredds/cache/wcs`	`tds-temp-cache-wcs`	WCS cache
`/usr/local/tomcat/content/thredds/cache/edal-java`	`tds-temp-cache-edal-java`	WMS tile cache
`/usr/local/tomcat/temp`	`tds-overflow/tomcat-temp`	Heap dump output dir
`/tmp`	`tds-overflow/tmp`	JVM temp / heap dumps

`service.yaml`

ClusterIP service exposing the TDS pod on port 8080.

`ingress.yaml`

External NGINX ingress serving all three FQDNs (tds.k8s.ucar.edu, tds.gdex.ucar.edu, thredds.rda.ucar.edu) with:

TLS via InCommon cert-manager issuer
Rate limiting (limit-rps, limit-connections) from values.yaml
EWMA load balancing across replicas
180 s proxy read timeout for large data requests

`deny-backend.yaml`

A minimal nginx deployment that returns HTTP 403 for every request. Used as the backend for ingress rules that need to block specific URL paths at the ingress layer rather than in the TDS application.

`ingress-dap4-block.yaml`

Routes all /thredds/dap4 traffic to the deny-backend (403). DAP4 is disabled because the TDS instance is not configured for it and open DAP4 endpoints cause client errors.

`ingress-wmsold-block.yaml`

Routes /thredds/wms/files/g/ds to the deny-backend (403). This old-style WMS path generates excessive error traffic and is not used by current clients.

`pvc-tds.yaml`

PersistentVolumeClaim (tds-persist, 1 Ti, CephFS RWX) for TDS index files, caches, and overflow temp dirs. Annotated with argocd.argoproj.io/sync-options: Prune=false so Argo CD never deletes it on sync.

`pvc-log.yaml`

PersistentVolumeClaim (logs-persist, 100 Gi, CephFS RWX) shared between the TDS container (write) and the log sidecar and log-stats CronJob (read).

`heapdump-clean-cronjob.yaml`

Runs every Monday at 1:01 AM. Deletes *.hprof heap dump files older than 7 days from tds-persist/tds-overflow/tomcat-temp. Prevents disk fill when TDS OOM-crashes repeatedly.

`log-clean-cronjob.yaml`

Runs daily at 3:00 AM. Deletes Tomcat access log files (localhost_access_log.*) older than 300 days from the logs PVC. Keeps disk usage bounded without losing recent history.

`log-stats-configmap.yaml`

Injects rda-tds-helm/scripts/log_stats.py into a ConfigMap (log-stats-script) so the CronJob can mount and run it without baking it into a container image.

`log-stats-cronjob.yaml`

Runs daily at 4:00 AM (after log cleanup at 3 AM). Executes log_stats.py inside a python:3.11-slim container with the logs PVC mounted. Parses all localhost_access_log.* files and appends a new daily summary row to access_log_stats.txt in the same logs volume.

`pv-s3-backup.yaml`

Runs daily at 12:30 AM (Mountain Time). Uses rclone to copy:

TDS index files from tds-persist → s3://gdex/tds-data/ on Boreas
Tomcat access logs (*.txt) from logs-persist → s3://gdex/tds-tomcat-logs/ on Boreas (when backup.logs.enabled: true)

Tolerates permission-denied errors (TDS caches), broken symlinks, and race-condition disappearing files. Uploads a run report to s3://gdex/tds-data/_reports/<timestamp>.txt. S3 credentials come from the backup-s3-creds Kubernetes Secret.

Daily Automated Pipeline

12:30 AM  pv-s3-backup (k8s CronJob)
            └─ rclone: tds-persist + logs-persist → Boreas S3

01:00 AM  gdex-tds-data-autoscan.sh (PBS, Casper)
            └─ auto_add_data_tds.py (Prefect)
                 └─ scan rdadb → create catalog_d*.xml → update catalog.xml
                      └─ write auto_add_data_tds_<datetime>.log
                           └─ git push branch → GitHub Actions → auto-PR → merge → main
                                └─ Argo CD detects main → redeploy TDS pod

03:00 AM  log-clean-cronjob (k8s CronJob)
            └─ delete access logs older than 300 days

04:00 AM  log-stats-cronjob (k8s CronJob)
            └─ log_stats.py: parse access logs → append to access_log_stats.txt

06:00 AM  gdex-tds-add-control.sh (PBS, Casper)
            └─ add_control_tds.py (Prefect)
                 └─ parse today's log → createCTL.py → dsrqst -sc → verify URL reachable

Monday 01:01 AM  heapdump-clean-cronjob (k8s CronJob)
            └─ delete *.hprof heap dumps older than 7 days

access_log_stats.txt written at 4 AM is uploaded to Boreas by the next day's 12:30 AM backup. src/gen_stats_plot.py reads from s3://gdex/tds-tomcat-logs/access_log_stats.txt and produces tds_usage_stats.html.

Prefect Workflows — `prefect-workflow/`

`auto_add_data_tds.py` — Daily autoscan (1:00 AM)

Scans rdadb for datasets not yet in the TDS catalog and adds supported ones (NetCDF, GRIB1/2).

Query all dataset IDs from rdadb
Query all dataset IDs in rda-tds/content/catalog.xml
For each new dataset: check format, check for CDF5 files (skip if any), check GLADE data files exist, check exclude list
Run src/createXML.py → insert catalogRef into catalog.xml in alphabetical order
Write prefect-workflow/auto_add_data_tds_<datetime>.log

Cron runner: ~/cron/tds/gdex-tds-data-autoscan.sh (PBS, #PBS -q gdex, 1 CPU / 10 GB / 30 min)

`add_control_tds.py` — Post-merge control registration (6:00 AM)

Reads today's autoscan log and registers each new dataset's TDS URL in the GDEX data portal so it appears in the download column.

Parse auto_add_data_tds_<date>*.log for dataset IDs
For each dataset ID:
- Run src/createCTL.py <dsid> to generate new CTL index lines
- Run dsrqst -gc -ds <dsid> to get existing control entries
- Validate each TDS URL is reachable (HTTP 200–399); skip unreachable URLs
- Run dsrqst -sc -ds <dsid> -if <combined_ctl> -md -nc to register
Write prefect-workflow/add_control_tds_<datetime>.log for successfully registered datasets

Runs 5 hours after autoscan to allow the Argo CD redeploy cycle to complete before URL validation.

Cron runner: ~/cron/tds/gdex-tds-add-control.sh (PBS, #PBS -q gdex, 1 CPU / 1 GB / 30 min)

`delete_control_tds.py` / `modify_control_tds.py`

Manual workflows for removing or modifying control entries on existing datasets.

`src/` — Helper Scripts

Script	Purpose
`createXML.py`	Generate `catalog_d<dsid>.xml` for a single dataset
`createCTL.py`	Generate CTL index lines for `dsrqst` registration
`createAllXMLS.bash`	Batch-run `createXML.py` for all datasets
`gen_stats_plot.py`	Load `access_log_stats.txt` from Boreas S3 and write `tds_usage_stats.html`

Deployment Workflow (code change)

Merge changes to rda-tds/ → GitHub Action builds and pushes Docker image to registry
Update the image digest in rda-tds-helm/values.yaml (webapp.tds.image)
Argo CD detects the Helm chart change and redeploys the pod

Changes to rda-tds-helm/ only (e.g., resource limits, CronJob schedules) redeploy without rebuilding the image.

K8s Health Check

Tool	URL	Purpose
Argo CD	https://mlc1-argo.k8s.ucar.edu	Chart sync state, pod health, rollout history
Grafana	https://grafana.k8s.ucar.edu	Access log metrics from the log sidecar

Checking the Container

kubectl exec -it deploy/rda-tds -n rda-tds -c rda-tds -- /bin/bash

Manual Steps — Adding a Dataset

If running the automated pipeline is not possible, the manual steps are:

# 1. Generate the individual catalog XML
python src/createXML.py d<dsid> rda-tds/content/

# 2. Add catalogRef to catalog.xml and commit/push → Argo CD redeploys

# 3. After redeploy, generate CTL lines
python src/createCTL.py d<dsid>

# 4. Get existing control entries
dsrqst -gc -ds d<dsid> > d<dsid>.orig.ctl

# 5. Append CTL lines from step 3 to d<dsid>.orig.ctl

# 6. Register the combined control file
dsrqst -sc -ds d<dsid> -if d<dsid>.orig.ctl -md -nc

Name		Name	Last commit message	Last commit date
Latest commit History 1,104 Commits
.github/workflows		.github/workflows
prefect-workflow		prefect-workflow
rda-tds-helm		rda-tds-helm
rda-tds		rda-tds
src		src
templates		templates
test		test
.gitignore		.gitignore
BACKUP.md		BACKUP.md
LICENSE		LICENSE
README.md		README.md
tds_usage_stats.html		tds_usage_stats.html

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

GDEX TDS — Deployment & Operations Guide

Overview

Repository Tree

Helm Chart — rda-tds-helm/

values.yaml

Templates

deployment.yaml

service.yaml

ingress.yaml

deny-backend.yaml

ingress-dap4-block.yaml

ingress-wmsold-block.yaml

pvc-tds.yaml

pvc-log.yaml

heapdump-clean-cronjob.yaml

log-clean-cronjob.yaml

log-stats-configmap.yaml

log-stats-cronjob.yaml

pv-s3-backup.yaml

Daily Automated Pipeline

Prefect Workflows — prefect-workflow/

auto_add_data_tds.py — Daily autoscan (1:00 AM)

add_control_tds.py — Post-merge control registration (6:00 AM)

delete_control_tds.py / modify_control_tds.py

src/ — Helper Scripts

Deployment Workflow (code change)

K8s Health Check

Checking the Container

Manual Steps — Adding a Dataset

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Helm Chart — `rda-tds-helm/`

`values.yaml`

`deployment.yaml`

`service.yaml`

`ingress.yaml`

`deny-backend.yaml`

`ingress-dap4-block.yaml`

`ingress-wmsold-block.yaml`

`pvc-tds.yaml`

`pvc-log.yaml`

`heapdump-clean-cronjob.yaml`

`log-clean-cronjob.yaml`

`log-stats-configmap.yaml`

`log-stats-cronjob.yaml`

`pv-s3-backup.yaml`

Prefect Workflows — `prefect-workflow/`

`auto_add_data_tds.py` — Daily autoscan (1:00 AM)

`add_control_tds.py` — Post-merge control registration (6:00 AM)

`delete_control_tds.py` / `modify_control_tds.py`

`src/` — Helper Scripts

Packages