Home

Welcome to the Data.gov wiki. Pages are grouped below by topic — see the sidebar for the full alphabetical list.

System components

Page	What it covers
Site components	Map of everything that makes up Data.gov
www.data.gov	Public homepage (WordPress, cloud.gov Pages)
catalog.data.gov	Dataset search/discovery app (Python/Flask + OpenSearch)
harvest.data.gov	Automated metadata ingestion service
inventory.data.gov	Agency-facing metadata management (CKAN)
Static sites	resources.data.gov, strategy.data.gov
Updating organization logos on Catalog	How to add/update an org's logo

Harvesting

Page	What it covers
Harvesting 2.0	Current harvest pipeline: architecture and terminology
Harvest O&M and Debugging	Operational procedures, Harvest API query examples
Broken Harvest Sources	Diagnosing agency-reported harvest source issues
Examples of Harvest Job Errors	Catalog of known error types and what they mean

Operations & monitoring

Page	What it covers
Runbook	Primary incident/on-call runbook
Operation and Maintenance Responsibilities	O&M role and duties
Operation and Maintenance Event Notes	Log of past O&M events
Automated O&M Tasks	Scheduled jobs/actions that self-report failures
Log Review	What's expected vs. what needs investigation, New Relic
Metrics	Jobs producing usage metrics
Dataset metrics	GA4 + New Relic dataset usage tracking
Google Analytics	GA account structure and access
Break Glass deployment	Emergency deploys outside normal CI/CD
How to Rotate New Relic License Key	License key rotation steps
Cloud.gov Apps Spaces Memory Usage	Memory usage tracking spreadsheet

Infrastructure & security

Page	What it covers
cloud.gov	Platform reference and helpful links
DNS	DNS change process (via 18F, Terraform)
TLS SSL Certificates	Certificate management
Firewall Change Requests	Ingress/egress firewall process
GSA VPN	VPN access to BSP environments
WAF and rate limiting	AWS WAF behavior and rate-limit handling
Egress & ingress proxies	Why/how we proxy traffic
SAML2 authentication	SAML2 setup (inventory.data.gov only)
Login.gov SAML certificate rotation steps	Annual cert rotation process
Add Login.gov as Authentication	Integrating Login.gov (Flask apps)
Dependency scanning	Snyk/Dependabot triage
Significant Change rubric	Security Impact Analysis rubric
Checklist for new repositories	Pre-production repo checklist

Team & process

Page	What it covers
Onboarding Offboarding	New team member setup and access
Pull Request Template	Current per-repo PR templates
Slack	Channel structure and usage
Team Collaboration and Pair Programming	Meeting/pairing conventions
Docker Best Practices	Docker/Compose conventions
Managing Forks	Fork ownership guidelines
Data.gov Mission and Strategy	Team history and mission
Coding Best Practices	⚠ Flagged outdated, needs review
Email	⚠ TODO, tracked in issue #3798

Legacy CKAN reference (inventory.data.gov only)

catalog.data.gov migrated off CKAN to a custom Python/Flask app in 2025. These pages now apply only to inventory.data.gov, which remains CKAN-based, unless noted otherwise.

Page	What it covers
CKAN commands	Common CKAN commands
CKAN API tasks	Recurring CKAN API tasks
Catalog CKAN 2.8 upgrade	Historical CKAN 2.8 upgrade notes
Solr on ECS - EFS	Solr ops (inventory only; catalog uses OpenSearch)
Duplicate Harvest Source Cleanup and Other Nuclear Harvest Fixes	Manual harvest-source cleanup (legacy H1.0)
Stuck Harvest Fix	Stuck-harvest procedures for legacy H1.0
Geospatial metadata	Geospatial metadata standards (mostly still current)
Collections	Collections handling

Deprecated / superseded

Kept for history — not needed day-to-day.

• Alerts Event - Standard Operating Procedure → moved to Runbook
• Web Application Security Scans → moved to Operation and Maintenance Responsibilities
• Keypair Rotation — deprecated
• Nessus Agent Upgrade — deprecated
• BSP or FCS requests — being phased out