Rewrite xpublish-opendap with native DAP2/DAP4 protocol layer

[jhamman]

This is a ground-up rewrite of xpublish-opendap. It replaces the opendap-protocol dependency with a native in-repo protocol implementation, adds full DAP4 support, and introduces an async-first architecture with
memory-bounded streaming.

Note

This PR was built with Claude but not entirely by Claude. I wrote and edited much of the design docs and drove on the high degree of testing contained here. That said, there is a big diff in this PR. I don't expect any human to volunteer to review the whole thing. Instead, I think we should start using this in real world applications for a bit to build confidence in it before deciding if its worth merging.

What changed

• Native protocol layer (xpublish_opendap/dap/): Clean-room implementation of DAP2 and DAP4 encoding, replacing the external opendap-protocol library. Includes a type system mapping 15 numpy dtypes to DAP wire types, a
  recursive-descent constraint expression parser, and xarray-to-DAP subsetting with memory estimation.
• DAP4 support: DMR (Dataset Metadata Response) XML generation, chunked binary encoding with CRC32 checksums and native byte order, DAP4 constraint syntax, and Dataset Services Response. All four DAP4 response types
  (.dmr, .dap, .dsr, error) alongside all six DAP2 types.
• Async streaming architecture: All route handlers are async def. Data loading runs in a thread pool executor with semaphore-bounded concurrency. Large arrays use slab streaming (split along the most-chunked dimension,
  yield per-slab) to bound HTTP response memory. A dask-graph fast path fuses dtype conversion into the computation graph for chunked data.
• Constraint-first pipeline: Subsetting is applied before data loading (not after), and requests exceeding max_request_memory_bytes (default 512 MB) are rejected with 413.

Test suite

In my opinion, this is the best part of this PR. We now have a very thorough test suite

440 tests:

• Unit tests: type mapping, constraint parsing, xarray adapter, DAP2 protocol (DDS/DAS/DODS/XDR encoding), DAP4 protocol (DMR/chunked binary/CRC32), slab streaming, dask-graph fast path, error responses, HTTP headers
• Cross-language integration tests: netCDF4-python, PyDAP, xarray (multiple engines), NCO (ncks, ncdump), Julia (NCDatasets.jl), R (ncdf4)
• 99% code coverage, strict mypy, ruff linting

Benchmarks

New benchmark suite (benchmarks/) with ~21 scenarios covering metadata generation, data responses, and constraint parsing across multiple dataset sizes. Includes an isolated IO benchmark (io_bench.py) for profiling the
encode pipeline without HTTP overhead, and a comparison workflow for regression detection between the release and feature branch.

Breaking changes

• Removed opendap-protocol dependency
• All handlers are now async (was sync)
• New module structure (dap/ replaces dap_xarray.py)
• DAP2 int8 promoted to Int16 (per OPeNDAP spec)

[jhamman]

Some examples from our the latest benchmarks: run on my laptop at home so all the caveats apply w.r.t. network bandwidth, etc.

=== air dataset ===

Metric: mean_latency_s
Baseline: feature/dap4@68babb6 (release_air) (release_air.json)

  [1] feature/dap4@68babb6 (release_air) (baseline)  2026-03-04T05:18:00.622551+00:00
  [2] feature/dap4@68babb6 (dap4_air)  2026-03-04T05:18:22.156148+00:00

Scenario                          [1]         [2]       delta
-------------------------------------------------------------
metadata-dds-full               5.4ms       2.1ms  -60.9% faster    [300B/215B]
metadata-das-full               8.1ms       2.0ms  -75.2% faster    [1.4KB/1.3KB]
metadata-ver                      N/A       0.6ms                [0B/37B]
metadata-dmr-full                 N/A       2.1ms                [0B/3.7KB]
metadata-dsr                      N/A       0.7ms                [0B/1.1KB]
metadata-dds-projected          4.7ms       2.0ms  -56.9% faster    [215B/215B]
data-dap2-scalar                6.1ms       3.6ms  -41.4% faster    [265B/264B]
data-dap2-single-timestep       6.1ms       3.6ms  -41.8% faster    [10.9KB/10.9KB]
data-dap2-10-timesteps          6.3ms       3.7ms  -41.8% faster    [104.1KB/104.1KB]
data-dap2-zonal-transect        6.2ms       3.4ms  -45.2% faster    [891B/890B]
data-dap2-strided                 N/A       3.6ms                [0B/14.1KB]
data-dap4-scalar                  N/A       3.8ms                [0B/3.8KB]
data-dap4-single-timestep         N/A       3.8ms                [0B/14.4KB]
data-dap4-10-timesteps            N/A       4.2ms                [0B/107.6KB]
-------------------------------------------------------------

=== ifs dataset === (icechunk data stored in cloudflare R2)

Metric: mean_latency_s
Baseline: feature/dap4@b2eca17 (release_ifs) (release_ifs.json)

  [1] feature/dap4@b2eca17 (release_ifs) (baseline)  2026-03-04T04:49:41.237493+00:00
  [2] feature/dap4@b2eca17 (dap4_ifs)  2026-03-04T05:08:19.784679+00:00

Scenario                          [1]         [2]       delta
-------------------------------------------------------------
metadata-dds-full              24.7ms       8.0ms  -67.7% faster    [3.2KB/3.1KB]
metadata-das-full              14.9ms       2.5ms  -83.3% faster    [5.5KB/5.4KB]
metadata-ver                      N/A       0.6ms                [0B/37B]
metadata-dmr-full                 N/A       3.0ms                [0B/11.9KB]
metadata-dsr                      N/A       0.6ms                [0B/1.1KB]
metadata-dds-projected          4.5ms       2.2ms  -50.0% faster    [301B/302B]
data-dap2-scalar              411.2ms     390.2ms  -5.1% faster    [377B/365B]
data-dap2-single-timestep    3367.4ms    3329.8ms  -1.1% ~same    [37.2KB/37.0KB]
data-dap2-10-timesteps      30376.1ms   30115.8ms  -0.9% ~same    [362.9KB/362.7KB]
data-dap2-zonal-transect      428.7ms     413.9ms  -3.5% ~same    [41.0KB/41.0KB]
data-dap2-two-var             778.7ms     864.4ms  +11.0% slower    [725B/706B]
data-dap2-strided                 N/A    8071.9ms                [0B/26.6KB]
data-dap4-scalar                  N/A     432.5ms                [0B/3.6KB]
data-dap4-single-timestep         N/A    3381.4ms                [0B/40.3KB]
data-dap4-10-timesteps            N/A   30107.4ms                [0B/365.9KB]
data-dap4-multi-var               N/A     832.3ms                [0B/4.4KB]
-------------------------------------------------------------

I would say these are not amazing but at least they are not a step backwards!

[abkfenris]

/Keanu Woah

This is gonna take a bit to poke through, but some thoughts so far:

• Multi-language testing is great! That was definitely something that hurt previously, and hearing about weird errors in specific languages with no ways to replicate those tests wasn't fun (here and elsewhere in Xpublish).
• I'm not sure what the current standard is for this, but it feels like some of the files for directing Claude/tracking it's progress through this work are mixed in where they might appear to apply for more general development. It feels like they are useful records, but maybe should be filed not at the top level.
  • Benchmarks are a little specific to this PR as well, but it would be cool to have them included as a part of the CI pipeline, even if slower PRs don't cause a failure.
• I'd love to see the prompts/sessions. Both to understand how to drive agents better, but I'm also thinking about how to do some OGC spec driven development on Xpublish-EDR to get that up to snuff. Also for a place to start with datatree comparability once we figure out what the overall Xpublish story might be for it.
• In some ways this feels like its so big that we're committing to all future development likely requiring at least some agentic assistance due to the amount that needs to be understood. Do we need AGENTS.md or similar files to help direct their development?
• I did not realize Github PR files view could break this chaotically so much with big PRs.

[abkfenris]

Some pondering through plugin.py and using that as a jumping board to start understanding the rest.

[abkfenris]

Do these need to be in the OpenDapPlugin class? They make it a little more confusing to read, and possibly test.

[abkfenris]

I think the responses could be streamed if the media types and headers are hoisted to the @router.get(), though maybe I'm reading where the errors are coming from wrong. I think this will also give more info for API docs clarity.

Suggested change

	@router.get(".das")
	async def das_response(
	request: Request,
	ds: xr.Dataset = Depends(deps.dataset),
	) -> Response:
	"""DAP2 Dataset Attribute Structure response."""
	try:
	raw_constraint = _extract_constraint(request)
	constraint = _parse_constraint(raw_constraint)
	plan = plan_subsetting(ds, constraint)
	subsetted = apply_plan(ds, plan)
	return dataset
	body = "".join(generate_das(subsetted))
	return Response(
	content=body,
	media_type=CONTENT_TYPES["das"],
	headers=_dap2_headers("das"),
	)
	@router.get(".das", media_type=CONTENT_TYPES["das"], headers=_dap2_headers("das"), response_class=StreamingResponse)
	async def das_response(
	request: Request,
	ds: xr.Dataset = Depends(deps.dataset),
	) -> Response:
	"""DAP2 Dataset Attribute Structure response."""
	try:
	raw_constraint = _extract_constraint(request)
	constraint = _parse_constraint(raw_constraint)
	plan = plan_subsetting(ds, constraint)
	subsetted = apply_plan(ds, plan)
	yield from generate_das(subsetted))

[abkfenris]

Might it be cleaner to split the shared planning steps off into their own dependency and include the exception handling they might trigger?

Suggested change

	@router.get(".dds")
	async def dds_response(
	request: Request,
	dataset_id: str = "default",
	ds: xr.Dataset = Depends(deps.dataset),
	cache: cachey.Cache = Depends(deps.cache),
	) -> dap.Dataset:
	"""Get a dataset that has been translated to opendap."""
	# get cached dataset if it exists
	cache_key = f"opendap_dataset_{dataset_id}"
	dataset = cache.get(cache_key)
	) -> Response:
	"""DAP2 Dataset Descriptor Structure response."""
	try:
	raw_constraint = _extract_constraint(request)
	constraint = _parse_constraint(raw_constraint)
	plan = plan_subsetting(ds, constraint)
	subsetted = apply_plan(ds, plan)
	# if not, convert the xarray dataset to opendap
	if dataset is None:
	dataset = dap_xarray.dap_dataset(ds, dataset_id)
	body = "".join(generate_dds(subsetted, dataset_id))
	return Response(
	content=body,
	media_type=CONTENT_TYPES["dds"],
	headers=_dap2_headers("dds"),
	)
	except DapError as e:
	return _error_response(e)
	except Exception:
	logger.exception("Unexpected error in DDS handler")
	return _error_response(
	DapError(code=500, message="Internal server error"),
	status_code=500,
	)
	def dap2_plan(request: Request, ds: xr.Dataset = Depends(deps.dataset)):
	"""Plan a DAP2 query"""
	try:
	raw_constraint = _extract_constraint(request)
	constraint = _parse_constraint(raw_constraint)
	plan = plan_subsetting(ds, constraint)
	except DapError as e:
	return _error_response(e)
	return plan
	@router.get(".dds")
	async def dds_response(
	dataset_id: str = "default",
	ds: xr.Dataset = Depends(deps.dataset),
	plan = Depends(dap2_plan)
	) -> Response:
	"""DAP2 Dataset Descriptor Structure response."""
	try:
	subsetted = apply_plan(ds, plan)
	body = "".join(generate_dds(subsetted, dataset_id))
	return Response(
	content=body,
	media_type=CONTENT_TYPES["dds"],
	headers=_dap2_headers("dds"),
	)
	except Exception:
	logger.exception("Unexpected error in DDS handler")
	return _error_response(
	DapError(code=500, message="Internal server error"),
	status_code=500,
	)

[abkfenris]

It might be easier to keep track of related functionality if instead of importing individual functions, the namespaces were imported.

Suggested change

	from xpublish_opendap.dap.constraint import (
	Constraint,
	parse_dap2_constraint,
	parse_dap4_constraint,
	)
	from fastapi.responses import StreamingResponse
	from xpublish import (
	Dependencies,
	Plugin,
	hookimpl,
	from xpublish_opendap.dap.dap2.das import generate_das
	from xpublish_opendap.dap.dap2.dds import generate_dds
	from xpublish_opendap.dap.dap2.dods import generate_dods
	from xpublish_opendap.dap.dap2.headers import (
	CONTENT_DESCRIPTIONS,
	CONTENT_TYPES,
	DAP2_HEADERS,
	)
	from xpublish_opendap.dap.dap2.responses import (
	generate_error,
	generate_help,
	generate_version,
	)
	from xpublish_opendap.dap.dap4.data import generate_dap4_data
	from xpublish_opendap.dap.dap4.dmr import generate_dmr
	from xpublish_opendap.dap.dap4.headers import CONTENT_TYPES as DAP4_CONTENT_TYPES
	from xpublish_opendap.dap.dap4.headers import DAP4_HEADERS
	from xpublish_opendap.dap.dap4.responses import generate_dap4_error, generate_dsr
	from xpublish_opendap.dap.xarray_adapter import apply_plan, plan_subsetting
	from xpublish_opendap.errors import DapError, RequestTooLargeError
	from xpublish_opendap.dap import constraint, dap2, dap4, xarray_adapter
	from xpublish_opendap.errors import DapError, RequestTooLargeError