refactor(waterdata): make parallel_chunks take an integer, not low/medium/high

Replace the three-tier "low"/"medium"/"high" enum with a plain positive integer: parallel_chunks(n) fans a call out into n sub-requests. More expressive (any n, not just 2/8/32), precise, and mirrors the int-valued API_USGS_CONCURRENT. Removes ParallelChunksLevel, the _LEVEL_CAPS/_MAX_PARALLEL_CHUNKS constants, and _resolve_level; validation is now an inline positive-int check (rejects 0, negatives, floats, bool, str). n is bounded below by the byte-limit minimum and above by the number of values to split; n=1 is an explicit no-op. Docstrings, NEWS, user guide, README, exports, and tests updated; 2/8/32 remain as documented examples.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: thodson-usgs

NEWS.md

@@ -1,4 +1,4 @@
-**07/01/2026:** Added `waterdata.parallel_chunks(...)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.parallel_chunks("high"): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. The level is one of `"low"`, `"medium"`, or `"high"` — typed as `waterdata.ParallelChunksLevel` (a `typing.Literal`), so a type checker rejects any other value and an invalid string raises `ValueError` at the `with`. Each level caps the *total* number of sub-requests the call is split into — `"low"` / `"medium"` / `"high"` cap at 2 / 8 / 32 overall, across every multi-value argument combined, not per argument. That ceiling is a parallel_chunks constant, deliberately independent of the fan-out concurrency (`API_USGS_CONCURRENT`): how finely a query splits is orthogonal to how many sub-requests run at once. Capping the aggressive end at 32 bounds the blast radius — an accidental `"high"` on a huge list, or a call with several multi-value arguments, can't explode into thousands of sub-requests. There is no "off" level: not entering the block *is* off. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.parallel_chunks`.
+**07/01/2026:** Added `waterdata.parallel_chunks(n)` — a context manager to control how finely the OGC `waterdata` getters split multi-value requests. By default the getters chunk only as much as the server's ~8 KB URL limit forces (the fewest sub-requests). But because every sub-request paginates, splitting a large result further costs little or no extra quota when each sub-request still spans many pages, so a caller who *knows* their pull is large can opt into a finer split for smoother progress, more even concurrency, and a smaller unit of retry/resume: `with waterdata.parallel_chunks(32): df, md = waterdata.get_daily(monitoring_location_id=many_sites)`. `n` is a positive integer (e.g. `2`, `8`, `32`) — the number of sub-requests to fan the call out into; a non-integer or non-positive value raises `ValueError` at the `with`. It caps the *total* sub-request count across every multi-value argument combined (not per argument), bounded below by what the byte limit already forces and above by how many values there are to split, so several multi-value arguments can't multiply past it and `n=1` asks for no extra fan-out. Each sub-request costs a request against your hourly rate limit, and because how many run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32) an `n` beyond that adds quota without adding parallelism, so the useful range is roughly `2` up to `API_USGS_CONCURRENT`. It is deliberately a scoped `with` block rather than an automatic behavior or an environment variable, since the library can't tell in advance whether a query is large — a short-window query might fit in a single page, where extra chunks would only burn quota. Also exported at the top level as `dataretrieval.parallel_chunks`.
 
 **06/23/2026:** **Breaking change (1.2.0):** the minimum supported Python is now **3.10** (`requires-python = ">=3.10"`). 3.9 support was already effectively broken — the `waterdata` module's dependencies (`anyio`, the test stack) require 3.10+, and the `waterdata` test modules already skipped on <3.10. `anyio` is now declared as a direct dependency (it is imported directly by `waterdata`), and the CI/ruff/mypy targets move to 3.10. Also fully removed the deprecated `variable_info` metadata property: the `NWIS_Metadata` override only warned and returned `None` (it relied on the defunct `get_pmcodes`), and the `BaseMetadata` abstract is gone too since nothing implemented it — accessing `.variable_info` now raises `AttributeError`. `site_info` is unaffected.
 

README.md

@@ -111,11 +111,12 @@ By default the getters split a multi-value request only as far as the server's
 ~8 KB URL limit forces — the fewest sub-requests. For a **large, paginated**
 pull that is needlessly conservative: every sub-request pages through its own
 results, so dividing the query into more, smaller sub-requests lets those pages
-be fetched **in parallel**. `parallel_chunks` opts a single call into that
-finer split. It pays off only when the result is large enough to span many
-pages *and* the query has a multi-value argument to divide (such as a list of
-monitoring locations); on a small query — or one with nothing to split — it just
-adds requests, so it is a deliberate, scoped `with` block, never the default.
+be fetched **in parallel**. `parallel_chunks(n)` opts a single call into that
+finer split, fanning it out into `n` sub-requests. It pays off only when the
+result is large enough to span many pages *and* the query has a multi-value
+argument to divide (such as a list of monitoring locations); on a small query —
+or one with nothing to split — it just adds requests, so it is a deliberate,
+scoped `with` block, never the default.
 
 ```python
 from dataretrieval import waterdata
@@ -124,34 +125,35 @@ from dataretrieval import waterdata
 # enough to span many pages, so it profits from a finer split.
 sites, _ = waterdata.get_monitoring_locations(state="Ohio", site_type_code="ST")
 
-with waterdata.parallel_chunks("high"):        # "low" | "medium" | "high"
+with waterdata.parallel_chunks(32):              # fan out into 32 sub-requests
     df, md = waterdata.get_daily(
         monitoring_location_id=sites["monitoring_location_id"].tolist(),
         parameter_code="00060",                  # discharge
         time="2004-01-01/2023-12-31",
     )
 ```
 
-`"high"` fans a call out into up to 32 sub-requests, `"medium"` up to 8, and
-`"low"` up to 2 — a fixed ceiling, so an accidental setting on a huge list can't
-explode into thousands of requests.
+`n` is the number of sub-requests to fan the call out into. It is capped by how
+many values there are to split, and each sub-request costs a request against
+your hourly [rate limit](https://api.waterdata.usgs.gov/signup/); since how many
+run *at once* is capped separately by `API_USGS_CONCURRENT` (default 32), the
+useful range is roughly `2` up to that value.
 
 Benchmark — 271 Ohio discharge sites (`get_daily`, `parameter_code="00060"`),
-cold cache, each level run against its own time window so results are not
-cache-served, with the page size fixed so every level fetches roughly the same
+cold cache, each `n` run against its own time window so results are not
+cache-served, with the page size fixed so every run fetches roughly the same
 number of pages (isolating the effect of parallelism):
 
-| level          | parallelism | pages | wall-clock                | speedup |
-| -------------- | ----------- | ----- | ------------------------- | ------- |
-| `default`      | 1           | 32    | 23.9 s / 27.4 s (2 runs)  | 1×      |
-| `"medium"` (8) | 8           | 37    | 4.1 s / 4.3 s             | ~6×     |
-| `"high"` (32)  | 32          | 44    | 2.0 s                     | ~12×    |
+| `n`  | parallelism | pages | wall-clock                | speedup |
+| ---- | ----------- | ----- | ------------------------- | ------- |
+| off  | 1           | 32    | 23.9 s / 27.4 s (2 runs)  | 1×      |
+| `8`  | 8           | 37    | 4.1 s / 4.3 s             | ~6×     |
+| `32` | 32          | 44    | 2.0 s                     | ~12×    |
 
 The gain comes from overlapping each sub-request's per-page latency and
 server-side work — a genuinely large pull at the default page size shows a
-similar multi-fold speedup. The extra sub-requests each cost one request against
-your hourly [rate limit](https://api.waterdata.usgs.gov/signup/), so reserve the
-aggressive levels for pulls you know are large.
+similar multi-fold speedup. The extra sub-requests each cost quota, so reserve a
+large `n` for pulls you know are large.
 
 Visit the
 [API Reference](https://doi-usgs.github.io/dataretrieval-python/reference/waterdata.html)

dataretrieval/__init__.py

@@ -44,10 +44,10 @@
     URLTooLong,
 )
 
-# Parallel-chunks control (a context manager) and its level type. Defined with
-# the chunker in ``dataretrieval.ogc.chunking``; surfaced here for a stable
-# public path ``from dataretrieval import parallel_chunks``.
-from dataretrieval.ogc.chunking import ParallelChunksLevel, parallel_chunks
+# Parallel-chunks control (a context manager). Defined with the chunker in
+# ``dataretrieval.ogc.chunking``; surfaced here for a stable public path
+# ``from dataretrieval import parallel_chunks``.
+from dataretrieval.ogc.chunking import parallel_chunks
 
 # Resumable chunk-interruption exceptions. They are defined in
 # ``dataretrieval.ogc.interruptions`` rather than ``dataretrieval.exceptions``
@@ -100,6 +100,5 @@
     "ServiceInterrupted",
     # parallel-chunks control (defined in ogc.chunking)
     "parallel_chunks",
-    "ParallelChunksLevel",
     "__version__",
 ]

dataretrieval/ogc/chunking.py

@@ -11,9 +11,9 @@
 
 Parallel chunks: the planner is conservative by default — it splits only as far as
 the byte limit forces. A caller who knows their result is large can opt into a
-finer split via the ``parallel_chunks`` context manager
-(``"low"`` / ``"medium"`` / ``"high"``); the resolved cap drives
-:meth:`ChunkPlan._refine`. See ``parallel_chunks`` for the why and the when.
+finer split via the ``parallel_chunks(n)`` context manager, which fans the query
+out into ``n`` parallel sub-requests; ``n`` drives :meth:`ChunkPlan._refine`. See
+``parallel_chunks`` for the why and the when.
 
 This module owns the *execution* half — the event loop and bounded
 concurrency that drive a plan to completion (``ChunkedCall``) plus the
@@ -77,7 +77,7 @@
 from collections.abc import Callable, Iterator
 from contextlib import contextmanager
 from contextvars import copy_context
-from typing import Any, Literal, cast, get_args
+from typing import Any, cast
 
 import httpx
 import pandas as pd
@@ -182,70 +182,16 @@ def get_active_client() -> httpx.AsyncClient | None:
 # Parallel-chunks dial: opt-in to fan a query out *more finely* than the byte
 # limit alone requires. Scoped to a ``with parallel_chunks(...):`` block (a
 # ContextVar), deliberately NOT an env var (see :func:`parallel_chunks` for
-# why). The ambient holds the resolved cap on the plan's total sub-request
-# count; ``0`` (the default, outside any block) means "chunk only as much as
-# the byte limit needs".
+# why). The ambient holds ``n`` — the requested cap on the plan's total
+# sub-request count; ``0`` (the default, outside any block) means "chunk only as
+# much as the byte limit needs".
 _parallel_chunks: Ambient[int] = Ambient("ogc_parallel_chunks", 0)
 
-#: The three accepted parallel_chunks levels, as a typing ``Literal`` so a type
-#: checker rejects any other value at the call site.
-ParallelChunksLevel = Literal["low", "medium", "high"]
-
-#: Valid levels derived from the type, so ``ParallelChunksLevel`` stays the single
-#: source of truth for what ``_resolve_level`` accepts (mirrors the
-#: ``get_args``-based ``_VALID_ON_TIE`` / ``_VALID_FILE_TYPES`` in sibling
-#: modules).
-_VALID_LEVELS: tuple[ParallelChunksLevel, ...] = get_args(ParallelChunksLevel)
-
-# The parallel_chunks ceiling on the plan's total sub-request count —
-# deliberately NOT tied to the concurrency default: fan-out *volume* (how many
-# sub-requests a query becomes) is orthogonal to how many run at once
-# (``API_USGS_CONCURRENT``). 32 is a sane ceiling on the whole call (across
-# every axis, not per axis — see ``ChunkPlan._refine``) that bounds the blast
-# radius of an accidental ``"high"`` on a very long list or several multi-value
-# arguments at once; the milder levels are a quarter and a sixteenth of it (the
-# three are spaced 4x apart).
-_MAX_PARALLEL_CHUNKS = 32
-_LEVEL_CAPS: dict[str, int] = {
-    "low": _MAX_PARALLEL_CHUNKS // 16,  # 2
-    "medium": _MAX_PARALLEL_CHUNKS // 4,  # 8
-    "high": _MAX_PARALLEL_CHUNKS,  # 32
-}
-
-
-def _resolve_level(level: ParallelChunksLevel) -> int:
-    """
-    Map a parallel_chunks level name to its total sub-request cap.
-
-    Parameters
-    ----------
-    level : {"low", "medium", "high"}
-        The user-supplied level.
-
-    Returns
-    -------
-    int
-        The maximum total sub-requests for the whole call at that level
-        (``2`` / ``8`` / ``32``).
-
-    Raises
-    ------
-    ValueError
-        If ``level`` is anything other than ``"low"``, ``"medium"``, or
-        ``"high"`` — raised eagerly (at the ``with`` statement) so a typo fails
-        loudly rather than silently doing nothing.
-    """
-    if level not in _VALID_LEVELS:
-        raise ValueError(
-            f"parallel_chunks level must be one of {_VALID_LEVELS}; got {level!r}."
-        )
-    return _LEVEL_CAPS[level]
-
 
 @contextmanager
-def parallel_chunks(level: ParallelChunksLevel) -> Iterator[None]:
+def parallel_chunks(n: int) -> Iterator[None]:
     """
-    Scope how finely the OGC getters chunk multi-value requests.
+    Fan the OGC getters' multi-value requests out into ``n`` parallel sub-requests.
 
     By default the Water Data / NGWMN getters chunk a request only as much as
     the server's ~8 KB URL-byte limit forces — the fewest sub-requests that
@@ -267,26 +213,27 @@ def parallel_chunks(level: ParallelChunksLevel) -> Iterator[None]:
     automatic behavior or a process-wide environment variable — scoping it to a
     ``with`` block keeps an aggressive setting from leaking into unrelated calls
     and accidentally spending quota. Outside any block the getters use the
-    conservative default; there is no "off" level because *not* entering the
-    block is off. Only the OGC getters (Water Data, NGWMN) read this; wrapping a
-    legacy NWIS call in the block is a harmless no-op.
+    conservative default. Only the OGC getters (Water Data, NGWMN) read this;
+    wrapping a legacy NWIS call in the block is a harmless no-op.
 
     Parameters
     ----------
-    level : {"low", "medium", "high"}
-        How aggressively to chunk within the block. Each level caps the
-        *total* number of sub-requests the call is split into — ``2`` / ``8``
-        / ``32`` for ``"low"`` / ``"medium"`` / ``"high"`` — or one
-        sub-request per remaining atom once that's fewer than the cap. The
-        cap applies to the whole call, not per multi-value argument: with
-        several multi-value arguments the axes are refined together against
-        the same shared ceiling (their cartesian product does not multiply
-        past it). The ceiling is fixed (it is *not* tied to
-        ``API_USGS_CONCURRENT``: how finely a query splits is orthogonal to
-        how many sub-requests run at once); capping ``"high"`` at ``32`` keeps
-        an accidental aggressive level on a very long list — or several
-        multi-value arguments at once — from exploding into thousands of
-        sub-requests.
+    n : int
+        The number of sub-requests to fan the whole call out into — a positive
+        integer such as ``2``, ``8``, or ``32``. It caps the plan's *total*
+        sub-request count (the cartesian product across every multi-value
+        argument combined, not per argument), so several multi-value arguments
+        cannot multiply past it. The actual count is bounded below by what the
+        ~8 KB URL limit already forces and above by the number of values there
+        are to split, so an ``n`` larger than the input allows simply yields one
+        sub-request per value; ``n=1`` asks for no extra fan-out.
+
+        Each sub-request fetches at least one page, so it costs at least one
+        request against your hourly rate limit — a larger ``n`` spends more
+        quota. And because how many sub-requests run *at once* is capped
+        separately by ``API_USGS_CONCURRENT`` (default 32), an ``n`` beyond that
+        adds quota without adding parallelism; the useful range is roughly ``2``
+        up to ``API_USGS_CONCURRENT``.
 
     Yields
     ------
@@ -295,22 +242,29 @@ def parallel_chunks(level: ParallelChunksLevel) -> Iterator[None]:
     Raises
     ------
     ValueError
-        If ``level`` isn't ``"low"``, ``"medium"``, or ``"high"`` — raised on
-        ``with`` entry, before any request is issued.
+        If ``n`` is not a positive integer — raised on ``with`` entry, before
+        any request is issued, so a bad value fails loudly rather than silently
+        doing nothing.
 
     Examples
     --------
     >>> from dataretrieval import waterdata
-    >>> with waterdata.parallel_chunks("high"):
+    >>> with waterdata.parallel_chunks(32):
     ...     df, md = waterdata.get_daily(
     ...         monitoring_location_id=many_sites, parameter_code="00060"
     ...     )  # doctest: +SKIP
 
     See Also
     --------
-    ChunkPlan._refine : the planning-side effect of the level.
+    ChunkPlan._refine : the planning-side effect of ``n``.
     """
-    with _parallel_chunks(_resolve_level(level)):
+    # ``bool`` is an ``int`` subclass but nonsensical here; reject it explicitly.
+    if not isinstance(n, int) or isinstance(n, bool) or n < 1:
+        raise ValueError(
+            f"parallel_chunks(n): n must be a positive integer (e.g. 2, 8, 32); "
+            f"got {n!r}."
+        )
+    with _parallel_chunks(n):
         yield
 
 
@@ -779,11 +733,11 @@ def wrapper(
             finalize: _Finalize = _passthrough_result,
         ) -> tuple[pd.DataFrame, Any]:
             limit = _OGC_URL_BYTE_LIMIT if url_limit is None else url_limit
-            # Read the parallel_chunks dial from the ambient set by
+            # Read the parallel_chunks dial ``n`` from the ambient set by
             # ``parallel_chunks`` (0 = off outside any such block; otherwise the
-            # per-axis sub-chunk cap). It only affects *planning*, done here up
-            # front, so a later resume — which re-issues the already-planned
-            # sub-requests — needs no snapshot.
+            # requested total sub-request cap). It only affects *planning*, done
+            # here up front, so a later resume — which re-issues the
+            # already-planned sub-requests — needs no snapshot.
             plan = ChunkPlan(
                 args, build_request, limit, max_chunks_per_axis=_parallel_chunks.get()
             )