feat(waterdata): get_queryables + queryables monitor + passthrough enablement (#333)

* feat(waterdata): add get_queryables + a live queryables monitor

Add `waterdata.get_queryables(collection)`, returning the OGC queryable
properties of a Water Data collection (`daily`, `continuous`,
`monitoring-locations`, ...) as a tidy `(DataFrame, BaseMetadata)` — one row per
filterable property with its type, title, and description.

Add `tests/waterdata_queryables_test.py`: offline parsing / error tests plus a
live monitor that compares each collection's advertised queryables against a
committed snapshot (`tests/data/waterdata_queryables.json`). The monitor fails
when the upstream API adds / removes / renames a queryable — the signal to
regenerate the snapshot and enable any new queryables on the matching getter.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sjb14HkwuCydKSKMsaXsgd

* feat(waterdata): enable arbitrary queryables as passthrough filters

The OGC data getters (`get_daily`, `get_continuous`, `get_peaks`, ...) exposed
~11 of each collection's ~50 queryables as named params; the rest — mostly the
shared monitoring-location attributes (`state_name`, `county_code`, `site_type`,
`altitude`, ...) now filterable on the data endpoints — were reachable only via
the raw `filter` CQL.

Accept any queryable as a passthrough kwarg: each OGC getter gains
`**queryables`, and the shared `_get_args` flattens it so an extra filter such
as `state_name="Wisconsin"` is normalized and sent exactly like a named param.
The service itself validates names (an unknown one returns HTTP 400 → typed
error), so no client-side queryable list is bundled.

The passthrough is provisional (see the PR description for the trade-off vs.
explicit per-property keyword arguments).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Sjb14HkwuCydKSKMsaXsgd

* refactor(waterdata): have get_queryables reuse _check_ogc_requests

get_queryables duplicated the GET + _raise_for_non_200 sequence that
_check_ogc_requests already does for the identical "queryables" request
shape (it's req_type's own default), and hardcoded OGC_API_URL instead of
going through _ogc_base_url_var like every other URL-building call site in
engine.py. _check_ogc_requests now also returns the raw httpx.Response
(needed for BaseMetadata), so get_queryables can call it directly; the one
existing caller (shaping._deal_with_empty's schema lookup) is updated to
unpack the new tuple.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* refactor(waterdata): simplify the queryables flatten in _get_args

A ``**queryables`` parameter is always a dict (empty when the caller passes
no passthrough kwargs), never None — so the None sentinel and the
``if queryables:`` truthiness guard were unnecessary scaffolding. Collapse to
a single ``local_vars.update(local_vars.pop("queryables", {}))``; updating
with an empty dict is a harmless no-op, so the common (no-passthrough) path
behaves identically.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(waterdata): flatten queryables before the state mutual-exclusion check; correct the passthrough docstring

Two issues surfaced by an xhigh code review of the **queryables passthrough:

1. State mutual-exclusion guard bypass. `_with_state` runs apply_state's
   `reject=("state_code", "state_name")` check on `locals()` before the
   `**queryables` flatten happened (that lived in `_get_args`, which runs
   after). So on get_time_series_metadata — whose `state_code` is NOT an
   explicit parameter — `get_time_series_metadata(state="Wisconsin",
   state_code="55")` let `state_code` land in `**queryables`, slipped past the
   guard, then flattened back in, silently sending BOTH state_name and
   state_code (server 400) instead of raising the documented ValueError.
   Fix: extract `_flatten_queryables` and call it at the top of `_with_state`
   (before apply_state) as well as in `_get_args`. The flatten pops its key so
   it's idempotent; the second call is a no-op. Now robust for every
   _with_state getter regardless of which reject target is an explicit param.

2. Misleading docstring example. The shared `**queryables` docstring asserted
   `state_name`/`site_type_code` as the example queryables, but
   channel-measurements and field-measurements-metadata expose neither, so the
   copied example would 400 for those two getters. Reword (uniformly across all
   11) to present those as common-but-not-universal attributes and point to
   get_queryables for the collection's actual queryables.

Adds a unit test pinning the passthrough-via-queryables conflict case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(waterdata): trim the **queryables docstring to point at get_queryables

Per review feedback: drop the examples and per-collection caveats; just tell
users to call get_queryables to see a collection's queryables. Applied
uniformly across all 11 getters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

Author: thodson-usgs

dataretrieval/ogc/engine.py

@@ -241,10 +241,14 @@ def _cql2_param(args: dict[str, Any]) -> str:
     return json.dumps(query, separators=(",", ":"))
 
 
-def _check_ogc_requests(endpoint: str, req_type: str = "queryables") -> dict[str, Any]:
+def _check_ogc_requests(
+    endpoint: str, req_type: str = "queryables"
+) -> tuple[dict[str, Any], httpx.Response]:
     """
     Sends an HTTP GET request to the specified OGC endpoint and request type,
-    returning the JSON response.
+    returning the parsed JSON body alongside the raw response (so a caller
+    that needs response-derived metadata, e.g. :class:`BaseMetadata`, doesn't
+    have to re-issue the request).
 
     Parameters
     ----------
@@ -258,6 +262,9 @@ def _check_ogc_requests(endpoint: str, req_type: str = "queryables") -> dict[str
     -------
     dict
         The JSON response from the OGC endpoint.
+    httpx.Response
+        The raw response, for callers that need it (URL, elapsed time,
+        headers).
 
     Raises
     ------
@@ -275,7 +282,7 @@ def _check_ogc_requests(endpoint: str, req_type: str = "queryables") -> dict[str
     _raise_for_non_200(resp)
     # ``Response.json`` is typed ``Any``; the OGC queryables/schema endpoints
     # return a JSON object, and callers index it as a dict.
-    return cast("dict[str, Any]", resp.json())
+    return cast("dict[str, Any]", resp.json()), resp
 
 
 def _ogc_query_params(

dataretrieval/ogc/shaping.py

@@ -190,7 +190,7 @@ def _deal_with_empty(
             # call (it goes away once requests move to their own module).
             from dataretrieval.ogc.engine import _check_ogc_requests
 
-            schema = _check_ogc_requests(endpoint=service, req_type="schema")
+            schema, _ = _check_ogc_requests(endpoint=service, req_type="schema")
             properties = list(schema.get("properties", {}).keys())
         return pd.DataFrame(columns=properties)
     return return_list

dataretrieval/waterdata/__init__.py

@@ -25,6 +25,7 @@
     get_latest_daily,
     get_monitoring_locations,
     get_peaks,
+    get_queryables,
     get_reference_table,
     get_samples,
     get_samples_summary,
@@ -62,6 +63,7 @@
     "get_monitoring_locations",
     "get_nearest_continuous",
     "get_peaks",
+    "get_queryables",
     "get_ratings",
     "get_reference_table",
     "get_samples",

dataretrieval/waterdata/api.py

@@ -38,6 +38,7 @@
     SAMPLES_URL,
     _accept_legacy_kwargs,
     _as_str_list,
+    _check_ogc_requests,
     _check_profiles,
     _construct_cql_request,
     _default_headers,
@@ -74,6 +75,7 @@ def get_daily(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Daily data provide one data value to represent water conditions for the
     day.
@@ -206,6 +208,10 @@ def get_daily(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -295,6 +301,7 @@ def get_continuous(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """
     Continuous data provide instantaneous water conditions.
@@ -421,6 +428,10 @@ def get_continuous(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -520,6 +531,7 @@ def get_monitoring_locations(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Location information is basic information about the monitoring location
     including the name, identifier, agency responsible for data collection, and
@@ -738,6 +750,10 @@ def get_monitoring_locations(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -808,6 +824,7 @@ def get_time_series_metadata(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Daily data and continuous measurements are grouped into time series,
     which represent a collection of observations of a single parameter,
@@ -975,6 +992,10 @@ def get_time_series_metadata(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -1080,6 +1101,7 @@ def get_combined_metadata(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Get combined monitoring-location and time-series metadata.
 
@@ -1182,6 +1204,10 @@ def get_combined_metadata(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -1277,6 +1303,7 @@ def get_latest_continuous(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """This endpoint provides the most recent observation for each time series
     of continuous data. Continuous data are collected via automated sensors
@@ -1406,6 +1433,10 @@ def get_latest_continuous(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -1478,6 +1509,7 @@ def get_latest_daily(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Daily data provide one data value to represent water conditions for the
     day.
@@ -1609,6 +1641,10 @@ def get_latest_daily(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -1682,6 +1718,7 @@ def get_field_measurements(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Field measurements are physically measured values collected during a
     visit to the monitoring location. Field measurements consist of measurements
@@ -1804,6 +1841,10 @@ def get_field_measurements(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -1873,6 +1914,7 @@ def get_field_measurements_metadata(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Get field-measurement metadata: one row per (location, parameter) series.
 
@@ -1926,6 +1968,10 @@ def get_field_measurements_metadata(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -1998,6 +2044,7 @@ def get_peaks(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """Get the annual peak streamflow / stage record for a monitoring location.
 
@@ -2056,6 +2103,10 @@ def get_peaks(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------
@@ -2198,6 +2249,68 @@ def get_reference_table(
     )
 
 
+def get_queryables(collection: str) -> tuple[pd.DataFrame, BaseMetadata]:
+    """List the queryable properties of a Water Data API collection.
+
+    Every OGC collection (``daily``, ``continuous``, ``monitoring-locations``,
+    ...) advertises the set of properties that can be filtered on -- exposed as
+    the typed keyword arguments of the matching ``get_*`` function, and usable
+    directly in a CQL2 ``filter``. This returns that set, so the available
+    filters can be discovered programmatically and monitored for upstream
+    additions.
+
+    Parameters
+    ----------
+    collection : string
+        The collection id, e.g. ``"daily"``, ``"continuous"``,
+        ``"monitoring-locations"``, or ``"time-series-metadata"``. See
+        :data:`dataretrieval.waterdata.types.WATERDATA_SERVICES` for the data
+        collections; reference collections (e.g. ``"parameter-codes"``) work
+        too.
+
+    Returns
+    -------
+    df : ``pandas.DataFrame``
+        One row per queryable, sorted by name, with columns ``queryable`` (the
+        property name), ``type``, ``title``, and ``description``.
+    md : :class:`dataretrieval.utils.BaseMetadata`
+        Metadata describing the request (URL, query time, response headers).
+
+    Raises
+    ------
+    DataRetrievalError
+        On an HTTP error response (e.g. an unknown ``collection`` yields a 404),
+        the typed subclass for the status.
+
+    Examples
+    --------
+    .. doctest::
+        :skipif: True  # network
+
+        >>> from dataretrieval import waterdata
+        >>> df, md = waterdata.get_queryables("daily")
+        >>> df.set_index("queryable").loc["state_name", "type"]
+        'string'
+    """
+    # The OGC queryables document is a JSON Schema whose ``properties`` map each
+    # filterable property name to a ``{title, type, description}`` definition.
+    body, response = _check_ogc_requests(endpoint=collection, req_type="queryables")
+    properties: dict[str, Any] = body.get("properties", {})
+    df = pd.DataFrame(
+        [
+            {
+                "queryable": name,
+                "type": prop.get("type"),
+                "title": prop.get("title"),
+                "description": (prop.get("description") or "").strip(),
+            }
+            for name, prop in sorted(properties.items())
+        ],
+        columns=["queryable", "type", "title", "description"],
+    )
+    return df, BaseMetadata(response)
+
+
 def get_codes(code_service: CODE_SERVICES) -> tuple[pd.DataFrame, BaseMetadata]:
     """Return codes from a Samples code service.
 
@@ -2916,6 +3029,7 @@ def get_channel(
     filter: str | None = None,
     filter_lang: FILTER_LANG | None = None,
     convert_type: bool = True,
+    **queryables: Any,
 ) -> tuple[pd.DataFrame, BaseMetadata]:
     """
     Channel measurements taken as part of streamflow field measurements.
@@ -3045,6 +3159,10 @@ def get_channel(
         and the lexicographic-comparison pitfall.
     convert_type : boolean, optional
         If True, converts columns to appropriate types.
+    **queryables : string or iterable of strings, optional
+        Any other queryable property of this collection, passed through as a
+        server-side filter. Call :func:`get_queryables` to see the queryables a
+        collection supports.
 
     Returns
     -------

dataretrieval/waterdata/utils.py

@@ -155,6 +155,23 @@
 }
 
 
+def _flatten_queryables(local_vars: dict[str, Any]) -> dict[str, Any]:
+    """Merge a getter's ``**queryables`` passthrough kwargs -- collected by
+    ``locals()`` under the ``queryables`` key -- up into ``local_vars`` as
+    top-level entries, so an extra server-side filter such as
+    ``state_name="Wisconsin"`` is normalized, mutual-exclusion-checked, and sent
+    exactly like a named param. See
+    :func:`dataretrieval.waterdata.get_queryables` for each collection's
+    filterable properties (the service rejects an unknown one with a 400).
+
+    ``**queryables`` always arrives as a dict (empty when unused) and the key is
+    popped, so this is a no-op on getters without the passthrough and idempotent
+    if called twice.
+    """
+    local_vars.update(local_vars.pop("queryables", {}))
+    return local_vars
+
+
 def _get_args(
     local_vars: dict[str, Any], exclude: set[str] | None = None
 ) -> dict[str, Any]:
@@ -163,8 +180,10 @@ def _get_args(
     Supplies the Water Data API's extended ``no_normalize`` set (numeric
     params such as ``water_year``, ``thresholds``, ``boundingBox``) so they
     keep their element types. See :func:`engine._get_args` for the full
-    normalization contract.
+    normalization contract. Also flattens any ``**queryables`` passthrough
+    (see :func:`_flatten_queryables`).
     """
+    _flatten_queryables(local_vars)
     return _engine_get_args(local_vars, exclude, no_normalize=_NO_NORMALIZE_PARAMS)
 
 
@@ -180,6 +199,12 @@ def _with_state(local_vars: dict[str, Any], *, to: str, into: str) -> dict[str,
     raw values (e.g. non-US FIPS); passing ``state`` together with either
     raises ``ValueError``.
     """
+    # Flatten ``**queryables`` first so a native state param arriving that way
+    # (e.g. ``get_time_series_metadata``'s ``state_code``, which isn't an
+    # explicit parameter) is visible to apply_state's mutual-exclusion guard.
+    # Otherwise ``state`` plus a passthrough ``state_code`` would slip past the
+    # check and silently send both.
+    _flatten_queryables(local_vars)
     return apply_state(
         local_vars, to=to, into=into, reject=("state_code", "state_name")
     )