Add DataFrame API, Expr DSL, functions module, and SDK ergonomics (#151)

* feat: add query_arrow/pandas/polars/pylist helpers on Client

Adds one-shot output ergonomics so callers don't have to reach for
.read_all().to_pandas() on the underlying Flight or ADBC reader. Each
helper accepts an optional `params` list that routes through the existing
ADBC parameterized path, otherwise it goes through Flight as before.

polars is gated behind an optional extra (`spicepy[polars]`) and raises
a clear ImportError if missing.

* style: apply black formatting to _client.py

* feat: add DataFrame API, expression DSL, functions module, and catalog helpers

Adds a credible subset of the datafusion-python surface on top of the
existing Flight + ADBC client.

Tier 1 (Client method additions):
- catalog introspection: catalogs(), schemas(), tables(), describe(),
  get_schema()
- explain(sql, analyze=, verbose=) wrapping EXPLAIN
- streaming/output: query_pydict(), query_batches() iterator, show()
- writers: write_parquet(), write_csv(), write_json() streaming Flight
  output to local files
- DataFrame entry points: table(), sql(), from_arrow(), from_pandas(),
  from_pydict()

Tier 2 (new modules):
- spicepy._sql: identifier and literal escape helpers
- spicepy._expr: Expr DSL with arithmetic/comparison/logical operator
  overloads, alias, cast, is_null, in_, between, asc/desc, CASE WHEN,
  window OVER(); col(), lit(), case() public builders
- spicepy.functions: aggregates (sum/avg/min/max/count/count_distinct/
  stddev/variance/median/...), math (abs/round/ceil/floor/sqrt/power/
  ln/log/exp), strings (lower/upper/length/trim/concat/substr/replace/
  regexp_match/starts_with/ends_with), date/time (now/current_date/
  date_trunc/date_part/extract), null/control flow (coalesce/nullif/
  ifnull/case), window-only (row_number/rank/dense_rank/percent_rank/
  cume_dist/lag/lead/first_value/last_value/nth_value)
- spicepy._dataframe.SpiceDataFrame: lazy SQL-compiling builder with
  select/with_column(s)/drop/rename/cast, filter/where/limit/head/
  offset, sort/order_by/distinct, union/intersect/except_,
  join (inner/left/right/full/semi/anti/cross) with key list or Expr,
  group_by().aggregate(), aggregate() (global), schema/explain,
  collect/to_arrow/to_pandas/to_polars/to_pylist/to_pydict/count/show
- inline VALUES path for small client-side data via from_arrow/
  from_pandas/from_pydict

239 new tests (test_sql, test_expr, test_functions, test_dataframe,
extensions to test_client).

* chore(bandit): skip B608 — escaped SQL composition is the SDK's job

* docs(expr): explain why subclasses must not override __eq__

Adds a note to Expr's docstring documenting that the comparison operators
build SQL expression trees (DSL pattern, same as SQLAlchemy, pandas,
polars, Ibis, datafusion-python) and that subclass __eq__ overrides
would silently break filtering/joins. Addresses a wave of code-quality
bot reviews that misapply a value-semantics __eq__ rule to a DSL.

* chore(deps): fold dependency updates into feature PR

* feat: add GITHUB_TOKEN environment variable for Spice installation in WSL on Windows

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: lukekim

.github/workflows/test.yml

@@ -97,7 +97,7 @@ jobs:
       # guest).
       - name: Set up WSL (Windows)
         if: matrix.os == 'windows-latest'
-        uses: Vampire/setup-wsl@887f39deb6c0976365e546926fe66f41b77d65ff # v6.1.0
+        uses: Vampire/setup-wsl@d1da7f2c0322a5ee4f24975344f67fc0f5baf364 # v7.0.0
         with:
           distribution: Ubuntu-24.04
           additional-packages: |
@@ -107,6 +107,8 @@ jobs:
       - name: install Spice in WSL (Windows)
         if: matrix.os == 'windows-latest'
         shell: wsl-bash {0}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
         run: |
           curl https://install.spiceai.org | /bin/bash
           $HOME/.spice/bin/spice install
@@ -220,7 +222,7 @@ jobs:
           pytest --cov=spicepy --cov-report=xml --cov-report=term-missing --ignore=tests/test_main.py tests/
       - name: Upload coverage to Codecov
         if: matrix.python-version == '3.12'
-        uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v4.5.0
+        uses: codecov/codecov-action@57e3a136b779b570ffcdbf80b3bdc90e7fab3de2 # v6.0.0
         with:
           files: ./coverage.xml
           fail_ci_if_error: false

pyproject.toml

@@ -10,9 +10,9 @@ readme = "README.md"
 license = {text = "Apache-2.0"}
 requires-python = ">=3.11"
 dependencies = [
-    "pyarrow>=23.0.1",
+    "pyarrow>=24.0.0",
     "pandas>=3.0.2",
-    "certifi>=2026.2.25",
+    "certifi>=2026.4.22",
     "requests>=2.33.1",
 ]
 authors = [
@@ -41,26 +41,30 @@ Issues = "https://github.com/spiceai/spicepy/issues"
 test = [
     "pylint>=4.0.5",
     "flake8>=7.3.0",
-    "ruff>=0.15.10",
-    "mypy>=1.20.0",
+    "ruff>=0.15.12",
+    "mypy>=1.20.2",
     "pytest>=9.0.3",
     "pytest-cov>=7.1.0",
     "pytest-xdist>=3.8.0",
     "pytest-timeout>=2.4.0",
     "pytest_httpserver==1.1.5",
-    "types-requests>=2.33.0",
-    "pandas-stubs>=3.0.0",
+    "types-requests>=2.33.0.20260408",
+    "pandas-stubs>=3.0.0.260204",
     "black>=26.3.1",
     "bandit>=1.9.4",
     "pandas>=3.0.2",
-    "pyarrow>=23.0.1",
+    "pyarrow>=24.0.0",
     "adbc-driver-flightsql>=1.11.0",
     "adbc-driver-manager>=1.11.0",
+    "polars>=1.0.0",
 ]
 params = [
     "adbc-driver-flightsql>=1.11.0",
     "adbc-driver-manager>=1.11.0",
 ]
+polars = [
+    "polars>=1.0.0",
+]
 
 # ============== Tool Configuration ==============
 
@@ -93,6 +97,8 @@ module = [
     "adbc_driver_manager.*",
     "certifi",
     "pandas",
+    "polars",
+    "polars.*",
     "_pytest.*",
     "pytest.*",
 ]
@@ -185,6 +191,14 @@ ignore = [
     "PLR2004", # Magic values ok in tests
     "ARG",    # Unused arguments ok in tests (fixtures)
     "T20",    # Print statements ok in tests
+    "N812",   # `functions as F` is the DataFrame convention
+    "S608",   # asserting SQL fragments is the point of these tests
+]
+"spicepy/_client.py" = [
+    "S608",   # SQL is constructed from escaped identifiers/literals; lint can't tell
+]
+"spicepy/_dataframe.py" = [
+    "S608",   # The DataFrame layer's entire job is composing SQL; identifiers and literals are escaped
 ]
 
 [tool.ruff.lint.isort]
@@ -246,9 +260,12 @@ directory = "htmlcov"
 
 [tool.bandit]
 exclude_dirs = ["tests", ".venv"]
-skips = ["B101"]  # assert_used
+skips = [
+    "B101",  # assert_used
+    "B608",  # hardcoded_sql_expressions: composing SQL with escaped identifiers/literals is this package's job
+]
 
 [dependency-groups]
 dev = [
-    "build>=1.4.3",
+    "build>=1.4.4",
 ]

requirements.txt

@@ -1,4 +1,4 @@
-pyarrow>=23.0.1
+pyarrow>=24.0.0
 pandas>=3.0.2
-certifi>=2026.2.25
+certifi>=2026.4.22
 requests>=2.33.1

spicepy/__init__.py

@@ -5,5 +5,19 @@
 """
 
 # flake8: noqa
+from . import functions
 from ._client import Client
+from ._dataframe import SpiceDataFrame
+from ._expr import Expr, case, col, lit
 from ._http import RefreshOpts
+
+__all__ = [
+    "Client",
+    "Expr",
+    "RefreshOpts",
+    "SpiceDataFrame",
+    "case",
+    "col",
+    "functions",
+    "lit",
+]