Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,38 @@ jobs:
use_oidc: ${{ !github.event.pull_request.head.repo.fork }}
fail_ci_if_error: false

# `test` uses whatever qsv was released last; this keeps the oldest qsv we accept working.
test-min-qsv:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
with:
python-version: "3.11"
- run: uv sync
- name: Download the qsv release named by MIN_QSV_VERSION
env:
GH_TOKEN: ${{ github.token }}
run: |
v=$(uv run python -c "import qsv_client; print(qsv_client.MIN_QSV_VERSION)")
mkdir -p "$RUNNER_TEMP/qsv"
gh release download "$v" --repo dathere/qsv --pattern "qsv-${v}-x86_64-unknown-linux-musl.zip" --dir "$RUNNER_TEMP/qsv"
unzip -o -q "$RUNNER_TEMP"/qsv/*.zip -d "$RUNNER_TEMP/qsv"
chmod +x "$RUNNER_TEMP"/qsv/qsv*
for b in qsv qsvlite qsvdp; do
reported=$("$RUNNER_TEMP/qsv/$b" --version)
case "$reported" in
"$b $v-"*) ;;
*) echo "expected $b $v, got: $reported"; exit 1 ;;
esac
done
- name: Integration tests (qsv)
run: QSV_BIN="$RUNNER_TEMP/qsv/qsv" uv run pytest -v tests/test_integration.py tests/test_integration_daemon.py
- name: Integration tests (qsvlite)
run: QSV_BIN="$RUNNER_TEMP/qsv/qsvlite" uv run pytest -q tests/test_integration.py
- name: Integration tests (qsvdp)
run: QSV_BIN="$RUNNER_TEMP/qsv/qsvdp" uv run pytest -q tests/test_integration.py

# Separate from `test` because Windows steps run in PowerShell and use the MSVC build.
test-windows:
strategy:
Expand Down
17 changes: 17 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,23 @@ All notable changes to this project are documented here. The format follows

## [Unreleased]

qsv 27.0.1 or later is now required, for everything. It is the first released qsv whose Data
Schematics connect to `qsv-clientd`, so there is no longer a separate, lower minimum for the
rest of the library.

### Changed

- **Behavior change:** qsv 27.0.1 or later is now required (`MIN_QSV_VERSION`, was 25.0.0).
An older qsv raises `QsvVersionError` before the first command, as a qsv older than 25.0.0
did before. `qsv-clientd serve` exits with that error instead of starting with a warning
that schematics can't connect, and the tray shows it as an error instead of a

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

When attaching to an already-running daemon, the tray does not verify the daemon's qsv version. An older daemon can therefore continue using an unsupported qsv without the tray reporting an error. Check the daemon's version during attachment and surface unsupported versions explicitly.

TIP React 👍 if this review is right, or reply with what's wrong and how it could be better to help improve the reviewer.

notification. Development builds of qsv that report 27.0.0 (the release that was
withdrawn) are now refused too; any qsv built since has a later version.
- `qsv-clientd` no longer gives `qsv get` a private cache of its own: every qsv it now accepts
keeps that cache readable by its owner only.
- CI runs the integration tests against qsv 27.0.1 (`qsv`, `qsvlite` and `qsvdp`) as well as
the latest release, so the minimum stays tested.

### Security

- On Windows, a `binary` that is a `.bat` or `.cmd` file (a sandbox wrapper, say) runs
Expand Down
10 changes: 6 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ uv run pytest --cov # with branch coverage (config in pypro
```

CI runs lint, format check, and mypy, plus tests on ubuntu/macos x Python 3.11–3.14 against
qsv, qsvlite, and qsvdp (Linux only).
qsv, qsvlite, and qsvdp (Linux only), all from the latest qsv release. The `test-min-qsv` job
runs the integration tests against the release `MIN_QSV_VERSION` names (Linux, all three).
On Windows (Python 3.11–3.14) it runs the full suite against qsv, plus the integration tests
against qsvlite. Windows teardown uses Job Objects via ctypes (`_ProcessTree` in `_process.py`);
type-check it with `uv run mypy --platform win32` (the lint job runs both).
Expand Down Expand Up @@ -48,9 +49,10 @@ below the base commit's, and new or changed lines need 100% coverage.

## Invariants

- qsv 25.0.0 is the minimum (`MIN_QSV_VERSION`): the first `--capabilities` qsv with JSON
errors. `--capabilities` is probed before the first run; a binary that rejects the flag
raises `QsvVersionError`. `min_version=` can raise the floor, not lower it.
- qsv 27.0.1 is the minimum (`MIN_QSV_VERSION`): the first released qsv whose Data Schematics
connect to `qsv-clientd` (25.0.0 was the first with `--capabilities` and JSON errors, so
older binaries reject the flag). `--capabilities` is probed before the first run; a binary
that rejects the flag raises `QsvVersionError`. `min_version=` can raise the floor, not lower it.
- Successful probes are cached process-wide in `client._caps_cache`, keyed by
(path, size, mtime_ns) and shared by `Qsv` and `AsyncQsv`. A per-key lock (per event loop
for async, held weakly so loops can be collected) dedups concurrent probes; sync and async
Expand Down
9 changes: 4 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,7 @@ subprocess pitfalls:
Schematic from `qsv viz smart` work with its live data, with an optional system tray to
run and control it. See [The qsv-clientd daemon](#the-qsv-clientd-daemon).

qsv itself is not bundled, and **qsv 25.0.0 or later** is required; live Data Schematics
need **qsv 27.0.1 or later** (with an older qsv, `qsv-clientd serve` and the tray say so).
qsv itself is not bundled, and **qsv 27.0.1 or later** is required.
Install it from the [qsv releases](https://github.com/dathere/qsv/releases) or a
[package manager](https://github.com/dathere/qsv#installation-options).

Expand Down Expand Up @@ -116,7 +115,7 @@ cancellation propagates, so expect up to a few `kill_grace` periods.
### Capabilities and version checks

```python
qsv = Qsv(min_version="25.1.0") # QsvVersionError on first use if older
qsv = Qsv(min_version="28.0.0") # QsvVersionError on first use if older
caps = qsv.capabilities
caps.binary, caps.version, caps.features, caps.commands
caps.has_feature("polars"), caps.has_command("describegpt")
Expand All @@ -129,7 +128,7 @@ client per call costs nothing extra after the first.

`qsv-clientd` (installed with the package) is a per-user daemon that runs qsv for local
processes, through `DaemonClient` / `AsyncDaemonClient` (the same `run()` as `Qsv`), and for
open **Data Schematics** from `qsv viz smart` (qsv 27.0.1 or later). Once you approve a
open **Data Schematics** from `qsv viz smart`. Once you approve a
schematic with a pairing code, it can drill into the rows behind a chart, refresh its
statistics, re-render itself, run read-only queries and fetch the latest data. Large results
are paged from the daemon, so every row is reachable.
Expand Down Expand Up @@ -179,7 +178,7 @@ See [docs/DAEMON.md](docs/DAEMON.md) for the security model and protocol.
| `QsvInferenceError` | `inference` (an LLM call failed) | 1 |
| `QsvWriteError` | `write` (e.g. disk full) | 1 |
| `QsvTimeout` | `timeout` (killed by this client) | 124 |
| `QsvVersionError` | `version` (binary older than 25.0.0, or than `min_version`) | |
| `QsvVersionError` | `version` (binary older than 27.0.1, or than `min_version`) | |
| `QsvError` | anything else; base class of all of the above | |

`QsvDaemonError` (the daemon refused a request; `status` and `code` say why) and its subclass
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ build-backend = "hatchling.build"
[project]
name = "qsv-client"
version = "0.6.1"
description = "Run the qsv CSV toolkit (25.0.0+) from Python: timeouts that kill the whole process tree, typed errors, capability detection, and a local daemon (with an optional system tray) for live Data Schematics (qsv 27.0.1+)."
description = "Run the qsv CSV toolkit (27.0.1+) from Python: timeouts that kill the whole process tree, typed errors, capability detection, and a local daemon (with an optional system tray) for live Data Schematics."
readme = "README.md"
license = "AGPL-3.0-or-later"
license-files = ["LICENSE"]
Expand Down
2 changes: 1 addition & 1 deletion src/qsv_client/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""Run the qsv CSV toolkit from Python.

Timeouts kill the whole process tree, failures raise typed exceptions, and capability
detection replaces parsing ``qsv --version``. Requires qsv 25.0.0 or later.
detection replaces parsing ``qsv --version``. Requires qsv 27.0.1 or later.
See https://github.com/dathere/qsv-client.
"""

Expand Down
4 changes: 2 additions & 2 deletions src/qsv_client/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
DEFAULT_BINARIES = ("qsv", "qsvmcp", "qsvdp", "qsvlite")

#: oldest qsv this client supports: the first with ``--capabilities`` and ``QSV_ERROR_FORMAT``
MIN_QSV_VERSION = "25.0.0"
MIN_QSV_VERSION = "27.0.1"

Arg = str | os.PathLike[str] | int | float

Expand Down Expand Up @@ -148,7 +148,7 @@ def __init__(
``QSV_LLM_APIKEY`` / ``QSV_LLM_BASE_URL`` / ``QSV_LLM_MODEL`` environment
variables, so the key never appears in the process table.
min_version: raise :class:`QsvVersionError` on first use if the binary is older.
The binary must be at least :data:`MIN_QSV_VERSION` (25.0.0) regardless;
The binary must be at least :data:`MIN_QSV_VERSION` (27.0.1) regardless;
this can only raise that floor. A string that is not a version
(``"major.minor.patch"``) raises ``ValueError`` here.
kill_grace: seconds to wait between the polite stop (SIGTERM, or
Expand Down
29 changes: 0 additions & 29 deletions src/qsv_client/daemon/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,40 +10,11 @@
from dataclasses import dataclass, field, fields
from pathlib import Path

from ..capabilities import parse_version

DEFAULT_PORT = 14470
"""Loopback port for browsers (Data Schematics). qsv pro's API uses 14462."""

API_VERSION = 1

# 27.0.0 was withdrawn before release, but builds of qsv master report it and have these
# features, so the version checks use it; what users are told names 27.0.1
LIVE_SCHEMATICS_QSV = "27.0.0"
"""The first qsv whose Data Schematics connect to qsv-clientd."""

LIVE_SCHEMATICS_RELEASE = "27.0.1"
"""The first released qsv whose Data Schematics connect to qsv-clientd."""

PRIVATE_GET_CACHE_QSV = "27.0.0"
"""The first qsv that keeps its `get` cache readable by its owner only (on Unix)."""


def schematics_notice(qsv_version: object) -> str | None:
"""Why schematics won't go live with this qsv, or None when they will (or it can't tell)."""
if not isinstance(qsv_version, str):
return None
try:
older = parse_version(qsv_version) < parse_version(LIVE_SCHEMATICS_QSV)
except ValueError:
return None
if not older:
return None
return (
f"qsv {qsv_version} writes Data Schematics that can't connect to qsv-clientd. "
f"Live schematics need qsv {LIVE_SCHEMATICS_RELEASE} or later."
)


def _home_override() -> Path | None:
# one directory for everything; used by tests and for running several daemons side by side
Expand Down
22 changes: 3 additions & 19 deletions src/qsv_client/daemon/fetch.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,8 @@
from typing import TYPE_CHECKING, Any

from .._process import _finish_despite_cancel
from ..capabilities import parse_version
from ._http import HttpError
from .config import PRIVATE_GET_CACHE_QSV, ensure_private_dir
from .config import ensure_private_dir
from .registry import Dataset

if TYPE_CHECKING:
Expand Down Expand Up @@ -133,20 +132,15 @@ async def download(
argv = ["get", url, "--name", name]
if scheme == "ckan":
argv += ["--ckan-api", _ckan_api_for(target)]
# both runs must see the same cache, or cache-fetch can't find the entry
env = _get_cache_env(daemon, caps.version_tuple)
# qsv writes --output with the umask's mode (often 0644), and staging_dir may be
# one others can read, so it writes into a directory only we can enter
private = Path(tempfile.mkdtemp(prefix=".qsv-clientd-", dir=staging_dir))
try:
output = private / incoming.name
try:
await daemon._browser_run(ds, argv, timeout=timeout, env=env)
await daemon._browser_run(ds, argv, timeout=timeout)
await daemon._browser_run(
ds,
["get", "cache-fetch", name, "--output", str(output)],
timeout=timeout,
env=env,
ds, ["get", "cache-fetch", name, "--output", str(output)], timeout=timeout
)
except HttpError as exc:
if exc.code == "qsv_timeout":
Expand Down Expand Up @@ -188,16 +182,6 @@ async def download(
return incoming, target.via


def _get_cache_env(daemon: Daemon, qsv_version: tuple[int, int, int]) -> dict[str, str] | None:
"""Before 27.0.0, qsv's get cache follows the umask, and downloads may be private, so give
it one only we can read. A QSV_CACHE_DIR the user set, and qsv 27.0.0+, are left alone."""
if qsv_version >= parse_version(PRIVATE_GET_CACHE_QSV):
return None
if daemon.config.inherit_env and "QSV_CACHE_DIR" in os.environ:
return None
return {"QSV_CACHE_DIR": str(ensure_private_dir(daemon.home / "qsv-cache"))}


async def _fetch_locked(daemon: Daemon, ds: Dataset, source: str, force: bool) -> dict[str, Any]:
cache = ensure_private_dir(daemon.home / "cache" / ds.id)
what = "the source URL" if ds.source_url else "the PID"
Expand Down
4 changes: 0 additions & 4 deletions src/qsv_client/daemon/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,6 @@
ensure_private_dir,
local_token_path,
runtime_dir,
schematics_notice,
socket_path_error,
state_dir,
write_private_file,
Expand Down Expand Up @@ -132,9 +131,6 @@ def socket_path(self) -> Path:
async def start(self) -> None:
caps = await self._qsv.capabilities()
log.info("qsv-clientd %s using %s %s", __version__, caps.binary, caps.version)
notice = schematics_notice(caps.version)
if notice:
log.warning("%s", notice)
ensure_private_dir(self.home)
# bind everything before publishing the token: a second instance that fails to bind
# must not replace the running daemon's credentials
Expand Down
2 changes: 1 addition & 1 deletion src/qsv_client/errors.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ class QsvNotFound(FileNotFoundError):


class QsvVersionError(QsvError):
"""The qsv binary is older than 25.0.0, or than the client's ``min_version``.
"""The qsv binary is older than 27.0.1, or than the client's ``min_version``.

Raised before the command runs. ``exit_code``, ``args_run`` and ``stderr`` describe the
``--capabilities`` check, not the command.
Expand Down
15 changes: 0 additions & 15 deletions src/qsv_client/tray/controller.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@
config_path,
config_template,
log_path,
schematics_notice,
)
from ..daemon.host import DaemonHost
from ..daemon.server import Daemon
Expand Down Expand Up @@ -81,7 +80,6 @@ def __init__(
self.notify: Notifier = nothing
"""Shows a desktop notification, on the worker thread."""
self._announced: set[str] = set()
self._qsv_noted: object = None
self._state = TrayState()
self._lock = threading.Lock()
self._emit_lock = threading.Lock()
Expand Down Expand Up @@ -225,17 +223,6 @@ def _announce(self, pairings: tuple[Pending, ...]) -> None:
)
self._announced = codes

def _note_qsv(self, status: Mapping[str, Any]) -> None:
"""Say once per qsv version when its schematics can't go live."""
qsv = status.get("qsv")
version = qsv.get("version") if isinstance(qsv, Mapping) else None
if version == self._qsv_noted:
return
self._qsv_noted = version
notice = schematics_notice(version)
if notice:
self.notify("Data Schematics won't go live", notice)

def _attached(self, client: DaemonClient) -> None:
status = self._status(client)
pairings = self._pairings(client, status)
Expand All @@ -251,7 +238,6 @@ def _attached(self, client: DaemonClient) -> None:
error=None,
)
self._announce(pairings)
self._note_qsv(status)
log.info("attached to qsv-clientd (pid %s)", status.get("pid"))

def _start(self) -> None:
Expand All @@ -274,7 +260,6 @@ def _start(self) -> None:
mode=Mode.HOSTED, status=status, pairings=pairings, datasets=datasets, service=None
)
self._announce(pairings)
self._note_qsv(status)
log.info("hosting qsv-clientd")

def _stop_host(self) -> None:
Expand Down
4 changes: 2 additions & 2 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ def fake_qsv(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Callable[[str],
"""Write an executable stand-in for qsv running the given Python body; return its path.

The client probes ``--capabilities`` before its first run, so unless ``body`` mentions
``--capabilities`` (to answer it itself), the stand-in answers as qsv 25.0.0.
``--capabilities`` (to answer it itself), the stand-in answers as qsv 27.0.1.

POSIX: a script with a shebang. Windows: the script plus a ``fake-qsv.cmd`` wrapper that
runs it (the extra ``cmd.exe`` is inside the run's Job Object, so it is killed with it).
Expand All @@ -58,7 +58,7 @@ def fake_qsv(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Callable[[str],
prelude = 'import sys; sys.stdout.reconfigure(newline="\\n")\n'
capabilities = (
'if sys.argv[1:] == ["--capabilities"]:\n'
' print(\'{"binary": "qsv", "version": "25.0.0", "error_formats": ["text", "json"]}\')\n'
' print(\'{"binary": "qsv", "version": "27.0.1", "error_formats": ["text", "json"]}\')\n'
" sys.exit(0)\n"
)

Expand Down
Loading
Loading