Fix six defects found by the final red-team, four of them mine

**Single-venv jobs could never return a result (critical, introduced by me).**
Result signing was added to the two-venv branch only, while the submitter
recorded a signing key unconditionally -- so any job that fell back to the
single-venv path (use_two_venv=False, or any two-venv setup failure or
timeout) produced a result the caller then refused as unsigned. A degraded but
working path became a hard failure. All four schedulers now sign on both
branches, via one shared `result_signing_lines()` so a branch cannot forget
again.

Testing that on tensor01 uncovered three more, each hidden behind the last:

* **The SSH fallback never built its venv.** Both of its fallback branches set
  `venv_info = None` and stopped; the SLURM path has always called
  `setup_remote_environment`. The generated script therefore activated a
  virtualenv nobody had created: "venv/bin/activate: No such file or
  directory".
* **`python -m venv` assumed a `python` that does not exist.** Python 3
  installs ship `python3`; `python` is only present where somebody added a
  compatibility symlink. `resolve_remote_python` now probes.
* **The dill install was best-effort.** `|| echo 'Package installation failed,
  continuing...'` swallowed the failure of the one package the job script
  cannot work without, and the job died twenty lines into a remote traceback
  with "'NoneType' object is not callable", naming neither.

Which finally exposed the real limit, which is not papered over: **the
single-venv path cannot bridge Python versions at all.** dill embeds CPython
bytecode, so a 3.12 payload on tensor01's `python3 (3.6.8)` gives
"code() takes at most 15 arguments (20 given)". It now fails at submit time
with something actionable:

    No python3.12 on the remote host, and dill payloads cannot cross Python
    minor versions. Found: python3 (3.6.8). Either install python3.12 there,
    set python_executable to a matching interpreter, or leave use_two_venv
    enabled so clustrix can build a conda environment at the right version.

**Silent chmod failure -> key theft -> code execution on the submitting
machine.** `execute_remote_command` never checked exit status, and job
directories were `job_<unix_seconds>` -- fully predictable. On a
world-writable remote_work_dir an attacker could pre-create the directory;
`mkdir -p` succeeds on it, the unchecked `chmod 700` fails unnoticed, the
signing key lands somewhere they can read, and they forge both result.pkl and
its HMAC. Directories are now created exclusively with `mkdir -m 700` and the
status checked, and their names carry four random bytes -- which also fixes
two jobs submitted in the same second overwriting each other's key.

**`1.5GB` produced `--mem=1.5G`**, which SLURM and PBS reject. Fractional
sizes round up (down would get the job killed).

Shell interpolations of config-derived paths are now `shlex.quote`d.

The agent also attacked and could not break: `make_portable_function` (its
source is a module constant), the HMAC's coverage of the downloaded bytes,
`compare_digest`'s operands, cleanup ordering, and the dropdown reentrancy
guard.

All three backends re-verified afterwards:

    slurm  PASSED  t08.hpcc.dartmouth.edu   python 3.12.13
    gpu    PASSED  tensor01.dartmouth.edu   python 3.12.13
    hf     PASSED  j-contextlab-6a8401ce...  python 3.12.14

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012gTBDPK16HUZ3kHQ2QyjuU

Author: jeremymanning

clustrix/executor_connections.py

@@ -217,14 +217,29 @@ def _configure_kubectl_for_provisioned_cluster(self, cluster_info: Dict[str, Any
             logger.error(f"Failed to configure kubectl for provisioned cluster: {e}")
             raise
 
-    def execute_remote_command(self, command: str) -> tuple:
-        """Execute command on remote cluster."""
+    def execute_remote_command(self, command: str, check: bool = False) -> tuple:
+        """Execute command on remote cluster.
+
+        ``check`` raises when the command exits non-zero. It is off by default
+        because most callers here inspect the output themselves and tolerate
+        failure, but anything whose failure would be *unsafe* rather than
+        merely unhelpful must opt in -- a `chmod 700` that quietly does nothing
+        leaves a secret readable.
+        """
         if self.ssh_client is None:
             raise RuntimeError(
                 "SSH client not connected. Call setup_ssh_connection() first."
             )
         stdin, stdout, stderr = self.ssh_client.exec_command(command)
-        return stdout.read().decode(), stderr.read().decode()
+        out = stdout.read().decode()
+        err = stderr.read().decode()
+        if check:
+            status = stdout.channel.recv_exit_status()
+            if status != 0:
+                raise RuntimeError(
+                    f"Remote command failed (exit {status}): {command}\n{err.strip()}"
+                )
+        return out, err
 
     def resolve_remote_path(self, path: str) -> str:
         """Expand a leading ``~/`` against the remote account's home directory.

clustrix/executor_core.py

@@ -5,6 +5,7 @@
 """
 
 import hashlib
+import shlex
 import hmac
 import time
 import tempfile
@@ -190,7 +191,7 @@ def _verify_result_signature(
 
         try:
             stdout, _ = self.connection_manager.execute_remote_command(
-                f"cat {remote_dir}/result.pkl.hmac 2>/dev/null"
+                f"cat {shlex.quote(f'{remote_dir}/result.pkl.hmac')} 2>/dev/null"
             )
         except Exception as e:  # pragma: no cover - defensive
             raise RuntimeError(

clustrix/executor_schedulers.py

@@ -6,6 +6,7 @@
 
 import os
 import secrets
+import shlex
 import time
 import tempfile
 import pickle
@@ -42,8 +43,18 @@ def _prepare_job_dir(self, remote_job_dir: str) -> str:
         read another user's command line out of `ps` -- which would hand the
         secret to exactly the people the 0700 directory is meant to exclude.
         """
+        # `mkdir -p` succeeds on a directory that already exists and is owned
+        # by somebody else, and an unchecked `chmod` then fails silently. Job
+        # directory names were fully predictable, so on a world-writable
+        # remote_work_dir an attacker could pre-create the directory, receive
+        # the signing key into it, and forge a result -- turning the defence
+        # into a code-execution path on the *submitting* machine. Create it
+        # exclusively, and refuse to continue if that fails.
         self.connection_manager.execute_remote_command(
-            f"mkdir -p {remote_job_dir} && chmod 700 {remote_job_dir}"
+            f"mkdir -p {shlex.quote(os.path.dirname(remote_job_dir))}", check=True
+        )
+        self.connection_manager.execute_remote_command(
+            f"mkdir -m 700 {shlex.quote(remote_job_dir)}", check=True
         )
         key = secrets.token_hex(32)
         self.connection_manager.create_remote_file(
@@ -73,7 +84,7 @@ def submit_slurm_job(
         work_dir = self.connection_manager.resolve_remote_path(
             self.config.remote_work_dir
         )
-        remote_job_dir = f"{work_dir}/job_{int(time.time())}"
+        remote_job_dir = f"{work_dir}/job_{int(time.time())}_{secrets.token_hex(4)}"
         result_key = self._prepare_job_dir(remote_job_dir)
 
         # Upload function data
@@ -198,7 +209,7 @@ def submit_pbs_job(
         work_dir = self.connection_manager.resolve_remote_path(
             self.config.remote_work_dir
         )
-        remote_job_dir = f"{work_dir}/job_{int(time.time())}"
+        remote_job_dir = f"{work_dir}/job_{int(time.time())}_{secrets.token_hex(4)}"
         result_key = self._prepare_job_dir(remote_job_dir)
 
         # Upload function data
@@ -245,7 +256,7 @@ def submit_sge_job(
         work_dir = self.connection_manager.resolve_remote_path(
             self.config.remote_work_dir
         )
-        remote_job_dir = f"{work_dir}/job_{int(time.time())}"
+        remote_job_dir = f"{work_dir}/job_{int(time.time())}_{secrets.token_hex(4)}"
         result_key = self._prepare_job_dir(remote_job_dir)
 
         # Upload function data
@@ -302,7 +313,7 @@ def submit_ssh_job(
         work_dir = self.connection_manager.resolve_remote_path(
             self.config.remote_work_dir
         )
-        remote_job_dir = f"{work_dir}/job_{int(time.time())}"
+        remote_job_dir = f"{work_dir}/job_{int(time.time())}_{secrets.token_hex(4)}"
         result_key = self._prepare_job_dir(remote_job_dir)
 
         # Upload function data
@@ -368,10 +379,27 @@ def setup_venv():
 
             except Exception as e:
                 logger.warning(f"Failed to setup two-venv environment: {e}")
-                # Fall back to original approach
+                # Fall back to the single-venv approach -- which means actually
+                # building that venv. Both fallback branches used to set
+                # venv_info = None and stop there, so the generated script
+                # activated a virtualenv nobody had created and every job died
+                # with "venv/bin/activate: No such file or directory". The
+                # SLURM path has always called this; the SSH path never did.
+                setup_remote_environment(
+                    self.connection_manager.ssh_client,
+                    remote_job_dir,
+                    func_data["requirements"],
+                    self.config,
+                )
                 updated_config.venv_info = None
         else:
             logger.info("Two-venv setup disabled, using basic environment setup")
+            setup_remote_environment(
+                self.connection_manager.ssh_client,
+                remote_job_dir,
+                func_data["requirements"],
+                self.config,
+            )
             updated_config.venv_info = None
 
         # Create execution script