don't overexpose persistence dir in config, minor fixes

Author: zxqfd555

.gitignore

@@ -223,9 +223,12 @@ benchmarks/*/results/
 benchmarks/*/compat/*.so
 
 # Runtime working directory of `serviette demo` (regenerable: corpus copy,
-# generated config, DuckDB store, persistence)
+# generated config, DuckDB store, workdir)
 serviette-demo/
 
+# Default `workdir_path` (persistence state, fingerprint)
+serviette-workdir/
+
 # Internal planning document — kept local, never committed (purged from
 # history on 2026-07-09).
 docs/ROADMAP.md

benchmarks/frames/configs/base-e5small-full.yaml

@@ -30,10 +30,10 @@ splitter:
 indexer:
   workers: 8
   spawn_first_port: 12000
+workdir_path: ${FRAMES_DATA}/workdir-e5small-full
 persistence:
   enabled: true
   backend: filesystem
-  path: ${FRAMES_DATA}/persist-e5small-full
 server:
   port: 8987
   serve_frontend: false

benchmarks/frames/configs/base-e5small.yaml

@@ -30,10 +30,10 @@ splitter:
 # thread/port settings of the full-dump run).
 indexer:
   workers: 8
+workdir_path: ${FRAMES_DATA}/workdir-e5small
 persistence:
   enabled: true
   backend: filesystem
-  path: ${FRAMES_DATA}/persist-e5small
 server:
   port: 8987
   serve_frontend: false

benchmarks/realtime-data-indexing/config.yaml

@@ -45,10 +45,10 @@ indexer:
 # Product default: persistence on. Lives inside the container (fresh each
 # run, since run_bench recreates containers), so runs stay cold-start honest
 # while exercising the same code path users get — including the disk-backed
-# parse cache under <path>/runtime_calls.
+# parse cache under <workdir>/persistence/runtime_calls.
+workdir_path: /tmp/serviette-workdir
 persistence:
   enabled: true
-  path: /tmp/persistence
 
 server:
   host: 0.0.0.0

docs/README.md

@@ -200,9 +200,9 @@ string, which would only surface later as a provider 401).
 | `splitter.chunk_overlap` | int | `50` | indexer | Overlap (used by `recursive`). |
 | `indexer.fetch_retries` | int | `3` | indexer | Retries (exponential backoff from 1s) of a document's byte fetch before it is given up on: indexed as empty and ERROR-logged. Remote sources fail transiently under bulk backfills. |
 | `indexer.workers` | int | `1` | indexer | Worker **processes** (sharded via `pathway spawn`). Raise for large backfills — each worker carries its own embedding stack (~1 GB with local embeddings); the benchmarks run with `8`. |
+| `workdir_path` | str | `./serviette-workdir` | indexer | serviette's working directory — everything it keeps between runs: the configuration fingerprint at its top level, the Pathway persistence state (and the parse cache, `runtime_calls/`) under `persistence/`. Relative to the process's working directory. Silent default — the wizard does not ask. |
 | `persistence.enabled` | bool | `true` | indexer | See [Persistence](#5-persistence). |
 | `persistence.backend` | `filesystem` | `filesystem` | indexer | Persistence backend. |
-| `persistence.path` | str | `./persistence` | indexer | serviette's data directory: the configuration fingerprint at its top level, the Pathway persistence state (and the parse cache, `runtime_calls/`) under `PStorage/`. Silent default — the wizard does not ask. |
 | `server.host` / `server.port` | str / int | `127.0.0.1` / `8989` | server | Bind address. Loopback by default; set `0.0.0.0` explicitly to listen on all interfaces (containers, remote access) — see [Security](#security--exposing-the-server). |
 | `server.serve_frontend` | bool | `true` | server | Serve the chat UI on `/` from the same port (API stays under `/api/v1`). |
 | `server.cors_origins` | list[str] | `[]` (disabled) | server | Opt-in CORS allowlist for third-party browser frontends calling the API directly from another origin. serviette's own UIs never need it. Prefer exact origins over `*`. |
@@ -367,14 +367,26 @@ the last run, so:
 - **documents removed while the indexer was down are correctly retracted** from
   the vector DB on the next run.
 
-The engine's state lives under `<persistence.path>/PStorage/` (the top level of
-`persistence.path` is reserved for serviette's own files, such as the
-configuration fingerprint); the same `PStorage/` also hosts the **parse cache**
-(`runtime_calls/`, via `pw.udfs.DefaultCache` — diskcache, LRU-bounded by
+The engine's state lives under `<workdir_path>/persistence/` (the top level of
+the working directory is reserved for serviette's own files, such as the
+configuration fingerprint — the engine logs an ERROR for every foreign entry
+in the directory it is given, so the two never share a folder); the same
+`persistence/` also hosts the **parse cache** (`runtime_calls/`, via
+`pw.udfs.DefaultCache` — diskcache, LRU-bounded by
 `indexer.parse_cache_size_gb`, default 8): extracted document text stays warm
 across restarts, so unchanged documents are neither re-downloaded nor
 re-parsed. Disabling persistence also disables the parse cache.
 
+The **configuration fingerprint** (`<workdir_path>/serviette-fingerprint.json`)
+records the chunking-relevant settings the working directory was indexed with
+(splitter, parser routing, embedder identity, library versions, chunk-id
+scheme). A change is reported as a diff on startup and the indexer refuses to
+run unless confirmed (interactively, or with
+`SERVIETTE_ACCEPT_FINGERPRINT_CHANGES=1`), because rows already in the vector
+DB keep their old chunk ids and would be left behind as orphans. The check
+runs with persistence on or off: a restart without persistence re-indexes
+everything, but does not remove those rows either.
+
 A document whose bytes cannot be fetched (after `indexer.fetch_retries`
 retries) or parsed does not stop the indexer: it is indexed as **empty** and
 logged at ERROR level with its path. That result sticks for this version of

serviette/config/schema.py

@@ -584,7 +584,7 @@ class IndexerConfig(BaseModel):
     # limits, connection resets — so 0 is only for tests.
     fetch_retries: int = Field(default=3, ge=0)
     # Advanced. Disk budget for the parse cache (pw.udfs.DefaultCache, stored
-    # under <persistence dir>/PStorage/runtime_calls, LRU-evicted). Size it at least to
+    # under <workdir>/persistence/runtime_calls, LRU-evicted). Size it at least to
     # the extracted-text volume of the corpus to avoid re-parse churn.
     parse_cache_size_gb: int = Field(default=8, ge=1)
     # Advanced. If set, the engine keeps the memoization cache of
@@ -610,34 +610,28 @@ class PersistenceConfig(BaseModel):
     Carries three things: incremental state across restarts (no re-embedding
     of unchanged documents), correct retraction of files deleted while the
     indexer was down, and the parse cache (``runtime_calls``). Advanced users
-    tune or disable it by editing the config directly; disabling also
-    disables the parse cache.
-
-    Layout: ``path`` is serviette's data directory. serviette's own artifacts
-    (the configuration fingerprint) live at its top level; the Pathway engine
-    gets the ``PStorage`` subdirectory (:meth:`engine_path`) to itself. The
-    engine treats every entry of its directory as persisted state and logs an
-    ERROR for anything it did not write, so the two must not share a folder.
+    disable it by editing the config directly; disabling also disables the
+    parse cache.
+
+    The state lives in the ``persistence`` subdirectory of the working
+    directory (``workdir_path``, see :meth:`ServietteConfig.persistence_dir`);
+    there is no separate path to configure.
     """
 
     model_config = ConfigDict(extra="forbid")
 
-    ENGINE_SUBDIR: ClassVar[str] = "PStorage"
-
     enabled: bool = True
     backend: Literal["filesystem"] = "filesystem"
-    # Relative to the indexer's working directory; writable out of the box.
-    path: str = "./persistence"
-
-    def data_path(self) -> Path:
-        """serviette's data directory (fingerprint and other own artifacts)."""
-
-        return Path(self.path)
 
-    def engine_path(self) -> Path:
-        """The directory handed to the Pathway persistence backend."""
-
-        return self.data_path() / self.ENGINE_SUBDIR
+    @model_validator(mode="before")
+    @classmethod
+    def _no_path_here(cls, data: Any) -> Any:
+        if isinstance(data, dict) and "path" in data:
+            raise ValueError(
+                "persistence.path is not a setting: the persistence state lives "
+                "in the working directory — set the top-level workdir_path instead."
+            )
+        return data
 
 
 class ServerConfig(BaseModel):
@@ -722,6 +716,13 @@ class ServietteConfig(BaseModel):
     parser: list[ParserRule] | None = None
     indexer: IndexerConfig = Field(default_factory=IndexerConfig)
 
+    # serviette's working directory: everything it keeps on disk between
+    # runs goes under here, each concern in its own subdirectory — the
+    # Pathway persistence state (``persistence/``, together with the parse
+    # cache), and the configuration fingerprint at the top level. Relative
+    # paths resolve against the process's working directory, like every other
+    # path in the config. Silent default — the wizard does not ask.
+    workdir_path: str = "./serviette-workdir"
     persistence: PersistenceConfig = Field(default_factory=PersistenceConfig)
     server: ServerConfig = Field(default_factory=ServerConfig)
     up: UpConfig = Field(default_factory=UpConfig)
@@ -730,6 +731,23 @@ class ServietteConfig(BaseModel):
     rag: RagConfig | None = None
     frontend: FrontendConfig | None = None
 
+    # -- working directory layout ---------------------------------------------
+
+    PERSISTENCE_SUBDIR: ClassVar[str] = "persistence"
+
+    def workdir(self) -> Path:
+        return Path(self.workdir_path)
+
+    def persistence_dir(self) -> Path:
+        """The directory handed to the Pathway persistence backend.
+
+        A subdirectory of its own because the engine treats every entry of the
+        directory it is given as persisted state and logs an ERROR for
+        anything it did not write.
+        """
+
+        return self.workdir() / self.PERSISTENCE_SUBDIR
+
     # -- per-component requirements -------------------------------------------
 
     def require(self, *fields: str) -> None:

serviette/demo/__init__.py

@@ -105,11 +105,8 @@ def build_demo_config(
             "path": str(demo_dir / "embeddings.duckdb"),
             "table": "demo_embeddings",
         },
-        "persistence": {
-            "enabled": True,
-            "backend": "filesystem",
-            "path": str(demo_dir / "persistence"),
-        },
+        "workdir_path": str(demo_dir / "workdir"),
+        "persistence": {"enabled": True, "backend": "filesystem"},
         "server": {"host": "127.0.0.1", "port": port},
     }
     if embedder == "sentence_transformer":
@@ -167,7 +164,7 @@ def _reset_index_if_embedder_changed(demo_dir: Path, embedder: str) -> None:
         embedder,
     )
     (demo_dir / "embeddings.duckdb").unlink(missing_ok=True)
-    shutil.rmtree(demo_dir / "persistence", ignore_errors=True)
+    shutil.rmtree(demo_dir / "workdir", ignore_errors=True)
 
 
 def prepare_demo_dir(

serviette/frontend/main.py

@@ -56,7 +56,11 @@ def create_app(config: ServietteConfig, *, client: httpx.AsyncClient | None = No
     async def lifespan(_: FastAPI):
         nonlocal client
         if client is None:
-            client = httpx.AsyncClient(base_url=fe.api_url, timeout=120.0)
+            # trust_env=False: the API is a sibling service (same host or
+            # network); routing its calls through the environment's
+            # http_proxy turns a working setup into "Could not reach the
+            # API" wherever a corporate proxy is configured.
+            client = httpx.AsyncClient(base_url=fe.api_url, timeout=120.0, trust_env=False)
         try:
             yield
         finally:

serviette/indexer/fingerprint.py

@@ -9,9 +9,9 @@
 keep orphaned rows.
 
 To catch that, the indexer stores the full risk-relevant objects (not just a
-hash) in ``<persistence dir>/serviette-fingerprint.json`` (next to the engine's
-own ``PStorage`` subdirectory, never inside it — the engine logs an ERROR for
-every foreign entry in its directory):
+hash) in ``<workdir>/serviette-fingerprint.json`` (at the top level of the
+working directory, never inside the engine's ``persistence/`` subdirectory —
+the engine logs an ERROR for every foreign entry in its directory):
 
 - the splitter config,
 - the embedder identity: type, model, ``document_prefix`` and every extra
@@ -24,10 +24,12 @@
   deterministic UDF too, so rows written under an older scheme cannot be
   retracted by a newer one.
 
-On startup with persistence enabled the stored objects are compared with the
-current ones. Any difference is reported as an explicit diff with a
-human-readable explanation of the risk, and the indexer refuses to start
-unless the user confirms — interactively on a TTY, or via
+On startup the stored objects are compared with the current ones. The check
+does not depend on persistence being enabled: without it a restart re-indexes
+every document, and rows written under the old settings (other chunk ids)
+stay in the vector DB just the same. Any difference is reported as an
+explicit diff with a human-readable explanation of the risk, and the indexer
+refuses to start unless the user confirms — interactively on a TTY, or via
 ``SERVIETTE_ACCEPT_FINGERPRINT_CHANGES=1`` in non-interactive deployments.
 Confirmation updates the stored fingerprint.
 """
@@ -183,11 +185,9 @@ def _confirm(diff_lines: list[str]) -> bool:
 
 
 def check_fingerprint(config: ServietteConfig) -> None:
-    """Verify (and maintain) the persisted fingerprint; may abort startup."""
+    """Verify (and maintain) the stored fingerprint; may abort startup."""
 
-    if not config.persistence.enabled:
-        return  # no persisted state to be inconsistent with
-    directory = config.persistence.data_path()
+    directory = config.workdir()
     directory.mkdir(parents=True, exist_ok=True)
     path = directory / _FILENAME
 
@@ -208,8 +208,8 @@ def check_fingerprint(config: ServietteConfig) -> None:
         return
 
     logger.critical(
-        "The indexing configuration differs from the one this persistence "
-        "directory (%s) was built with:\n%s",
+        "The indexing configuration differs from the one this working "
+        "directory (%s) was indexed with:\n%s",
         directory,
         "\n".join("  " + line for line in diff_lines),
     )
@@ -218,7 +218,8 @@ def check_fingerprint(config: ServietteConfig) -> None:
             "Refusing to start: the configuration change above can corrupt "
             "incremental updates of already-indexed documents. Either revert "
             "the change, re-index from scratch (drop the vector collection "
-            f"and {directory}), or set {_ACCEPT_ENV}=1 / confirm interactively "
+            f"and the working directory {directory}), or set {_ACCEPT_ENV}=1 / "
+            "confirm interactively "
             "to accept the risks."
         )
     path.write_text(json.dumps(current, indent=2, sort_keys=True))