Merge pull request #732 from p2pool-starter-stack/develop-v1.11

v1.11: collapse fleet-resilience & hardening features into develop

Author: VijitSingh97

CHANGELOG.md

@@ -9,6 +9,93 @@ Pithead ships as **one product, one version** — the version lives in the top-l
 [`VERSION`](VERSION) file and every released image is tagged with it. Releases are cut
 per the process in [`docs/dev/releasing.md`](docs/dev/releasing.md).
 
+## [Unreleased]
+
+### Added
+
+- **Confirm-gated config editing for operationally-disruptive settings** (#719). The dashboard
+  config editor can now commit a small set of disruptive-but-recoverable settings behind a
+  type-to-confirm, instead of refusing them outright: the four service data directories (a move
+  re-syncs), the stratum port (rigs must repoint), the Monero/Tari clearnet initial-sync toggles
+  (the host IP is exposed during IBD, then auto-reverts to Tor), and enabling Monero pruning. Each
+  renders editable with a "you'll type `APPLY` to confirm" affordance; the typed confirmation rides
+  to the host approval gate, which requires it before the change proceeds and records the apply in
+  the control audit log as a distinct `commit-confirmed` action.
+
+### Changed
+
+- **The control-channel security perimeter stays host-only** (#719). Type-to-confirm is UX
+  friction, not a security control — a compromised dashboard that can set a field can also fill the
+  confirm box — so the perimeter is unchanged: wallets and view keys, the dashboard login and onion
+  settings, the control channel itself, the Tor egress firewall, the stratum password, node
+  endpoints and credentials, and the per-rig hosts and tokens all remain refused from the dashboard,
+  as does the heavier direction of a confirm-gated key (disabling pruning forces a full re-sync).
+- **Dashboard-confirmed data-directory moves are allowlisted to the stack's data root** (#728).
+  #719 made the four `*_DATA_DIR` moves confirmable from the dashboard behind a typed `APPLY`, but
+  the destination was still checked only by the host-side blocklist (`assert_safe_dir`), which
+  passes any non-catastrophic absolute path. A confirmed move from the dashboard is now further
+  held to an allowlist: the new location must sit under the stack's own data root (the install
+  dir's `data/`) or a parent the stack already keeps data in, else the move is refused even with
+  the typed confirmation and stays host-CLI only. The host `./pithead apply` path keeps the wider
+  blocklist — a shell operator already has filesystem-wide reach; only the dashboard-reachable move
+  is tightened, closing the destination trust-escalation the confirm-gate opened.
+- **Opt-in local miner** (#593). A box that runs the stack 24/7 can mine with its spare CPU by
+  co-locating a RigForge worker on the stack host. `./pithead setup` now asks "Also mine on this
+  machine with its spare CPU?" (off by default; also the new `local_miner.enabled` config flag),
+  and setup/apply print the two values a RigForge install needs — the stack's own stratum URL
+  (loopback `127.0.0.1:3333`, or the configured `p2pool.stratum_bind`/`stratum_port`) and the
+  stratum secret already in `.env`. The co-located worker self-registers through the proxy like any
+  other rig. Pithead only declares the intent and hands off those values; RigForge owns all
+  host-level tuning (HugePages, GRUB, MSR, governor) and the miner service. See
+  [docs/workers.md](docs/workers.md#mine-on-the-stack-host-itself).
+- **Warm XvB donation state on a backup stack (#249).** On a two-host failover pair — same wallet,
+  workers listing both hosts in `pools[]` — the backup's XvB donation controller used to cold-start
+  when the fleet failed over to it: the closed-loop split restarted from the feedforward estimate
+  and re-ramped for hours, over- or under-shooting the credited tier until it reconverged. The
+  controller's commanded donation fraction is now persisted, so a plain restart resumes the warmed
+  split instead of re-seeding cold. A backup can also point `xvb.standby.source` at the primary
+  dashboard's new read-only `/api/xvb-standby` endpoint; it periodically pulls the primary's
+  controller state and holds it as standby (inspectable in `/api/state`), then adopts it the first
+  time it actually donates at failover — so the split resumes warm. One-way (backup pulls from
+  primary), inert unless configured, and never acted on while the primary is authoritative (an idle
+  backup has no workers, so its controller stays on P2Pool). The pull follows the dashboard's
+  privacy-safe egress rule: an `.onion` source, a public IP, or any hostname rides the bridge Tor
+  SOCKS (the primary sees a Tor exit, never the backup's IP); only a provably-private/loopback IP
+  literal dials direct as a LAN hop — so the pull never opens a clearnet path, and the Security
+  panel reports its route.
+- **Opt-in fail-closed miner hold on an unrecoverable dashboard health failure** (#490). New
+  `dashboard.fail_closed` toggle, default **off**. The dashboard is an observability layer, not the
+  mining datapath (`xmrig-proxy` → `p2pool` → `monerod` runs independently of it), so by default an
+  unhealthy condition only alerts (Telegram/Healthchecks/webhook) and shows a badge while mining
+  continues. Set it `true` and a genuinely unrecoverable failure — the SQLite database failing to
+  rebuild after its own auto-heal attempt (disk full, permissions), or the `dashboard` container
+  itself crash-looping past the #337 debounce — holds `p2pool` and `xmrig-proxy` using the same
+  #35 sync-gate stop/start mechanism, with a `Miner held (fail-closed)` badge. Unlike the sync
+  gate's one-way latch it re-checks every cycle and releases on its own once the condition clears.
+  A transient write blip, a slow query, or a single failed external fetch never trips it — those
+  still only alert. Gated by the #33 control-approval path like other `dashboard.*` toggles.
+- **`/api/state` exposure for three of the #196 telemetry-backbone series (Tier-1).** The backbone
+  PR (#600) added five persisted SQLite tables with capture, storage, and retention, but shipped
+  without surfacing them to the client. This slice exposes three — `blocks` (pool block-found
+  events), `disk_growth` (hourly monerod-DB-size + host-disk-usage samples), and `xvb_history`
+  (~5-min XvB-credited scalar samples) — as range-filtered arrays under those same keys, bounded
+  at the existing 700-point chart cap for the two higher-cadence series. `network_history` and
+  `worker_history` are a separate (Tier-2) slice, not touched here. No chart renders any of these
+  series yet — that's a further follow-up.
+
+### Security
+
+- **The out-of-band audit trail is now flood-capped per worker (#724).** The `rig-edit` audit
+  source (#530) reads a worker's reported change id off the unauthenticated LAN worker feed. The
+  #530 deterministic row id collapses *repeats* of one change id to a single row, but not *distinct*
+  ones — so a malicious or compromised device presenting as a worker could report a fresh random
+  change id every poll, writing a new permanent `audit_events` row each ~30s cycle and slowly
+  filling the dashboard database (the table has no pruning). New `rig-edit` rows are now capped per
+  worker per rolling hour; beyond the cap the extra rows are dropped behind a single `rate-limited`
+  marker row and a logged warning, so the flood stays visible instead of growing the table without
+  limit. A genuine occasional rig change still records normally, and the non-attacker-controllable
+  `host-edit` and mirrored `control.log` rows are unaffected.
+
 ## [1.10.2] - 2026-07-21
 
 ### Fixed

SECURITY.md

@@ -65,14 +65,21 @@ The stack's defaults:
   API (over the stack's Tor SOCKS), refuses any mismatch or non-release tag, and limits attempts
   to one per 10 minutes — the container cannot choose an image, tag, or registry. Enabling the
   channel without a dashboard password is a validation error, on a published onion it additionally
-  requires Tor client authorization, and every mutation is audited host-side. Commits are default-denied against an explicit allowlist of
-  operational settings: a commit that changes any env key off that list — in every direction
-  (enabling, changing, or disabling) — is refused, as is anything the change preview flags
-  destructive. Wallets, dashboard auth and onion exposure, the control channel itself, the Tor
-  egress firewall, clearnet toggles, node endpoints, binds, and every credential are off the
-  list, and a key added in the future stays un-committable until deliberately listed. Those
-  edits must be applied from the host CLI; out-of-band approval is tracked in
-  [#338](https://github.com/p2pool-starter-stack/pithead/issues/338).
+  requires Tor client authorization, and every mutation is audited host-side. Commits are default-denied against an explicit allowlist. Low-risk
+  operational settings commit directly; a small set of operationally-disruptive ones — data-directory
+  moves, the stratum port, enabling clearnet initial sync, and enabling pruning — commit only behind
+  a typed confirmation in the dashboard, and only in that direction. A dashboard-confirmed
+  data-directory move is further held to an **allowlist** (#728): the new location must sit under the
+  stack's own data root (the install dir's `data/`) or a parent the stack already keeps data in;
+  a move to any other absolute path is refused even with the typed confirmation and stays host-CLI
+  only. The host CLI keeps its wider blocklist check — a shell operator already has filesystem-wide
+  reach. Everything else is refused in
+  every direction, as is anything the change preview flags destructive (including the heavy direction
+  of a confirm-gated key, e.g. disabling pruning, which forces a full re-sync). The security
+  perimeter — wallets and view keys, dashboard auth and onion exposure, the control channel itself,
+  the Tor egress firewall, node endpoints, binds, every credential, and the per-rig hosts and tokens —
+  is never dashboard-committable, with or without the typed confirmation. A key added in the
+  future stays un-committable until deliberately listed. Those edits must be applied from the host CLI.
 - Attack visibility (#349): Caddy writes a JSON access log for every dashboard vhost (LAN and
   onion), and the control channel's host-side audit log records who changed what (setting names
   only, never values). The dashboard surfaces both read-only — a burst of 401s is the
@@ -82,6 +89,18 @@ The stack's defaults:
   anonymous prober stored XSS against the operator. Both logs are size-bounded (Caddy's native
   rolling; a trim-before-append cap in the audit writer). Neither ever records a secret: Caddy
   redacts credential headers by default, and the audit writer logs key names only.
+- Out-of-band change detection (#530): the audit trail above only sees requests the dashboard
+  itself handled. Its poll loop separately watches for a `config.json` change with no matching
+  control-channel commit, and a worker control-API report for a change the dashboard never sent,
+  and appends both — `host-edit` / `rig-edit` — to the same trail, keys or worker names only. The
+  persisted trail (mirrored `control.log` rows plus these two out-of-band kinds) lives in the
+  dashboard's own database, not just the log tail, so the Security panel's hour/day/month grouping
+  covers more than `control.log`'s own trimmed window. The `rig-edit` source reads off the
+  unauthenticated worker feed, so it is rate-capped per worker (#724): a rig reporting distinct
+  change_ids on every poll can add at most a bounded number of rows per hour before the rest are
+  dropped behind a single `rate-limited` marker — one LAN device can't grow the database without
+  limit. The `host-edit` and mirrored `control.log` rows are not attacker-controllable and are not
+  capped.
 
 ### Telegram control commands (#338)
 

build/dashboard/mining_dashboard/config/config.py

@@ -429,6 +429,15 @@ def _nonneg(v):
     if c.strip()
 ]
 
+# Opt-in fail-closed miner hold on an UNRECOVERABLE health failure (#490), `dashboard.fail_closed`,
+# default false. The dashboard is an observability layer, not the mining datapath (xmrig-proxy ->
+# p2pool -> monerod is independent of it), so the default posture is alert-only: a cosmetic
+# dashboard fault must never idle the fleet. `true` reuses the #35 sync-gate's own mechanism
+# (stop/start SYNC_GATE_CONTAINERS) to hold the miner until the failure clears — see
+# DataService._apply_fail_closed_gate for what counts as "unrecoverable" (narrowly: DB recovery
+# itself failing, or the dashboard container crash-looping — never a transient blip).
+DASHBOARD_FAIL_CLOSED = os.environ.get("DASHBOARD_FAIL_CLOSED", "false").strip().lower() == "true"
+
 # Debounce: a node must be unreachable this long before it's declared DOWN, and reachable
 # this long before recovery — so a single transient timeout or a brief restart doesn't
 # kick every miner to their backups (and back) on a blip.
@@ -689,6 +698,21 @@ def _telegram_event_enabled(name, default=True):
 # Switching overhead (ms) to account for connection ramp-up time
 XVB_SWITCH_OVERHEAD_MS = 5000
 
+# --- XvB warm-standby, backup stack (#249, config.json: xvb.standby.source) ---
+# On a two-host failover pair (same wallet, workers list both in pools[]), the BACKUP points this at
+# the PRIMARY's dashboard so it can pull the primary's XvB controller state and hold it as standby —
+# so when workers fail over, the backup resumes the donation split warm instead of cold-ramping for
+# hours. Blank (the default) = off: an ordinary single stack never pulls. The value is the full URL
+# of the primary's read-only endpoint (e.g. https://<primary-onion>/api/xvb-standby, or a LAN
+# http://host:port/api/xvb-standby); an .onion source rides the bridge Tor SOCKS, a LAN source dials
+# direct — no new clearnet egress either way. It is a capability URL (it can carry the primary's
+# dashboard basic-auth as userinfo), so pithead renders it into the owner-only .env and masks it in
+# the control-channel config, exactly like the Healthchecks ping URL.
+XVB_STANDBY_SOURCE = os.environ.get("XVB_STANDBY_SOURCE", "").strip()
+# How often the backup re-pulls the primary's state (seconds). A code-level constant, not a knob:
+# the state moves slowly (the closed loop steps once per 10-min cycle), so a minute is ample.
+XVB_STANDBY_INTERVAL_S = 60
+
 # --- Data Retention Policies ---
 HISTORY_RETENTION_SEC = 30 * 24 * 3600  # 30 Days
 # How long an offline worker lingers in the live "Workers Alive" table before it falls off (#182).

build/dashboard/mining_dashboard/main.py

@@ -15,6 +15,7 @@
 from mining_dashboard.service.data_service import DataService
 from mining_dashboard.service.storage_service import StateManager
 from mining_dashboard.service.telegram_commands import TelegramCommandBot
+from mining_dashboard.service.xvb_standby import XvbStandbyPuller
 from mining_dashboard.web.server import create_app
 
 logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
@@ -37,20 +38,29 @@ def build_app() -> web.Application:
     # On-demand Telegram command interface (#45). Reads the snapshot data_service already collects;
     # a no-op unless telegram.enabled + telegram.commands.enabled + bot_token + chat_id are set.
     telegram_bot = TelegramCommandBot(data_service)
+    # Backup-stack warm-standby puller (#249): pulls the primary's XvB controller state so a
+    # failover resumes warm. Inert unless xvb.standby.source is configured.
+    xvb_standby = XvbStandbyPuller(state_manager)
 
     async def start_background_tasks(app):
         """Initializes background services upon web application startup."""
         app["data_task"] = asyncio.create_task(data_service.run())
         app["algo_task"] = asyncio.create_task(algo_service.run())
         app["telegram_task"] = asyncio.create_task(telegram_bot.run())
+        app["xvb_standby_task"] = asyncio.create_task(xvb_standby.run())
 
     async def cleanup_background_tasks(app):
         """Stops background tasks and closes resources on shutdown."""
         app["data_task"].cancel()
         app["algo_task"].cancel()
         app["telegram_task"].cancel()
+        app["xvb_standby_task"].cancel()
         await asyncio.gather(
-            app["data_task"], app["algo_task"], app["telegram_task"], return_exceptions=True
+            app["data_task"],
+            app["algo_task"],
+            app["telegram_task"],
+            app["xvb_standby_task"],
+            return_exceptions=True,
         )
         if "state_manager" in app:
             app["state_manager"].close()

build/dashboard/mining_dashboard/service/alert_service.py

@@ -765,6 +765,16 @@ async def process(self, **signals):
                 self._wallet_edges(signals.get("observed_wallet", ""))
             except Exception as exc:  # never let the tripwire break the data loop
                 logger.debug("Wallet baseline update failed (%s)", type(exc).__name__)
+            # Keep the container-health debounce state current even with every sink off (#490):
+            # `dashboard.fail_closed` reads `self.containers.is_confirmed_bad("dashboard")` off
+            # the same tracker this alerting path would otherwise be the only feeder for. No
+            # alert fires here — just the state update the alerting branch below does anyway.
+            containers = signals.get("containers")
+            if containers is not None:
+                try:
+                    self.containers.update(containers, now=signals.get("now"))
+                except Exception as exc:  # never let a tracker bug break the data loop
+                    logger.debug("Container-health update failed (%s)", type(exc).__name__)
             return []
         try:
             alerts = self.evaluate(**signals)