fix(capture): missing response bodies, serial false positives — v0.6.1 (#38)

* fix(sanitization): serial number false positives, JS variable detection, pipe testability

Two bugs found via TG3442DE captures: (1) SN prefix in JSON keys like
SNRLevel matched the serial pattern due to missing word boundary, and
(2) JS variable assignments like `var js_SerialNumber = 'value'` were
not detected. Fixes both with spec-first approach: document the 100%
confidence boundary for scanner passes, then TDD the implementation.

Additionally extracts _sanitize_pipe_value() from the 70-line nested
closure into a module-level function for direct unit testing, removes
dead code (unreachable version-string guard for private IPs), and adds
37 fixture-driven test cases covering 9 previously untested scanner
passes. html.py coverage: 73% → 95%.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(capture): patch missing response bodies from eager capture cache

Playwright's HAR recorder fetches bodies lazily at context.close() via
CDP Network.getResponseBody. If a navigation evicts the response from
Chrome's buffer before the flush, the body is lost. This adds eager
body capture via page.on("response") and patches missing bodies into
the HAR before sanitization.

Also removes the no-op context.route("**/*") handler that enabled the
CDP Fetch domain, which can interfere with Network domain body capture.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore(release): bump version to 0.6.1

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Ken Schulz <kwschulz@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Author: kwschulz

CHANGELOG.md

@@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.6.1] - 2026-04-08
+
+### Fixed
+
+- **Missing response body on initial page load** — Playwright's HAR recorder fetches bodies lazily at `context.close()` via CDP `Network.getResponseBody`. If a navigation (e.g., form POST) evicts the response from Chrome's buffer before the flush, the body is lost — headers and sizes are correct but `content.text` is absent. Added eager body capture via `page.on("response")` for text content types and a post-capture `_patch_missing_bodies()` step that fills missing bodies from the cache before sanitization.
+- **Serial number false positives** — Reduced false positive serial number detections in sanitization. Improved JS variable name detection and pipe-delimited pattern testability.
+
+### Removed
+
+- **No-op route handler** — Removed `context.route("**/*", lambda route: route.continue_())` which enabled the CDP Fetch domain unnecessarily, potentially interfering with HAR body capture.
+
 ## [0.6.0] - 2026-04-06
 
 ### Changed
@@ -464,4 +475,5 @@ har-capture sanitize input.har --patterns custom-allowlist.json
 [0.5.0]: https://github.com/solentlabs/har-capture/compare/v0.4.5...v0.5.0
 [0.5.1]: https://github.com/solentlabs/har-capture/compare/v0.5.0...v0.5.1
 [0.6.0]: https://github.com/solentlabs/har-capture/compare/v0.5.1...v0.6.0
-[unreleased]: https://github.com/solentlabs/har-capture/compare/v0.6.0...HEAD
+[0.6.1]: https://github.com/solentlabs/har-capture/compare/v0.6.0...v0.6.1
+[unreleased]: https://github.com/solentlabs/har-capture/compare/v0.6.1...HEAD

docs/ARCHITECTURE.md

@@ -175,6 +175,8 @@ The pattern system is how har-capture stays domain-agnostic while supporting dom
 
 This means a consumer like cable_modem_monitor ships its own pattern file and gets domain-tuned sanitization without har-capture carrying any modem-specific code. Consumers can layer multiple `--patterns` arguments for incremental customization.
 
+**Confidence boundary:** Domain `pii.patterns` entries run as Pass 0 auto-redaction — they must have 100% confidence (zero false positives). Domain `heuristics.detectors` entries flag values for interactive review and can tolerate lower confidence. When a domain-specific pattern cannot guarantee zero false positives, it belongs in `heuristics.detectors`, not `pii.patterns`.
+
 See [Pattern Spec](specs/PATTERN_SPEC.md) for file schemas, merge semantics, and the loader/cache architecture.
 
 ## Functional Specs

docs/specs/CAPTURE_SPEC.md

@@ -159,13 +159,14 @@ Controls the `wait_until` argument to Playwright's `page.goto()`. Accepts any Pl
 
 ### Internal Decomposition
 
-`capture_device_har()` is the public API — its signature is unchanged. Internally, it delegates to four extracted functions that can each be tested independently:
+`capture_device_har()` is the public API — its signature is unchanged. Internally, it delegates to five extracted functions that can each be tested independently:
 
 ```
 capture_device_har()
 ├── Pre-flight checks (check_playwright, check_browser_installed)
 ├── _resolve_capture_paths()   → CapturePathInfo
 ├── _run_browser_session()     → BrowserSessionResult
+├── _patch_missing_bodies()    → patches temp HAR in-place
 ├── _inject_har_metadata()     → modifies temp HAR in-place
 └── _run_post_capture_pipeline() → CaptureResult
 ```
@@ -256,6 +257,7 @@ class BrowserSessionResult:
     browser_cookies: list[Any]               # Cookies after page load
     web_storage_local: list[dict[str, Any]]  # localStorage entries per origin
     web_storage_session: dict[str, str]      # sessionStorage key/value pairs
+    captured_bodies: dict[str, bytes]        # Eagerly captured response bodies
     success: bool
     error: str | None
 ```
@@ -397,7 +399,33 @@ This is more robust than Playwright's `networkidle` (500ms) — it waits for `_D
 
 #### `--no-wait-for-data` Behavior
 
-When disabled, no JS injection, no quiescence polling, and no `framenavigated` listener. A context-level route (`context.route("**/*", ...)`) still handles cache control. `page.goto()` with `wait_until="networkidle"` is the only wait mechanism.
+When disabled, no JS injection, no quiescence polling, and no `framenavigated` listener. `page.goto()` with `wait_until="networkidle"` is the only wait mechanism. The eager response body capture listener (`page.on("response")`) is always active regardless of this flag.
+
+### Eager Response Body Capture
+
+#### Problem
+
+Playwright's `record_har_content="embed"` captures response bodies lazily — it calls CDP `Network.getResponseBody` when `context.close()` flushes the HAR to disk. If a navigation event causes Chrome to evict the response data from its network buffer before the flush, the body is lost. Headers, sizes, and timing are correct (captured synchronously from Network domain events), but `content.text` is absent and `content.size` is `-1`.
+
+This typically affects the initial page load (e.g., a login form) when the user or JavaScript submits a form quickly, triggering a navigation that supersedes the first response.
+
+#### Solution: Eager Body Capture via Response Listener
+
+Before navigation, a `page.on("response")` listener is registered that eagerly calls `response.body()` for text-based content types (`text/*`, `application/json`, `application/xml`). Bodies are stored in `BrowserSessionResult.captured_bodies` keyed by `"<method>|<url>|<status>"`.
+
+After Playwright writes the HAR and before metadata injection, `_patch_missing_bodies()` scans HAR entries for responses that have `bodySize > 0` or `_transferSize > 0` but no `content.text`. For each missing body, it looks up the key in the captured bodies cache and patches the body into the HAR entry.
+
+Text bodies are stored as plain UTF-8 strings. Non-UTF-8 bodies fall back to base64 encoding with `content.encoding = "base64"`.
+
+#### `_patch_missing_bodies(temp_path, captured_bodies) -> int`
+
+- Reads the raw HAR from `temp_path`
+- For each entry missing `content.text` with `bodySize > 0` or `_transferSize > 0`: looks up `"<method>|<url>|<status>"` in `captured_bodies` and patches the body
+- Writes the patched HAR back to `temp_path`
+- Returns the number of entries patched
+- Handles corrupt HAR files gracefully (returns 0)
+
+Testable with: zero mocks (real temp file).
 
 ### Timeout vs Interactive Mode
 
@@ -459,9 +487,13 @@ Error messages are sanitized via `_sanitize_error_message()` to remove any embed
 
 ## Post-Capture Processing
 
+### Body Patching
+
+Immediately after the browser closes and writes the raw HAR, `_patch_missing_bodies()` scans for entries with missing response bodies and patches them from the eagerly captured body cache. This runs before metadata injection and sanitization so that downstream processing sees complete responses.
+
 ### Metadata Injection
 
-After the browser closes and writes the raw HAR:
+After body patching:
 
 ```python
 har_data["log"]["_probes"] = probes_data

docs/specs/PATTERN_SPEC.md

@@ -306,7 +306,15 @@ Examples for `network-device`: `qam256`, `atdma`, `bpi+`, `honor mdd`, `dhcpclie
 
 ### Section: `pii.patterns`
 
-Additional PII detection patterns. Same schema as `pii.json` `patterns` entries.
+Additional PII patterns for deterministic auto-redaction (Pass 0 of the HTML scanner). Same schema as `pii.json` `patterns` entries.
+
+**Confidence requirement:** Every pattern runs as auto-redact — the matched value is replaced without user review. Patterns MUST achieve 100% confidence. If a pattern cannot meet this bar, use `heuristics.detectors` instead.
+
+| Criterion      | `pii.patterns`              | `heuristics.detectors`      |
+| -------------- | --------------------------- | --------------------------- |
+| Confidence     | 100% — zero false positives | Lower confidence acceptable |
+| Action         | Auto-redact (irreversible)  | Flag for user review        |
+| Pipeline stage | Pass 0 (scanner)            | Heuristic engine            |
 
 ### Built-in Domain: `network_device.json`
 

docs/specs/SANITIZATION_SPEC.md

@@ -209,30 +209,33 @@ MIME-type based routing:
 
 The engine runs sequential passes over HTML/JavaScript content (numbered 0–16 in the code, with sub-passes like 0b, 2b, 7a/7b, 8b). Each pass uses regex substitution with callback functions that invoke the hasher.
 
-| Pass | Scanner                       | Pattern                                         | Redaction                              |
-| ---- | ----------------------------- | ----------------------------------------------- | -------------------------------------- |
-| 0    | Custom patterns               | Domain-specific PII regex                       | Per-pattern prefix                     |
-| 0b   | Web storage                   | `localStorage.setItem('KEY', 'VALUE')`          | Auto-redact if key is sensitive        |
-| 1    | MAC addresses                 | `([0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}`         | `hasher.hash_mac()`                    |
-| 2    | Serial numbers (inline)       | `SN\|S/N\|Serial Number` + value                | `hasher.hash_value(val, "SERIAL")`     |
-| 2b   | Serial numbers (table)        | `<td>Label</td><td>VALUE</td>`                  | `hasher.hash_value(val, "SERIAL")`     |
-| 3    | Account/subscriber IDs        | `Account\|Subscriber\|Customer\|Device` + value | `hasher.hash_value(val, "ACCOUNT")`    |
-| 4    | Private IPs                   | RFC 1918 ranges (preserves gateway IPs)         | `hasher.hash_ip(ip, is_private=True)`  |
-| 5    | Public IPs                    | Non-private, non-reserved                       | `hasher.hash_ip(ip, is_private=False)` |
-| 6    | IPv6 addresses                | Full + compressed, validated via `ipaddress`    | `hasher.hash_ipv6()`                   |
-| 7    | Passwords/passphrases         | `password=value`, `passphrase=value`            | `hasher.hash_value(val, "PASS")`       |
-| 7a   | SSID text labels              | SSID labels in HTML text nodes                  | `hasher.hash_value(val, "WIFI")`       |
-| 7b   | JS password objects           | JavaScript object password fields               | `hasher.hash_value(val, "PASS")`       |
-| 8    | Password inputs               | `<input type="password" value="...">`           | `hasher.hash_value(val, "PASS")`       |
-| 8b   | SSID inputs                   | SSID-related input fields                       | `hasher.hash_value(val, "WIFI")`       |
-| 9    | Session tokens                | 20+ char alphanumeric with label prefix         | `hasher.hash_value(val, "TOKEN")`      |
-| 10   | CSRF tokens                   | CSRF tokens in meta tags                        | `hasher.hash_value(val, "CSRF")`       |
-| 11   | Email addresses               | `user+tag@sub.domain.co.uk`                     | `hasher.hash_email()`                  |
-| 12   | Config paths                  | `.cfg` file references                          | `hasher.hash_value(val, "CONFIG")`     |
-| 13   | Vendor JS vars                | Motorola `var CurrentPw_24g = '...'`            | `hasher.hash_value(val, "PASS")`       |
-| 14   | Pipe-delimited (tagValueList) | `var name = "val1\|val2\|val3"`                 | Per-value heuristic analysis           |
-| 15   | Pipe-delimited (other)        | Other pipe-delimited variables                  | Per-value heuristic analysis           |
-| 16   | SSID fields in JS             | `ssid_24g: 'value'`, `guest_ssid: 'value'`      | `hasher.hash_value(val, "WIFI")`       |
+| Pass | Scanner                       | Pattern                                             | Redaction                              |
+| ---- | ----------------------------- | --------------------------------------------------- | -------------------------------------- |
+| 0    | Custom patterns               | Domain-specific PII regex                           | Per-pattern prefix                     |
+| 0b   | Web storage                   | `localStorage.setItem('KEY', 'VALUE')`              | Auto-redact if key is sensitive        |
+| 1    | MAC addresses                 | `([0-9A-Fa-f]{2}[:-]){5}[0-9A-Fa-f]{2}`             | `hasher.hash_mac()`                    |
+| 2    | Serial numbers (inline)       | `\bSN\b\|S/N\|Serial Number` + value                | `hasher.hash_value(val, "SERIAL")`     |
+| 2b   | Serial numbers (table)        | `<td>Label\b</td><td>VALUE</td>`                    | `hasher.hash_value(val, "SERIAL")`     |
+| 2c   | JS serial variables           | Names with serial+Number/Num/No or ending in serial | `hasher.hash_value(val, "SERIAL")`     |
+| 3    | Account/subscriber IDs        | `Account\|Subscriber\|Customer\|Device` + value     | `hasher.hash_value(val, "ACCOUNT")`    |
+| 4    | Private IPs                   | RFC 1918 ranges (preserves gateway IPs)             | `hasher.hash_ip(ip, is_private=True)`  |
+| 5    | Public IPs                    | Non-private, non-reserved                           | `hasher.hash_ip(ip, is_private=False)` |
+| 6    | IPv6 addresses                | Full + compressed, validated via `ipaddress`        | `hasher.hash_ipv6()`                   |
+| 7    | Passwords/passphrases         | `password=value`, `passphrase=value`                | `hasher.hash_value(val, "PASS")`       |
+| 7a   | SSID text labels              | SSID labels in HTML text nodes                      | `hasher.hash_value(val, "WIFI")`       |
+| 7b   | JS password objects           | JavaScript object password fields                   | `hasher.hash_value(val, "PASS")`       |
+| 8    | Password inputs               | `<input type="password" value="...">`               | `hasher.hash_value(val, "PASS")`       |
+| 8b   | SSID inputs                   | SSID-related input fields                           | `hasher.hash_value(val, "WIFI")`       |
+| 9    | Session tokens                | 20+ char alphanumeric with label prefix             | `hasher.hash_value(val, "TOKEN")`      |
+| 10   | CSRF tokens                   | CSRF tokens in meta tags                            | `hasher.hash_value(val, "CSRF")`       |
+| 11   | Email addresses               | `user+tag@sub.domain.co.uk`                         | `hasher.hash_email()`                  |
+| 12   | Config paths                  | `.cfg` file references                              | `hasher.hash_value(val, "CONFIG")`     |
+| 13   | Vendor JS vars                | Motorola `var CurrentPw_24g = '...'`                | `hasher.hash_value(val, "PASS")`       |
+| 14   | Pipe-delimited (tagValueList) | `var name = "val1\|val2\|val3"`                     | Per-value heuristic analysis           |
+| 15   | Pipe-delimited (other)        | Other pipe-delimited variables                      | Per-value heuristic analysis           |
+| 16   | SSID fields in JS             | `ssid_24g: 'value'`, `guest_ssid: 'value'`          | `hasher.hash_value(val, "WIFI")`       |
+
+**Pass 2c precision rule:** Matches variable names containing the compound `serial` + `number`/`num`/`no` (with optional separator), and names ending with `serial`. Does NOT match `serial` followed by unrelated suffixes (`Protocol`, `Port`, `Baud`, `ization`). Bare `serial` is excluded — too ambiguous for auto-redact.
 
 ### Web Storage Scanner (Pass 0b)
 
@@ -551,3 +554,4 @@ Detects common redaction markers to warn users before double-sanitizing.
 1. **Cookie metadata is preserved** — Cookie attributes (HttpOnly, Secure, SameSite, Path, Domain, Expires) are detected and not redacted. Only cookie values are redacted.
 1. **Credit card detection requires Luhn** — A 16-digit number is only redacted as a credit card if it passes Luhn checksum validation.
 1. **Global find-replace in Pass 2** — User-selected redactions are applied via string replacement on the serialized JSON, ensuring all occurrences (headers, body, URLs) are caught.
+1. **Scanner passes require 100% confidence** — Every regex in the HTML scanner pipeline (passes 0–16) auto-redacts without user review. A pattern that produces false positives is a bug. Patterns that cannot achieve 100% confidence belong in the heuristic engine (flagged for user review), not the scanner pipeline.

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "har-capture"
-version = "0.6.0"
+version = "0.6.1"
 description = "HAR capture and PII sanitization library for network traffic analysis"
 readme = "README.md"
 license = "MIT"

src/har_capture/__init__.py

@@ -24,7 +24,7 @@
 
 from __future__ import annotations
 
-__version__ = "0.6.0"
+__version__ = "0.6.1"
 
 # Re-export public API for convenience
 from har_capture.sanitization import (