feat(traceability): add 21 CFR Part 11 control traceability matrix - #627
Draft
bdeitte wants to merge 104 commits into
Draft
feat(traceability): add 21 CFR Part 11 control traceability matrix#627bdeitte wants to merge 104 commits into
bdeitte wants to merge 104 commits into
Conversation
…1 example Covers @Control-* Gherkin tagging convention, a new `vip trace` command that joins tagged results.json output against a supplied control list, and a new opt-in examples/part11_validation extension as a worked starting point. PDF generation and cross-run history are explicitly out of scope.
Verified the spec's claims against the codebase and a live pytest probe, and researched real-world CSV/Part 11 artifacts (vendor compliance matrices, hapi-ds/ALC). Corrected: SARIF and JUnit do not carry markers (results.json only); unregistered @Control-* marks emit PytestUnknownMarkWarning and are fatal under strict warning settings, so a small plugin change is required; gherkin.py's first-tag-wins heuristic can be hijacked by a control tag; results.json is not deterministically ordered. Added: three-valued coverage so 'not verifiable by automated test' is distinct from 'coverage gap'; a richer controls.toml carrying reference, risk, verification and responsibility; a provenance header and stable sort order in the export; and a scope section explaining that most of Part 11, including all of subpart C, is not evidenceable against Posit Team.
Adds a new section 2 covering four items previously listed as deferred: attributability (hostname, git SHA, CI run URL), per-test start/finish timestamps, a results.json schema_version with a compatibility policy, and a SHA-256 sidecar for tamper-evidence. Renumbers the following sections. Verified against the code first: report.start/stop exist and survive xdist worker-to-controller serialization (checked under -n 0 and -n 2), and the collection site fires on the call phase or a setup-phase skip, so started_at excludes fixture setup — documented rather than glossed. Two risks called out with the design: CI checkouts embed credentials in the git remote, so userinfo is redacted before recording (results.json is an uploaded artifact, and plugin.py already strips absolute paths for the same reason); and the checksum is tamper-evidence within a trusted pipeline, not tamper-proofing, since anyone who can edit the file can regenerate the sidecar.
The schema_version tests covered unknown-major and unknown-minor but not the case that actually happens: an archived results.json written before this work lands, with no schema_version, timestamps, or execution block. load_results already uses .get() with defaults throughout, so the new fields follow that pattern and take dataclass defaults.
…rning Review found the filterwarnings approach does not cover pytest's --strict-markers mode. Confirmed against the pinned pytest 9.1.1: strict markers calls fail() and aborts collection rather than warning, so no warning filter can reach it. Replaced with a pytest_configure pre-scan that registers each discovered control-* tag via addinivalue_line. Verified this collects and passes under --strict-markers and -W error together, removes the warnings entirely, and still delivers the tags to results.json -- one mechanism for both failure modes, so the filterwarnings entry is dropped. Test plan now covers both paths separately and combined. Also corrected the Context, which still described results.json as producing 'timestamped, versioned' per-check results. That is the post-section-2 state, not today's: only a run-level generated_at exists and there is no schema version, so a per-control row cannot currently claim its own timestamp.
Twelve TDD tasks in three shipping phases: evidence-record provenance (schema version, per-test timestamps, execution attribution, sha256 sidecar), control tagging (gherkin marker derivation plus strict-marker-safe registration), then the traceability export, example and docs. Deviates from the spec in one place: build_traceability_matrix goes in a new src/vip/traceability.py rather than reporting.py, which already holds the data model and three writers. Follows the existing report_html.py split precedent.
Review found the part11 example issued DELETE /v1/audit_logs/1 against the configured deployment. In a regulated environment the audit trail is the evidence, so deleting a real record to prove records cannot be deleted does the exact harm the control exists to prevent, and breaks VIP's non-destructive test contract. Now reads the Allow header via OPTIONS instead. Pushed into ConnectClient as domain methods rather than generic HTTP verbs, so the destructive shape is not expressible at the step layer; includes a regression test asserting only OPTIONS is ever issued. Also: ConnectClient has no generic get/delete/options at all, so all three methods the example calls are new -- the plan now specifies them with tests instead of a check-first note. And redact_userinfo now guards .port, which urlsplit parses lazily and which raises ValueError on 'https://host:bad/x', escaping the never-fail contract during report writing.
PR #618 adds a native Quarto/Typst PDF edition and restructures the report layer. Verified against its branch: it does not touch reporting.py, plugin.py or gherkin.py, so the evidence-record and tagging work (plan tasks 1-6) cannot conflict with it. It does collide with cli.py, the pyproject force-include block and AGENTS.md, which tasks 10-12 all touch. Retired the PDF non-goal, which assumed an external team owned rendering -- VIP now renders its own. Whether the matrix becomes a PDF section is recorded as an open decision rather than silently assumed either way; recommendation is to ship the CSV/JSON export first and not render an unsettled data model into two backends. Updated the moved references: the Gherkin step lookup went to report_content.py:261, and provenance rendering split into provenance_rows plus one renderer per backend.
Task 13 builds the format-neutral rows in report_content.py; task 14 renders them in report_html and report_typst together, which #618's AGENTS.md requires so the two editions cannot drift. Gated on #618 merging: task 13 modifies report_content.py, which does not exist on main. Placed last so tasks 1-12 still ship if that PR slips. Two hazards handled explicitly. Control descriptions are customer-authored, so a bare # is live Typst markup -- every cell goes through _lit, with a test asserting #panic() in a description renders inert. And render_document grows an optional third parameter rather than a required one, so a deployment with no control list gets byte-identical output to today. Also fixes two review findings on the connect client: the ad-hoc unauthenticated probe now uses verify_with_env_ca (trust_env=False disables SSL_CERT_FILE too, so without it the probe fails TLS against a corporate CA the pooled client accepts), and unauthenticated_status gets real tests asserting it sends no credentials.
Review found --controls copying controls.toml into the report directory left persistent state: the directory survives between runs, so one 'vip report --controls ...' would make every later plain 'vip report' silently render a traceability section from a stale list nobody asked for. For an evidence document that is the wrong failure mode, and it contradicted the plan's own no-flag-no-section requirement. Now passed as VIP_CONTROLS through the env dict _quarto_render already forwards to subprocess.run, so the choice is scoped to exactly the render that asked for it. Verified the four cases (set+exists, set+missing, unset, empty). Also validates the control list in run_report so a malformed file fails before Quarto starts rather than inside a notebook cell. Second fix: the recording_client fixture patched httpx.Client before building ConnectClient, but BaseClient.__init__ constructs its own pooled client (base.py:135), so instances[0] was the pooled client -- which legitimately carries credentials. The credential assertion would have inspected the wrong object entirely. Fixture now builds the client first, then patches, and asserts exactly one ad-hoc client was created.
Review found controls_path() uses os.environ and warnings.warn, but reporting.py imports neither -- so every report render would have raised NameError before deciding VIP_CONTROLS was unset, breaking the no-controls path that every existing user takes. Swept the plan for the same defect class rather than fixing only the reported instance: plugin.py already has datetime/timezone/Path/warnings and the plan adds hashlib, gherkin.py has Path, connect.py has httpx. reporting.py was the only gap. Also made the report_content import instructions explicit for both backends -- they previously said 'alongside the existing imports' without naming TRACEABILITY_HEADERS and traceability_rows or their sort position.
External review noted load_results silently accepts any schema version while the documented policy says consumers refuse an unknown major. The spec scopes the hard error to vip trace, and that stands -- load_results runs inside Quarto cells where a raise is an unreadable traceback. But silent acceptance is wrong, so task 10 now also adds a non-fatal warning there, with tests for both halves.
sassoftware/sas-validation-scenarios was offered as the concrete example of the data structures to build. It is a Locust functional/load-testing framework for SAS Viya: no occurrence of 21 CFR, Part 11, GxP, traceability, IQ/OQ/PQ, qualification or deviation anywhere in its 234 files, and its INTRODUCTION.md explicitly states the scenarios do not guarantee compliance with any industry-specific regulatory requirements. It is SAS's analogue of VIP itself, so the comparison is peer-to-peer and corroborates the scoping in section 7.1 rather than redirecting it. No design change follows. Zero of the three references offered so far actually demonstrate the artifact, which suggests it lives in commercial CSV tooling rather than open source.
Add CONTROL_TAG_PREFIX constant and modify tag line parsing to skip control-prefixed tags when choosing the feature marker. Collect all tags in a new tags list returned by parse_feature_file().
Review of the earlier task-10 amendment found an empty python fence left by the edit, and its load_results snippet had a 101-char line that would have failed ruff's 100-char limit the moment an implementer copied it verbatim. Reflowed the condition into a nested if. Also clarified the RESULTS_SCHEMA_VERSION comment. Two independent reviews (#953, #954) read it as requiring a bump per additive commit; it now says outright that one version covers everything introduced alongside it and that the next bump is for the next change after this ships.
Task 7's review found the plan's own snippet accepted a non-string description (truthiness check, no isinstance), crashed with a raw TypeError on an unhashable verification value (frozenset membership hashes its operand), accepted a whitespace-only description, and reported a directory as 'not found'. All four broke the module's stated contract that malformed input leaves as a ControlListError naming the control. Corrected the snippet to match what shipped, with the ordering constraint stated: isinstance must precede .strip() or a non-string reopens the same class of violation.
…es trace_error The docstring claimed _lit escapes #, * or $ in an exception message. It doesn't -- it escapes the quote and backslash that could terminate the string literal early. The characters stay inert because the string is rendered verbatim, not because they're individually escaped.
test_warning_renders_in_red_bold used a fixture that produced no
traceability warning at all, then guarded its assertions behind an
`if 'rgb("#dc2626")' in out:` that was therefore always False -- the
test passed unconditionally regardless of what the code did. Switch
the fixture to a failing control, which reliably produces a warning,
and assert unconditionally.
Also rename the "Covered and executed" summary-table row label to
"Covered, executed and passing" in traceability_summary_rows. The
JSON export's covered_and_executed key counts a superset (any
executed control, passing or not), while the report table now splits
failing controls into their own "Covered, failing" row; the same-named
metric disagreed between the two artifacts of one compliance run. The
JSON key is left unchanged -- it already names what it counts, and
changing it would be a schema change. Updates the two stale row-label
assertions this uncovered, in test_report_content.py and
test_report_traceability.py.
Three fixes: - docs/reporting.md: the summary bullet said the JSON summary "splits covered into covered_and_executed and covered_not_executed" -- stale since covered_failed was added, and never an exclusive split since covered_and_executed already includes failing controls. Describe the three counts as overlapping and point out the report's own summary table partitions differently (mutually exclusive rows). - .github/workflows/example-report.yml: the comment above the compliance render step claimed a bad VIP_CONTROLS "fails loudly rather than going green with the section missing". It doesn't -- quarto render still exits 0 and the job stays green; only the rendered artifact shows the marker. No step, env var, or command changed, comment only. - report/index.qmd: removed a dead `_trace_error = None` leftover from mirroring vip-report.qmd -- never read anywhere in the cell.
index.qmd rendered a failed traceability build through IPython.display.Markdown, which passes raw HTML through in Quarto. The message carries the VIP_CONTROLS path and control ids read straight out of a customer-authored controls.toml -- the same fully untrusted text render_traceability already escapes, and the report is published publicly. The Typst edition routes the identical value through _lit; the HTML edition did not. Add report_html.render_traceability_error, which escapes the interpolated error through the module's _esc, and call it from the except branch via display(HTML(...)). The heading still renders on the failure path, so a regulated reader never sees the section vanish.
Two sites compounded into a false attestation. verify_results_checksum's basename fallback collected every matching entry and verified if any digest agreed, so a sidecar with several path-qualified results.json lines could attest to a file it does not describe. _rehome_sidecar then rewrote every basename match to the destination name, turning one wrong exact entry plus one right path-qualified entry into two results.json lines -- which the exact-match branch happily accepted, defeating the precedence that branch exists to enforce. verify_results_checksum now raises when the selected entries carry more than one distinct digest, on both selection paths, and says how to proceed. Entries that agree (case and path qualification aside) still verify. _rehome_sidecar keeps exact-name precedence: it rewrites exact entries when there are any, falls back to the basename only when exactly one candidate exists, and otherwise leaves the lines alone rather than inventing an authority the source never had.
`vip report --controls --results /external/results.json` with an empty or unreadable /external/results.json.sha256 rendered happily. The chain: _rehome_sidecar correctly declines to manufacture a destination sidecar from a source that parses to zero entries, and a missing destination sidecar is legal and benign, so the --controls gate -- which only ever looked at the destination -- saw nothing to refuse. `vip trace` rejects that same input as a truncated attestation. Verify the source against its own sidecar before the copy whenever --controls is set, so the compliance render is never more permissive than `vip trace` on identical bytes. Plain `vip report` stays lenient, and a source with genuinely no sidecar stays benign on both paths. Also chdir the four TestReportControls cases that still rendered into the repo's own report/ directory, where they raced each other under xdist over a shared results.json.
Make the mypy error in _resolve_trace_format explicit: the condition now checks `out is not None` before accessing `out.suffix` in the warning message, so the type narrowing is unambiguous to mypy. This error is pre-existing and unrelated to the traceability review fixes on this branch. The function is byte-identical at commit 5fbd4d0 and does not exist on main at all; it was carried through the initial traceability review PR without running full mypy checks. Runtime behavior is unchanged: the warning still prints only when inferred format disagrees with explicit format.
_rehome_sidecar's basename fallback refused to rewrite whenever more than one entry shared a basename, even when they carried identical digests -- diverging from verify_results_checksum's distinct-digest rule and silently dropping checksum verification across a `vip report --results` copy. Compare matched digests case-insensitively and rewrite them together when they agree, matching the verifier's rule exactly.
Three gaps found reviewing this branch against FDA's Computer Software Assurance guidance, which lists what the record of an assurance activity should contain. Nothing recorded who performed the testing. attribution.py identified a machine and a commit, which answers "which execution" but not "which person is accountable". VIP_PERFORMED_BY names an operator; without it VIP falls back to the CI actor, then the local login, and tags the value with which source it came from so an auditor reading a service-account name knows a human did not type it. --vip-no-attribution remains the opt-out. The execution block never reached either report edition. It has been written into results.json since attribution landed, but only vip trace --format json rendered it, so the PDF a customer archives and hands to an auditor was anonymous. Extending report_content.provenance_rows covers both backends at once. An absent block omits the rows rather than showing five "not recorded" lines, and a dirty tree is flagged next to the commit it cannot be reproduced from. The customer's risk rating reached the CSV and JSON exports but not the rendered matrix, so the artifact an auditor sees read as a flat checklist under a framework that is explicitly risk-based. It renders as a subline under the control id, like reference, rather than as a fifth column the Typst table has no width for. VALIDATION-PACKAGE.md now states where VIP sits against the CSA record checklist -- four of five items, with review and approval the customer's by design -- and points at actions/attest-build-provenance as the honest upgrade path from the sha256 sidecar.
A CI actor rendered bare read in the archived artifact exactly like an explicitly named accountable operator, which is the distinction the source field exists to make -- GITHUB_ACTOR on a scheduled run is whoever last edited the workflow, and is frequently a service account. Only an explicit VIP_PERFORMED_BY now renders unqualified. An unrecognized source, or a block carrying an identity with no source at all, is labelled rather than promoted to explicit by rendering it bare. Also narrows two doc sentences that overclaimed: the report renders five fields from the execution block, not the whole of it.
load_controls silently dropped any key it did not recognise, so a customer who wrote phase = "OQ" got no error and no column -- and so did a typo like referance, from a file whose whole job is to be the regulatory mapping of record. Unknown keys are now an error naming the key and the recognised set. Rejecting alone would only fix half of it: a regulated customer's control list carries fields VIP cannot anticipate, such as an IQ/OQ/PQ phase, a SOP reference or a control owner. [controls.<id>.extra] takes those and carries them into the CSV as trailing columns and into the JSON as a per-control object. Values must be strings for the same reason the built-in pass-through fields must, and a key colliding with an existing column is rejected rather than emitting a duplicate the spreadsheet resolves by taking the last one. Extra columns append after CSV_COLUMNS rather than slotting in beside the control metadata, so the fixed set stays an identical leading prefix whatever a given customer's control list carries. Neither report edition renders them.
run_verify forwarded --vip-report only when the value was non-empty, so `vip verify --report ''` left the plugin on its own default and wrote report/results.json -- the one thing the invocation asked it not to do. The plugin side already honored an empty value; only the CLI dropped it. Nothing else reads args.report, and argparse's default is a non-empty path, so the empty string is the only invocation whose behavior changes. Honoring it exposes a second edge, closed in the same commit. junit.xml and results.sarif are written as siblings of results.json and built by reloading it, so they cannot exist without it. While --report '' was ignored the two flags could be combined and junit still appeared; now the combination would run the whole product suite and produce nothing. The CLI refuses it before starting the suite, naming whichever of --format or --ci asked for the sibling formats. docs/reporting.md documented the old behavior as a known limitation, along with the pass-through workaround it needed. Both are gone.
Adding [controls.<id>.extra] made a CSV header cell user-controlled for the first time -- every fieldname was previously a hardcoded CSV_COLUMNS entry, so writer.writeheader() writing them raw was safe. TOML permits a quoted key, so '"=HYPERLINK(...)" = "v"' is a legal control list whose column name reaches the header unneutralized, past the protection every row value gets. Closed at both layers. load_controls rejects a key starting with one of the formula characters, which is the better error because a column named after a formula is never a legitimate regulatory field, and rejecting keeps the CSV header and the JSON key identical rather than making one sprout an apostrophe. render_csv then writes the header through the same neutralization as the rows, covering a ControlSpec built in code rather than loaded from TOML. The prefix set is now named once and shared, so the two checks cannot drift. TOML rejects a raw newline or tab inside a key on its own, so the tests reach those two through TOML escape sequences rather than assuming a second gate that is not the one under test.
The README never mentioned vip scaffold, the three example templates, or vip trace, so the extensibility that makes VIP useful to a regulated customer was reachable only from the website. The new section lists the templates, shows the scaffold-then-verify loop, and gives the 21 CFR Part 11 example its own subsection, since that is the one a customer arrives asking about. It points at VALIDATION-PACKAGE.md and repeats the scope limit rather than selling the matrix, because a reader who takes a green matrix into a validation meeting believing it is an attestation is the failure this example is written to prevent. Also adds scaffold and trace to the CLI command table, which had both missing.
# Conflicts: # selftests/test_cli_verify.py # selftests/test_report_content.py # selftests/test_reporting.py # src/vip/plugin.py # src/vip/report_content.py
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces compliance traceability support across VIP’s results pipeline: it adds execution attribution + per-test timestamps to results.json, provides a control-list join (vip trace) to generate a traceability matrix, and renders that matrix into both the HTML and PDF report outputs. It also ships a worked 21 CFR Part 11 scaffold example and wires a “compliance report” example into the website and publishing workflows.
Changes:
- Extend
results.jsonsemantics (schema versioning, per-test timestamps, execution attribution) and make report renderers consume them safely. - Add end-to-end compliance traceability rendering (HTML + Typst/PDF), including explicit “render failure” visibility to avoid silent omissions.
- Add a 21 CFR Part 11 example scaffold + docs, and publish a second “compliance traceability report” example on the website.
Reviewed changes
Copilot reviewed 61 out of 62 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| website/src/pages/report.astro | Links the standard example report page to the compliance traceability report. |
| website/src/pages/getting-started.astro | Adds “Compliance traceability” documentation and documents VIP_PERFORMED_BY. |
| website/src/pages/compliance-report.astro | New page embedding the compliance example report + PDF download. |
| website/src/components/Header.astro | Adds “Compliance Report” to the site nav. |
| src/vip/reporting.py | Adds schema versioning, per-test timestamps, and execution attribution loading with warning-only compatibility handling. |
| src/vip/report_typst.py | Renders traceability section into the Typst/PDF edition, including explicit failure visibility. |
| src/vip/report_html.py | Renders traceability section into the HTML edition with escaping + visible failure marker. |
| src/vip/report_content.py | Adds shared content model for traceability section + execution provenance rows. |
| src/vip/gherkin.py | Adds control-tag parsing support and ensures control tags don’t become derived feature markers. |
| src/vip/config.py | Clarifies behavior when the config file is missing (products deselected, not skipped). |
| src/vip/clients/packagemanager.py | Adds snapshot index reachability probe for reproducibility controls. |
| src/vip/clients/connect.py | Adds audit log client helpers to support Part 11 example scenarios. |
| src/vip/clients/base.py | Adds credential-free unauthenticated_status probe used by access-control controls. |
| src/vip/attribution.py | New module collecting host/git/CI/operator attribution with redaction + “never fail” behavior. |
| selftests/test_traceability_matrix.py | New unit coverage for traceability matrix behavior, provenance, and checksum handling. |
| selftests/test_traceability_controls.py | New unit coverage for controls.toml parsing, validation, and security constraints. |
| selftests/test_results_timestamps.py | Verifies per-test timestamps are written and old files load as None. |
| selftests/test_results_schema.py | Verifies schema version loading and warning behavior is numerically correct. |
| selftests/test_reporting.py | Adds schema-version warning tests for load_results. |
| selftests/test_report_typst.py | Adds Typst backend coverage for traceability section and error rendering/escaping. |
| selftests/test_report_traceability.py | New end-to-end tests for traceability section rendering in both backends. |
| selftests/test_report_content.py | Adds coverage for failing-controls display, traceability warnings, and execution provenance rendering. |
| selftests/test_gherkin_control_tags.py | New tests ensuring control tags don’t change derived markers and tags are collected. |
| selftests/test_feature_roots.py | New tests ensuring feature root discovery works from subdirectories + respects prune rules. |
| selftests/test_control_marker_registration.py | New tests ensuring control markers are registered for strict markers / unknown mark warnings. |
| selftests/test_connect_audit_client.py | New tests for Connect audit helpers + unauthenticated probe behavior. |
| selftests/test_cli_verify.py | Tests forwarding --vip-report even when empty and refusing incompatible format combos. |
| selftests/test_cli_scaffold.py | Ensures scaffold output doesn’t leak __pycache__/.pyc artifacts. |
| selftests/test_attribution.py | New tests for attribution collection, redaction, CI detection, and “never fail” behavior. |
| selftests/test_21CFR_part11_example.py | New tests asserting the 21 CFR Part 11 example is complete, bundled, and collectable. |
| selftests/conftest.py | Adds helper matrix_from_statuses for traceability-related test fixtures. |
| report/vip-report.qmd | Adds traceability rendering (via VIP_CONTROLS) to the Typst/PDF report generation. |
| report/styles.css | Adds styling for traceability caveat/warning text. |
| report/index.qmd | Adds conditional traceability section rendering with safe error handling and HTML escaping. |
| README.md | Documents vip scaffold and vip trace, and introduces the 21 CFR Part 11 example at a high level. |
| pyproject.toml | Ensures the new scaffold example is bundled into the wheel via force-include. |
| examples/21CFR_part11_validation/VALIDATION-PACKAGE.md | New guide explaining VIP’s role/limits in a regulated validation package. |
| examples/21CFR_part11_validation/test_21CFR_part11_workbench.py | New Workbench Part 11 example steps + scenarios. |
| examples/21CFR_part11_validation/test_21CFR_part11_workbench.feature | New Workbench Part 11 example feature with control tags. |
| examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.py | New Package Manager Part 11 example steps + scenarios. |
| examples/21CFR_part11_validation/test_21CFR_part11_packagemanager.feature | New Package Manager Part 11 example feature with control tags. |
| examples/21CFR_part11_validation/test_21CFR_part11_connect.py | New Connect Part 11 example steps + scenarios. |
| examples/21CFR_part11_validation/test_21CFR_part11_connect.feature | New Connect Part 11 example feature with control tags. |
| examples/21CFR_part11_validation/README.md | New example README describing scope/limits and how to use control tagging + traceability. |
| examples/21CFR_part11_validation/part11_refusal.py | New shared refusal assertion used by access-control scenarios. |
| examples/21CFR_part11_validation/controls.toml | New worked controls.toml demonstrating automated + not-automatable controls. |
| examples/21CFR_part11_validation/conftest.py | New fixtures to override endpoints/repo/snapshot for the example. |
| docs/test-architecture.md | Documents control tagging and updates scaffold template inventory. |
| AGENTS.md | Updates agent-facing project inventory and documents new traceability/attribution components. |
| .gitignore | Ignores new generated artifacts (sha256 sidecar, junit.xml, sarif) and staging dirs. |
| .github/workflows/website.yml | Downloads and publishes the compliance example report artifact. |
| .github/workflows/website-preview.yml | Downloads compliance report artifact and adds preview link. |
| .github/workflows/example-report.yml | Produces and uploads both standard + compliance example report artifacts. |
| .dockerignore | Ensures the new scaffold example is included in Docker build contexts where needed. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
website-preview.yml grew 9 lines earlier in the file, shifting the checkout step zizmor.yml's artipacked ignore targets from line 100 to 109, so CI started flagging it as a new finding. Separately, TestDisablingTheResultsFile ran run_verify with no --config, which only passed because a gitignored local vip.toml happened to sit in the repo root; in CI's clean checkout the config-not-found check fired before the refusal being tested, changing the expected exit code.
Contributor
|
Preview Links
|
traceability_summary_rows called control_rows(matrix) purely to count coverage values, duplicating the scenario-list build both renderers already do for the table itself; count straight from matrix.entries instead. unauthenticated_status hard-coded a 30s timeout that could diverge from the client's configured/scaled timeout, and assumed a customer-supplied endpoint path was already slash-prefixed.
A standalone Compliance Report page confused users and understated that the standard Example Report already serves most compliance needs, since both are rendered by the same vip report with an optional --controls attached. Removes the page, its nav link, and the CI work that built it (a second pytest pass against the Part 11 example extension, a second Quarto render, and the example-compliance-report artifact). Getting Started and the Example Report page now point readers at examples/21CFR_part11_validation on GitHub instead.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
I'm still figuring out if or what parts of this is a good idea, so this is still in draft. There is also definitely some wording fixes to do in here. I don't need anyone to look into this until a later week. I'm looking to see if we can do more with https://www.rzsoftware.com/blog/21-fscr-part-11
vip trace, which joinsresults.jsonagainst acontrols.tomlcontrol list and renders a CSV/JSON traceability matrix, distinguishing a control that is tagged from one whose scenarios actually ran and passedexamples/21CFR_part11_validation/scaffold template with aVALIDATION-PACKAGE.mdguide describing what VIP automates and what a customer must still authorTest plan
uv run pytest selftests/(2191 passed, 3 skipped)uvx ruff@0.15.0 checkandformat --checkon all directories