Commit 7ca488e
test(e2e): strengthen fast coverage for live contracts (#10480)
<!-- markdownlint-disable MD041 -->
## Outcome
Deterministic onboarding and messaging contracts that previously
depended on live E2E runs now have fast integration or `e2e-support`
coverage. Changes to live E2E tests and their helper modules must
include a mapped, non-comment fast-test change.
This PR does not remove, disable, or shorten any live E2E target.
## Reason
Recent E2E-related merges exposed three gaps: the shared onboarding
lifecycle mock modeled `runner.run` but not `runner.runCapture`;
deterministic messaging state and proof parsing lacked fast regression
coverage; and helper-only live E2E changes were outside the existing
parity check. Review also found that the Hermes Slack credential scan
crossed unsafe trust boundaries.
## Changes
- Centralizes the onboarding fixture sandbox identity and keeps
lifecycle state consistent across `runner.run` and `runner.runCapture`.
A subprocess integration test protects the composed fixture contract.
- Adds fast tests for persisted channel lifecycle state, Google Chat
provider-egress proof parsing, Hermes Slack API proof parsing, messaging
setup, sandbox identity, sparse checkout, and related E2E mappings.
- Extends the parity manifest and checker so non-comment changes to a
live test or any declared `liveSources` helper require a non-comment
change to a mapped fast test.
- Keeps raw Slack tokens on the host during file/log/process leak scans.
The sandbox receives only token byte lengths and SHA-256 fingerprints
through `openshell sandbox exec`; its scanner returns `LEAK`, `OK`, or
`EMPTY` without receiving or returning credential values.
- Adds behavior tests that execute fake OpenShell and SSH binaries,
prove SSH is never invoked, verify no raw token enters the transport
payload, exercise leak and clean fingerprint scans, and check transport
failures propagate.
- Describes Google Chat `401` and Slack authentication errors only as
provider-egress evidence. These responses cannot prove which
authorization value reached the provider; the controlled-capture
messaging E2E remains the owner of actual Slack credential-rewrite
proof.
- Removes unapproved workflow/YAML source-shape tests. The repository
source-shape budget is zero; no production workflow is changed by this
PR.
## Verification
Latest head `4c28a1b507273c054fba9299f4158217e2ca7f22` (the verified
PR-specific fixes are unchanged from
`3127999877fdfc0a75f548378e31db3557fdf6f2`; this head merges current
`main`):
- Focused `e2e-support` tests — 18/18 passed.
- E2E parity integration tests — 15/15 passed.
- Shared onboarding integration migration — 101 tests passed; the
post-merge composed fixture contract also passed 2/2.
- Codebase growth guardrails — 32/32 passed.
- `npm run test:projects:check` — passed.
- `npm run source-shape:check` — passed with zero source-shape cases.
- `npm run checks:repository` — passed.
- Real PR-diff E2E parity check — passed.
- `npm run validate:pr` — passed, including TypeScript, secret scanning,
E2E phase checks, formatting, lint, and repository policy gates.
- Exact-head live E2E on merged head `4c28a1b` — 3/3 passed: Hermes
lifecycle
[run](https://github.com/NVIDIA/NemoClaw/actions/runs/33151543391), plus
OpenClaw lifecycle and Slack isolation
[run](https://github.com/NVIDIA/NemoClaw/actions/runs/33152685951). An
initial Hermes attempt received an external provider HTTP 403 before
sandbox creation; the bounded retry passed.
## Review notes
- Sensitive-path context: this PR changes test fixtures and live E2E
assertion helpers for onboarding, credentials, policies, sandboxes, and
messaging. It does not change production source behavior. No
sensitive-path review waiver is claimed.
- Raw Slack credentials remain host-side during the scan; neither the
OpenShell command arguments nor sandbox stdin contain credential values.
- The deleted issue-9880 workflow parser must not be restored without a
reviewed security or compatibility exception to the repository's zero
source-shape budget.
---
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
---------
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>1 parent 29e79e7 commit 7ca488e
40 files changed
Lines changed: 2024 additions & 1181 deletions
File tree
- scripts/checks
- src/lib/actions/sandbox
- test
- automation/e2e
- e2e
- docs
- live
- support
- helpers
- onboarding
- security
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
11 | 13 | | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
15 | 17 | | |
| 18 | + | |
16 | 19 | | |
17 | 20 | | |
18 | 21 | | |
| |||
23 | 26 | | |
24 | 27 | | |
25 | 28 | | |
| 29 | + | |
26 | 30 | | |
27 | 31 | | |
28 | 32 | | |
| |||
50 | 54 | | |
51 | 55 | | |
52 | 56 | | |
53 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
54 | 61 | | |
55 | 62 | | |
56 | 63 | | |
| |||
91 | 98 | | |
92 | 99 | | |
93 | 100 | | |
| 101 | + | |
94 | 102 | | |
95 | 103 | | |
96 | 104 | | |
| |||
106 | 114 | | |
107 | 115 | | |
108 | 116 | | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
109 | 125 | | |
110 | 126 | | |
111 | 127 | | |
| |||
126 | 142 | | |
127 | 143 | | |
128 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
129 | 157 | | |
130 | 158 | | |
131 | 159 | | |
| |||
135 | 163 | | |
136 | 164 | | |
137 | 165 | | |
138 | | - | |
139 | | - | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
140 | 186 | | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
141 | 199 | | |
| 200 | + | |
142 | 201 | | |
143 | 202 | | |
144 | 203 | | |
| |||
161 | 220 | | |
162 | 221 | | |
163 | 222 | | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
164 | 235 | | |
165 | 236 | | |
166 | 237 | | |
| |||
172 | 243 | | |
173 | 244 | | |
174 | 245 | | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
179 | 250 | | |
180 | 251 | | |
181 | 252 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
151 | 151 | | |
152 | 152 | | |
153 | 153 | | |
154 | | - | |
| 154 | + | |
155 | 155 | | |
156 | 156 | | |
157 | 157 | | |
| |||
0 commit comments