Skip to content

Commit c6f2ed9

Browse files
guohddguohdd
authored
Merge branch 'main' into hongshi/macos_app_crashes
2 parents 9796d6f + 8fcfa7f commit c6f2ed9

1,380 files changed

Lines changed: 35935 additions & 10566 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/e2e-audit/SKILL.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
---
2+
name: e2e-audit
3+
description: Judge whether Agent behavior belongs in a new-e2e, integration, or unit test
4+
allowed-tools: Read, Glob, Grep, Bash
5+
argument-hint: "<behavior description> | <path-to-test-file-or-dir> [more paths...]"
6+
model: sonnet
7+
---
8+
9+
Decide whether assertions about proposed or existing Agent behavior belong in
10+
a full `new-e2e` test, an integration test, or a unit test. Produce only a
11+
verdict and rationale. Never edit, move, or delete tests.
12+
13+
## Principle
14+
15+
Analyze observable claims and failure modes, not entire test files or individual
16+
`assert` or `require` calls. Group checks that validate the same contract as one
17+
assertion, including implicit claims such as successful installation, startup,
18+
or command execution.
19+
20+
Choose the cheapest test that preserves the boundary and failure mode that
21+
matter. Do not ask only whether an assertion *can* be made with mocks; ask what
22+
the assertion would stop validating if its dependencies were replaced.
23+
24+
### E2E justified
25+
26+
Use `new-e2e` when the deployed Agent or its real environment is part of the
27+
behavior being validated. This includes boundaries such as:
28+
29+
- installation, packaging, permissions, or merged deployed configuration;
30+
- process or service lifecycle, cross-process communication, or CLI behavior
31+
that depends on the running Agent;
32+
- kernel, operating-system, cloud, container-orchestrator, or network behavior
33+
that cannot be represented faithfully by a local dependency;
34+
- user-visible data reaching fakeintake when the Agent's real assembly,
35+
configuration, encoding, or forwarding path is material to the test.
36+
37+
A behavior is not E2E-worthy merely because the current test reaches it through
38+
SSH, a CLI, or remote infrastructure. If those layers add no relevant coverage,
39+
use a lower-level test.
40+
41+
### Should be an integration test
42+
43+
Use an integration test when the important boundary can be preserved locally,
44+
for example by wiring components with `fx.Test`, using a local fakeintake, or
45+
using a real local daemon, driver, or hardware dependency. A real local
46+
dependency does not by itself require `new-e2e`.
47+
48+
### Should be a unit test
49+
50+
Use a unit test when the behavior is isolated logic and does not require a real
51+
component graph, process, or external dependency.
52+
53+
### Layered coverage and residual E2E value
54+
55+
Duplicating an E2E assertion in a unit or integration test is useful when the
56+
lower-level test preserves its failure mode: it provides faster PR feedback,
57+
more deterministic failures, and easier debugging. Do not treat this useful
58+
duplication as waste by itself.
59+
60+
After identifying lower-level coverage, reevaluate what the E2E test uniquely
61+
validates. Repeating the same assertion through the deployed Agent can still be
62+
valuable when it catches assembly, configuration, packaging, lifecycle, or
63+
forwarding failures that the lower-level test cannot. If nearly all material
64+
assertions have equivalent lower-level coverage and the E2E test preserves no
65+
meaningful additional boundary, its feedback no longer justifies its
66+
provisioning, runtime, and maintenance cost; recommend removing it. Base this
67+
decision on residual failure coverage, not only the number of duplicated
68+
assertions.
69+
70+
## Procedure
71+
72+
### Proposed behavior
73+
74+
1. List the observable claims that the proposed test would assert.
75+
2. For each assertion, state the failure it is intended to catch.
76+
3. Identify the smallest environment that preserves each failure mode.
77+
4. Classify each assertion. If E2E is justified, name the real boundary that
78+
would be lost in a lower-level test.
79+
5. Give an overall verdict based on the assertion requiring the broadest real
80+
boundary.
81+
82+
If the description does not establish the relevant assertions or boundaries,
83+
ask for the missing information or return an explicitly uncertain verdict.
84+
85+
### Existing tests
86+
87+
1. Resolve every concrete suite. For directories, find all `*_test.go` files.
88+
Follow shared suites, setup code, helpers, and provisioners rather than
89+
judging files in isolation.
90+
2. Read each test and subtest, including setup, gating, environment updates,
91+
and cleanup. Inspect the production code it exercises before deciding that
92+
an assertion can be tested at a lower level.
93+
3. Inventory the observable claims in each test. Include implicit claims from
94+
setup and lifecycle operations, and group checks that cover the same
95+
contract.
96+
4. For each assertion, state the failure it catches, identify the smallest
97+
environment that preserves that failure, note equivalent lower-level
98+
coverage, and classify it using the principle above.
99+
5. Give the suite-level verdict:
100+
- If any assertion needs the deployed Agent boundary, the suite remains E2E;
101+
name the assertion and boundary that determine this verdict.
102+
- If none does, recommend an integration or unit test as appropriate.
103+
6. Recommend duplicating suitable assertions at a lower level when that would
104+
provide faster PR feedback, more deterministic failures, or easier
105+
debugging, even if the suite initially remains E2E.
106+
7. After accounting for that lower-level coverage, identify the failure modes
107+
and real boundaries that only the E2E test preserves. If nearly all material
108+
assertions are duplicated and no meaningful E2E-only boundary remains,
109+
recommend removing the E2E test. Weigh its residual value against runtime,
110+
provisioning, flakiness, and maintenance cost; do not assume all cost is
111+
paid only once at suite setup.
112+
8. When reviewing several suites, briefly note redundant provisioners or
113+
equivalent coverage that could be consolidated.
114+
115+
For large reviews, inspect files in parallel if possible, then verify and
116+
synthesize the results.
117+
118+
## Output
119+
120+
Use one of these verdicts:
121+
122+
- **E2E justified**
123+
- **Should be an integration test**
124+
- **Should be a unit test**
125+
126+
For proposed behavior, classify each expected assertion and return an overall
127+
verdict with a short reason. For existing tests, classify each material
128+
assertion and return one verdict per concrete suite, naming the assertion and
129+
boundary that determine it. State what the E2E test uniquely validates after
130+
accounting for lower-level coverage, or say that no material E2E-only boundary
131+
remains. When several assertions differ, use a concise table with these
132+
columns: assertion, failure caught, smallest environment, and classification.
133+
Add only material uncertainty, lower-level candidates, or consolidation
134+
opportunities.
135+
136+
### Examples
137+
138+
**Input:** Verify that installing the Agent package creates a running service
139+
with the expected permissions and that data reaches fakeintake after a reboot.
140+
141+
**Output:** **E2E justified** — The package installation, service lifecycle,
142+
permissions, reboot, and forwarding path are the behavior under test; a
143+
lower-level test would not preserve those deployed-system boundaries.
144+
145+
**Input:** Verify that the assembled Agent components transform a payload and
146+
send it to a local fakeintake.
147+
148+
**Output:** **Should be an integration test** — A locally assembled component
149+
graph and fakeintake preserve the component wiring and payload boundary without
150+
provisioning remote infrastructure.
151+
152+
**Input:** Verify that the configuration parser rejects a negative timeout and
153+
applies the default when the field is absent.
154+
155+
**Output:** **Should be a unit test** — This is isolated parsing and validation
156+
logic that does not require a real component graph, process, or external
157+
dependency.
158+
159+
Do not propose implementation changes unless the user asks for them.

0 commit comments

Comments
 (0)