You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: sites/wonderland/docs/testing/campaign-processes.md
+51-53Lines changed: 51 additions & 53 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,7 +8,7 @@ We have built these guidelines for every Wonderland member to follow. As we are
8
8
9
9
## Responsibilities: who does it?
10
10
11
-
### Continuous Quality Assurance (QA), which includes:
11
+
### Continuous Quality Assurance (QA), which includes
12
12
13
13
-**Unit Tests**
14
14
-**Integration Tests**
@@ -30,38 +30,38 @@ Sliding responsibility, based on the timeline (research/design team, then team l
30
30
## Timeline: when do we do it?
31
31
32
32
- Before development:
33
-
- Collect our first protocol properties
33
+
- Collect our first protocol properties
34
34
- During development:
35
-
- Write unit tests
36
-
- Write integration tests
37
-
- Update protocol properties
35
+
- Write unit tests
36
+
- Write integration tests
37
+
- Update protocol properties
38
38
- At the end of development:
39
-
- Internal review
40
-
- Advanced testing:
41
-
- Finish collecting properties
42
-
- Fuzzing campaigns
43
-
- (Symbolic execution)
39
+
- Internal review
40
+
- Advanced testing:
41
+
- Finish collecting properties
42
+
- Fuzzing campaigns
43
+
- (Symbolic execution)
44
44
45
45
## Guidelines: how do we do it?
46
46
47
-
### Design Phase:
47
+
### Design Phase
48
48
49
49
While the project is still being defined and refined, we start preparing the advanced tests by collecting some invariants. This initial collection is often a first draft, as many of these invariants might still change during the development.
50
50
51
51
### Implementation Phase
52
52
53
53
-**Unit Tests (QA)**
54
-
- Unit tests *must* be implemented continuously, cross-testing is recommended, with developer A implementing the logic and developer B writing the unit tests.
55
-
- Aim for path coverage if the logic has few independent conditions; otherwise, branch coverage is preferred.
56
-
- Tools like Bulloak *might* be used for listing branches/conditions to test.
57
-
- Good unit tests *must* have a comprehensive branch test in addition to a high coverage, tending toward 100% by the end of the implementation phase.
58
-
- CI *might* have an action to enforce a non-inferior branch coverage on new PRs.
59
-
- Tests which are not implemented *must* use `vm.skip(true)`, and must NOT be commented out or left as an empty body. The linter should prohibit empty blocks in tests.
60
-
- Unit tests *must* be solitary tests, testing only one part of the logic and mocking the rest.
61
-
- The whole range of values leading to the same behaviour should be used in fuzzed parameters.
62
-
- Only one path should be tested at a time, especially when using fuzzed inputs.
63
-
- Forge being efficient, the number of runs made by the fuzzer shouldn't be reduced below the default 1000 (if tests are taking too long to run with this value, then a design fault should be suspected)
64
-
- Use forge-std's `bound` when there is more than a single value to exclude from the fuzzed value. `vm.assume` should be avoided as much as possible, or only to exclude a single value, like `address(0)` for instance (attention should be made to multiple "hidden" assumes)
54
+
- Unit tests *must* be implemented continuously, cross-testing is recommended, with developer A implementing the logic and developer B writing the unit tests.
55
+
- Aim for path coverage if the logic has few independent conditions; otherwise, branch coverage is preferred.
56
+
- Tools like Bulloak *might* be used for listing branches/conditions to test.
57
+
- Good unit tests *must* have a comprehensive branch test in addition to a high coverage, tending toward 100% by the end of the implementation phase.
58
+
- CI *might* have an action to enforce a non-inferior branch coverage on new PRs.
59
+
- Tests which are not implemented *must* use `vm.skip(true)`, and must NOT be commented out or left as an empty body. The linter should prohibit empty blocks in tests.
60
+
- Unit tests *must* be solitary tests, testing only one part of the logic and mocking the rest.
61
+
- The whole range of values leading to the same behaviour should be used in fuzzed parameters.
62
+
- Only one path should be tested at a time, especially when using fuzzed inputs.
63
+
- Forge being efficient, the number of runs made by the fuzzer shouldn't be reduced below the default 1000 (if tests are taking too long to run with this value, then a design fault should be suspected)
64
+
- Use forge-std's `bound` when there is more than a single value to exclude from the fuzzed value. `vm.assume` should be avoided as much as possible, or only to exclude a single value, like `address(0)` for instance (attention should be made to multiple "hidden" assumes)
65
65
66
66
**Example:**
67
67
@@ -93,32 +93,32 @@ function test_foo(uint256 inputToTest) external {
93
93
> Note: This example would show a case where using vm. Assuming with low runs can lead to incorrect conclusions about the non-existence of multiples of 10.
94
94
95
95
-**Integration Tests (QA)**:
96
-
- They *should* be conducted during development (top-down with stubs/bottom-up with drivers), this avoids having a test debt at the end of the project and helps find potential high-level bugs along the way.
97
-
- If the protocol interacts with other external contracts, they should be run on a fork of the relevant network, at a set block height, using `createSelectFork(string calldata urlOrAlias, uint256 block)` or `createSelectFork(string calldata urlOrAlias, bytes32 transaction)`.
98
-
- Integration tests should be conducted following user stories, leading from an entry point A to an end-state B. They are therefore stateful and have multiple transactions.
96
+
- They *should* be conducted during development (top-down with stubs/bottom-up with drivers), this avoids having a test debt at the end of the project and helps find potential high-level bugs along the way.
97
+
- If the protocol interacts with other external contracts, they should be run on a fork of the relevant network, at a set block height, using `createSelectFork(string calldata urlOrAlias, uint256 block)` or `createSelectFork(string calldata urlOrAlias, bytes32 transaction)`.
98
+
- Integration tests should be conducted following user stories, leading from an entry point A to an end-state B. They are therefore stateful and have multiple transactions.
99
99
-**E2E Tests (QA)**:
100
-
- These are considered *optional*,* as they might be less relevant in some projects (for instance, strictly onchain deliverable versus bridge with heavy onchain dependencies)
101
-
- They *might* be horizontal E2E from entry point to end-state, or vertical across the stack if significant logic is performed off-chain.
100
+
- These are considered *optional*,* as they might be less relevant in some projects (for instance, strictly onchain deliverable versus bridge with heavy onchain dependencies)
101
+
- They *might* be horizontal E2E from entry point to end-state, or vertical across the stack if significant logic is performed off-chain.
102
102
-**Differential Testing (QA)**: differential testing *should* be performed during refactor (e.g., using Forge snapshot for gas comparison).
103
103
-**Slither in CI**: Consider adding Slither to the CI workflow, but be mindful of potential codebase clutter from skip comments. Care is to be taken not to be over-sensitive; better to have a few real positives than a lot of false negatives. Setting on failing on high or medium severities *might* help in this regard.
104
104
-**slither-mutate (coming with medusa/echidna) or [vertigo-rs](https://github.com/RareSkills/vertigo-rs)**: both are still in development, it *might* provide an extra safety net, without creating a huge overhead. It *may* be run occasionally against the whole test suite, uncovering untested logic. For instance, after installing slither, just run `slither-mutate path/to/contract/contract.sol-- test-cmd yarn test:unit` or the python invocation of vertigo (the "rs" means RareSkills, not Rust…)
105
105
106
106
### Pre and during internal review
107
107
108
108
-**Properties & Invariants Documentation**:
109
-
- A (short) brainstorm session *might* be scheduled, to finalize the list of invariants which will be studied
110
-
- An actor-based approach *might* be used, to keep the focus on higher level invariants
111
-
- The testing team lead *must* compile the properties in a table in `src/test/invariant/PROPERTIES.md`. This file must include at least three columns:
112
-
-**Id:** A sequential number.
113
-
-**Properties**: Clear, concise English descriptions.
114
-
- Covered: An empty check box
115
-
- A fourth column *might* be used, for classification purposes:
116
-
-**Type**: Categorise using [Certora's categories](https://github.com/Certora/Tutorials/blob/master/06.Lesson_ThinkingProperties/Categorizing_Properties.pdf):
117
-
- Valid state: The protocol can only be in valid states based on the spec.
118
-
- State transition: Protocol state changes follow specific orders or conditions.
119
-
- Variable transition: Protocol variables change in defined ways.
120
-
- High-level property: Properties of the system as a whole.
121
-
- Unit test: Target specific functions for detailed examination.
109
+
- A (short) brainstorm session *might* be scheduled, to finalize the list of invariants which will be studied
110
+
- An actor-based approach *might* be used, to keep the focus on higher level invariants
111
+
- The testing team lead *must* compile the properties in a table in `src/test/invariant/PROPERTIES.md`. This file must include at least three columns:
112
+
-**Id:** A sequential number.
113
+
-**Properties**: Clear, concise English descriptions.
114
+
- Covered: An empty check box
115
+
- A fourth column *might* be used, for classification purposes:
116
+
-**Type**: Categorise using [Certora's categories](https://github.com/Certora/Tutorials/blob/master/06.Lesson_ThinkingProperties/Categorizing_Properties.pdf):
117
+
- Valid state: The protocol can only be in valid states based on the spec.
118
+
- State transition: Protocol state changes follow specific orders or conditions.
119
+
- Variable transition: Protocol variables change in defined ways.
120
+
- High-level property: Properties of the system as a whole.
121
+
- Unit test: Target specific functions for detailed examination.
122
122
123
123
Example
124
124
@@ -129,13 +129,13 @@ Example
129
129
| 2 | Transfer between addresses shouldn't modify supply| High level |[]|
130
130
```
131
131
132
-
-**Invariant Implementation (QA/AT)**: The lead/advanced tester *must* implement the invariants starting from higher-level to lower-level properties in the `PROPERTIES.md` file. Complex protocols may require deployment tricks for Medusa.
133
-
-**Echnidna/Medusa (AT)**: *should* be used for stateful testing, using the coverage (HTML autogenerated) to discover additional properties.
132
+
-**Invariant Implementation (QA/AT)**: The lead/advanced tester *must* implement the invariants starting from higher-level to lower-level properties in the `PROPERTIES.md` file.
133
+
-**Forge (AT)**: *should* be used for stateful testing, using coverage guidance.
134
134
-**Kontrol / Halmos (AT)**: *might* be used for formal verification (symbolic execution or using K formalism)
135
135
- Any finished campaign *must* be run for at least a couple of weeks on the dedicated server
136
136
-**CI Workflow (AT)**:
137
-
- Medusa and Halmos/Kontrol *should* be in the CI workflow, once the advanced tests are finalised.
138
-
- Mutation tests *might* be used as testing suite validation tools, granted *the branch coverage is 100%.*
137
+
- Forge and Halmos/Kontrol *should* be in the CI workflow, once the advanced tests are finalised.
138
+
- Mutation tests *might* be used as testing suite validation tools, granted *the branch coverage is 100%.*
139
139
140
140
## Repo organisation: How do we keep sanity?
141
141
@@ -156,17 +156,15 @@ Example
156
156
├── PROPERTIES.md
157
157
├── fuzz/
158
158
| ├── handlers/
159
-
| | ├── HandlerA.sol
160
-
| | ├── HandlerAgents.sol
161
-
| | └── HandlerParent.sol
162
-
| ├── properties/
163
-
| | ├── PropertiesA.sol
164
-
| | └── PropertiesParent.sol
165
-
│ ├── FuzzTest.t.sol
166
-
│ ├── Reproducers.t.sol
159
+
| | ├── HandlerA.t.sol
160
+
| | ├── GhostState.t.sol
161
+
| | └── BaseHandler.t.sol
162
+
│ ├── Invariant.t.sol
163
+
│ ├── HandlersTarget.t.sol
164
+
│ ├── ExecutionPaths.t.sol
167
165
| └── Setup.sol
168
166
└── symbolic/
169
167
└── KontrolTest.t.sol
170
168
```
171
169
172
-
Contract name *should* reflect the test technique used, `contract UnitMyContract` for instance.
170
+
Contract name *should* reflect the test technique used, `contract UnitMyContract` for instance.
0 commit comments