Skip to content

Commit b6381db

Browse files
vbrkicTTvbrkicTT
committed
RGG implementation
RGG implementation RGG documentation Reformat RGG code Removed global compiler config DepricatedVerifyConfig Use common forge verification Move random seed logic to test_graphs Rename PyBuda to Forge Single RGG pytest test_commands for rgg No reducing microbatch size for forge and torch constants Store test code in pytest report Operator repo update Operator support for RGG classified
1 parent 3cf50ae commit b6381db

20 files changed

+3061
-1
lines changed

‎forge/forge/op_repo/forge_operators.py‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,19 @@
3838
OperatorParamNumber("max", float, 0, 100),
3939
],
4040
),
41+
OperatorDefinition("atan", "forge.op.Atan", 1),
4142
OperatorDefinition("sine", "forge.op.Sine", 1),
4243
OperatorDefinition("cosine", "forge.op.Cosine", 1),
4344
OperatorDefinition("abs", "forge.op.Abs", 1),
4445
OperatorDefinition("tanh", "forge.op.Tanh", 1),
45-
OperatorDefinition("cumsum", "forge.op.CumSum", 1),
46+
OperatorDefinition(
47+
"cumsum",
48+
"forge.op.CumSum",
49+
1,
50+
forward_params=[
51+
OperatorParamNumber("dim", int, -3, 3),
52+
],
53+
),
4654
OperatorDefinition("argmax", "forge.op.Argmax", 1),
4755
OperatorDefinition("logical_not", "forge.op.LogicalNot", 1),
4856
OperatorDefinition("dropout", "forge.op.Dropout", 1),

‎forge/test/random/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# SPDX-FileCopyrightText: © 2024 Tenstorrent AI ULC
2+
3+
# SPDX-License-Identifier: Apache-2.0

‎forge/test/random/rgg/README.md‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
## RGG Random Graph Generator
2+
3+
Random Graph Generator supports testing of randomly generated graphs. Tests based on RGG can be triggered as regular pytests and executed in a single run of pytest. Tests are performed as a bulk operation for the specified number of tests.
4+
5+
### Execution
6+
7+
For each random model RGG passes through steps:
8+
9+
- Generate random model for specific random_seed
10+
- Verify model via verify_module
11+
12+
Source code of each randomly generated model with a pytest function can be automatically stored in a folder, ex `test/random_tests/` if configured.
13+
14+
## Run
15+
16+
Entrypoint for RGG pytests is in `test_graphs.py` module
17+
18+
Example command for running Forge RGG tests generated via random graph algorithm
19+
20+
```shell
21+
LOGURU_LEVEL=DEBUG FRAMEWORKS=FORGE RANDOM_TEST_SEED=0 RANDOM_TEST_COUNT=5 VERIFICATION_TIMEOUT=60 MIN_DIM=3 MAX_DIM=4 MIN_OP_SIZE_PER_DIM=16 MAX_OP_SIZE_PER_DIM=64 OP_SIZE_QUANTIZATION=1 MIN_MICROBATCH_SIZE=1 MAX_MICROBATCH_SIZE=8 NUM_OF_NODES_MIN=5 NUM_OF_NODES_MAX=10 NUM_OF_FORK_JOINS_MAX=5 CONSTANT_INPUT_RATE=20 SAME_INPUTS_PERCENT_LIMIT=10 pytest -svv forge/test/random/test_graphs.py
22+
```
23+
24+
Example command for running PyTorch RGG tests generated via random graph algorithm
25+
26+
```shell
27+
LOGURU_LEVEL=DEBUG FRAMEWORKS=PYTORCH RANDOM_TEST_SEED=0 RANDOM_TEST_COUNT=5 VERIFICATION_TIMEOUT=60 MIN_DIM=4 MAX_DIM=4 MIN_OP_SIZE_PER_DIM=4 MAX_OP_SIZE_PER_DIM=8 OP_SIZE_QUANTIZATION=1 MIN_MICROBATCH_SIZE=1 MAX_MICROBATCH_SIZE=1 NUM_OF_NODES_MIN=3 NUM_OF_NODES_MAX=5 NUM_OF_FORK_JOINS_MAX=5 CONSTANT_INPUT_RATE=20 SAME_INPUTS_PERCENT_LIMIT=10 pytest -svv forge/test/random/test_graphs.py
28+
```
29+
30+
31+
## Helper scripts
32+
33+
To start interacting with RGG tests load helper commands via
34+
35+
```sh
36+
source forge/test/random/test_commands.sh
37+
```
38+
39+
Available commands
40+
41+
| Command | Description |
42+
| --------------------- | --------------------------------------------------------------------- |
43+
| `print_help` | Print commands and current query parameters. |
44+
| `print_query_docs` | Print docs for all available query parameters. |
45+
| `print_params` | Print current query parameters values. |
46+
| `select_test_graphs` | Select test_graps pytest function. |
47+
| `pytest` | Run all tests specfied via a query parameters. |
48+
| `with-params pytest` | Print params before and after test run. |
49+
50+
Usage examples
51+
52+
Run 5 tests for PYTHORCH framework
53+
54+
```sh
55+
RANDOM_TEST_COUNT=5 FRAMEWORKS=PYTORCH with-params pytest
56+
```
57+
58+
59+
## Configuration
60+
61+
Configuration of RGG is supported via `RandomizerConfig`
62+
63+
Parameters includes configuration of:
64+
65+
- framework
66+
- number of tests
67+
- number of nodes
68+
- min and max size of an operand dimension
69+
- ...
70+
71+
For more details about configuration please take a look at `forge/test/random/rgg/config.py`.
72+
73+
Please refer to full list of supported enviroment variables in [README.debug.md](../README.debug.md)
74+
75+
*Test specific environment variables that can be used to fine tune default behavior of Forge RGG tests.*
76+
77+
## Parameters
78+
79+
Full list of supported query parameters
80+
81+
| Parameter | Description | Supported values | Default |
82+
| ----------------------------- | ----------------------------------------------------------------- | ----------------------------- | --------------------- |
83+
| FRAMEWORKS | List of frameworks. | PYTORCH, FORGE | |
84+
| ALGORITHMS | List of algorithms. | RANDOM | RANDOM |
85+
| CONFIGS | List of config names. | DEFAULT, UNSTABLE, ... | DEFAULT |
86+
| RANDOM\_TEST\_SEED | Initial seed for RGG. | | 0 |
87+
| RANDOM\_TEST\_COUNT | Number of random tests to be generated and executed. The parameter generate test_index in range from 0 to RANDOM\_TEST\_COUNT-1. | | 5 |
88+
| RANDOM\_TESTS\_SELECTED | Limiting random tests to only selected subset defined as comma separated list of test indexes. E.x. "3,4,6" | | Default is no limitation if not specified or empty. |
89+
| VERIFICATION\_TIMEOUT | Limit time for inference verification in seconds. | | 60 |
90+
| MIN\_DIM | Minimal number of dimensions of input tensors. | | 3 |
91+
| MAX\_DIM | Maximum number of dimensions of input tensors. | | 4 |
92+
| MIN\_OP\_SIZE\_PER\_DIM | Minimal size of an operand dimension. | | 16 |
93+
| MAX\_OP\_SIZE\_PER\_DIM | Maximum size of an operand dimension. Smaller operand size results in fewer failed tests. | | 512 |
94+
| OP\_SIZE\_QUANTIZATION | Quantization factor for operand size. | | 1 |
95+
| MIN_MICROBATCH_SIZE | Minimal size of microbatch of an input tensor. | | 1 |
96+
| MAX_MICROBATCH_SIZE | Maximum size of microbatch of an input tensor. | | 8 |
97+
| NUM\_OF\_NODES\_MIN | Minimal number of nodes to be generated by RGG. | | 5 |
98+
| NUM\_OF\_NODES\_MAX | Maximum number of nodes to be generated by RGG. | | 10 |
99+
| NUM\_OF\_FORK\_JOINS\_MAX | Maximum number of fork joins to be generated by random graph algorithm in RGG. | | 50 |
100+
| CONSTANT\_INPUT\_RATE | Rate of constant inputs in RGG in percents. | | 50 |
101+
| SAME\_INPUTS\_PERCENT\_LIMIT | Percent limit of nodes which have same value on multiple inputes. | | 10 |
102+
103+
104+
Test configuration parameters
105+
106+
| Parameter | Description |
107+
| ------------------------- | --------------------------------------------------------------------------------------------- |
108+
| SKIP_FORGE_VERIFICATION | Skip Forge model verification including model compiling and inference |
109+
110+
111+
## Development
112+
113+
Entrypoint for RGG impplementation is `process_test` module
114+
115+
Parameters of process_test pytest:
116+
117+
- test_index - index of a test
118+
- random_seed - random seed of a test
119+
- test_device - target test device
120+
- randomizer_config - test configation parameters
121+
- graph_builder_type - algorithm
122+
- framework - target framework

‎forge/test/random/rgg/__init__.py‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# SPDX-FileCopyrightText: © 2024 Tenstorrent AI ULC
2+
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
6+
from .datatypes import TensorShape
7+
from .datatypes import RandomizerConstantNode
8+
from .datatypes import (
9+
RandomizerInputNode,
10+
RandomizerNode,
11+
ExecutionContext,
12+
RandomizerParameters,
13+
RandomizerGraph,
14+
RandomizerConfig,
15+
)
16+
from .datatypes import NodeShapeCalculationContext
17+
from .datatypes import RandomizerTestContext
18+
from .datatypes import ModelBuilder, Framework
19+
from .datatypes import Algorithm
20+
from .config import get_randomizer_config_default
21+
from .utils import StrUtils, GraphUtils
22+
from .utils import DebugUtils
23+
from .base import GraphBuilder
24+
from .base import RandomizerRunner, RandomizerCodeGenerator, process_test
25+
from .frameworks import Frameworks
26+
from .frameworks import FrameworkTestUtils
27+
from .algorithms import Algorithms
28+
from .algorithms import GraphNodeSetup
29+
from .algorithms import RandomGraphAlgorithm
30+
31+
__all__ = [
32+
"TensorShape",
33+
"RandomizerConstantNode",
34+
"RandomizerInputNode",
35+
"RandomizerNode",
36+
"ExecutionContext",
37+
"RandomizerParameters",
38+
"RandomizerGraph",
39+
"RandomizerConfig",
40+
"NodeShapeCalculationContext",
41+
"RandomizerTestContext",
42+
"ModelBuilder",
43+
"Framework",
44+
"get_randomizer_config_default",
45+
"StrUtils",
46+
"GraphUtils",
47+
"DebugUtils",
48+
"Framework",
49+
"Algorithm",
50+
"GraphBuilder",
51+
"RandomizerRunner",
52+
"RandomizerCodeGenerator",
53+
"process_test",
54+
"Frameworks",
55+
"FrameworkTestUtils",
56+
"Algorithms",
57+
"GraphNodeSetup",
58+
"RandomGraphAlgorithm",
59+
]

0 commit comments

Comments
 (0)