These two locations have full model integration tests:
Those tests are both currently run as part of the https://github.com/iree-org/iree/blob/main/.github/workflows/pkgci_regression_test.yml workflow.
Together with #48, it should be clear where new tests should go in source code and within workflows in this repository and in iree-org/iree.
The documentation at https://iree.dev/developers/general/testing-guide/#external-test-suites should also be kept up to date.
File storage
Both of those test suites currently store files in the 'sharkpublic' Azure storage account.
- For some models we should be able to download from https://huggingface.co/models instead, though the import process into .mlir may involve using packages outside of iree-org like https://github.com/nod-ai/shark-ai.
- If the community is comfortable with files that need special permissions to maintain, we could continue using that storage account
- We could see if the Linux Foundation has storage we could use somehow, and grant access to that using iree-org GitHub membership or some other open process
Test flags
The tests at https://github.com/iree-org/iree/tree/main/experimental/regression_suite/shark-test-suite-models in particular have long lists of flags and hardware requirements since most model development activities have been focusing on getting great performance on a small number of devices (e.g. AMD MI300 GPUs). For a common test suite, I would prefer to use default or generic flags, not long lists of flags, transform scripts, or tuning specs.
These two locations have full model integration tests:
Those tests are both currently run as part of the https://github.com/iree-org/iree/blob/main/.github/workflows/pkgci_regression_test.yml workflow.
Together with #48, it should be clear where new tests should go in source code and within workflows in this repository and in iree-org/iree.
The documentation at https://iree.dev/developers/general/testing-guide/#external-test-suites should also be kept up to date.
File storage
Both of those test suites currently store files in the 'sharkpublic' Azure storage account.
Test flags
The tests at https://github.com/iree-org/iree/tree/main/experimental/regression_suite/shark-test-suite-models in particular have long lists of flags and hardware requirements since most model development activities have been focusing on getting great performance on a small number of devices (e.g. AMD MI300 GPUs). For a common test suite, I would prefer to use default or generic flags, not long lists of flags, transform scripts, or tuning specs.