Skip to content

Rework expected results config files in ONNX ops test suite #25

Description

@ScottTodd

See the previous issue for this at nod-ai/AMD-SHARK-TestSuite#253

Config files for the ONNX operator tests use this schema:

# List of configuration files following this schema:
# {
# "config_name": str,
# "iree_compile_flags": list of str,
# "iree_run_module_flags": list of str,
# "skip_compile_tests": list of str,
# "skip_run_tests": list of str,
# "expected_compile_failures": list of str,
# "expected_run_failures": list of str
# }
#
# For example, to test on CPU with the `llvm-cpu` backend and `local-task` device:
# {
# "config_name": "cpu_llvm_task",
# "iree_compile_flags": ["--iree-hal-target-backends=llvm-cpu"],
# "iree_run_module_flags": ["--device=local-task"],
# "skip_compile_tests": [],
# "skip_run_tests": [],
# "expected_compile_failures": ["test_abs"],
# "expected_run_failures": ["test_add"],
# }
#
# The list of files can be specified in (by order of preference):
# 1. The `--config-files` argument
# e.g. `pytest ... --config-files foo.json bar.json`
# 2. The `IREE_TEST_CONFIG_FILES` environment variable
# e.g. `set IREE_TEST_CONFIG_FILES=foo.json;bar.json`
# 3. A default config file used for testing the test suite itself

(aside: that schema could be encoded in a file for validation/reference, rather than just included in a comment)

Right now test cases are included in one of these lists or not mentioned at all

  • skip_compile_tests
  • skip_run_tests
  • expected_compile_failures
  • expected_run_failures

While this lets us add new tests without needing to update existing files, it doesn't make it clear how many tests are included and which are passing.

Now that test results can be automatically reflected back into config files using https://github.com/iree-org/iree-test-suites/blob/main/onnx_ops/update_config_xfails.py, we could for example

A) also list passing tests:

"passing_tests": [
  "test_abs",
]

B) list test statuses directly:

"tests": {
  "test_abs": "pass",
  "test_add": "skip_compile",
  "test_div": "skip_run",
  "test_mul": "fail_compile",
  "test_sub": "fail_run",
}

I like option B, and I've started in a similar direction with #23. That has a single test function per model that runs all stages (import, compile, run). Tests set their expected result using for example @pytest.mark.xfail(raises=IreeRunException) or @pytest.mark.xfail(raises=IreeCompileException)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions