Failing tests for `check-torch-mlir`

[mdehling]

Both with in-tree or out-of-tree builds I run into the following issue of failing tests.

Is this a problem with my build setup?

build (main)$ ninja check-torch-mlir
[0/1] Running the torch-mlir regression tests
Enabling sparsity propagation tests
Enabling Torch v2.3+ tests
FAIL: TORCH_MLIR :: python/compile.py (99 of 110)
******************** TEST 'TORCH_MLIR :: python/compile.py' FAILED ********************
Exit Code: 1

Command Output (stderr):
--
RUN: at line 1: /agicl-dev/m.dehling/torch-mlir/mlir_venv/bin/python3.10 -s /agicl-dev/m.dehling/torch-mlir/test/python/compile.py 2>&1 | FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/compile.py
+ /agicl-dev/m.dehling/torch-mlir/mlir_venv/bin/python3.10 -s /agicl-dev/m.dehling/torch-mlir/test/python/compile.py
+ FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/compile.py
/agicl-dev/m.dehling/torch-mlir/test/python/compile.py:26:16: error: CHECK-LABEL: expected string not found in input
# CHECK-LABEL: TEST: test_enable_ir_printing
               ^
<stdin>:1:1: note: scanning from here
Traceback (most recent call last):
^
<stdin>:1:11: note: possible intended match here
Traceback (most recent call last):
          ^

Input file: <stdin>
Check file: /agicl-dev/m.dehling/torch-mlir/test/python/compile.py

-dump-input=help explains the following input dump.

Input was:
<<<<<<
            1: Traceback (most recent call last):
label:26'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
label:26'1               ?                         possible intended match
            2:  File "/agicl-dev/m.dehling/torch-mlir/test/python/compile.py", line 6, in <module>
label:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            3:  from torch_mlir import torchscript
label:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
            4: ImportError: cannot import name 'torchscript' from 'torch_mlir' (unknown location)
label:26'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>

--

********************
FAIL: TORCH_MLIR :: python/fx_importer/sparsity/sparse_test.py (100 of 110)
******************** TEST 'TORCH_MLIR :: python/fx_importer/sparsity/sparse_test.py' FAILED ********************
Exit Code: 2

Command Output (stderr):
--
RUN: at line 6: /agicl-dev/m.dehling/torch-mlir/mlir_venv/bin/python3.10 /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py | FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py
+ FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py
+ /agicl-dev/m.dehling/torch-mlir/mlir_venv/bin/python3.10 /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py
Traceback (most recent call last):
  File "/agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py", line 19, in <module>
    from torch_mlir_e2e_test.linalg_on_tensors_backends.refbackend import (
ModuleNotFoundError: No module named 'torch_mlir_e2e_test'
FileCheck error: '<stdin>' is empty.
FileCheck command line:  FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py

--

********************
********************
Failed Tests (2):
  TORCH_MLIR :: python/compile.py
  TORCH_MLIR :: python/fx_importer/sparsity/sparse_test.py


Testing Time: 7.76s

Total Discovered Tests: 110
  Passed: 108 (98.18%)
  Failed:   2 (1.82%)
FAILED: tools/torch-mlir/test/CMakeFiles/check-torch-mlir /agicl-dev/m.dehling/torch-mlir/build/tools/torch-mlir/test/CMakeFiles/check-torch-mlir
cd /agicl-dev/m.dehling/torch-mlir/build/tools/torch-mlir/test && /agicl-dev/m.dehling/torch-mlir/mlir_venv/bin/python3.10 /agicl-dev/m.dehling/torch-mlir/build/./bin/llvm-lit -sv /agicl-dev/m.dehling/torch-mlir/build/tools/torch-mlir/test
ninja: build stopped: subcommand failed.
build (main)$

[yemyhdtrc6088]

hi,do you resolve this problem?
ImportError: cannot import name 'torchscript' from 'torch_mlir' (unknown location)
I find torch-mlir/python/torch_mlir dont have torchscript.py like fx.py

[mdehling]

No solution from me; I found the same. I think the tests are not up-to-date.

[yemyhdtrc6088]

No solution from me; I found the same. I think the tests are not up-to-date.

I changed the compile commands,and can import torchscript
here is my compile command:
cmake -GNinja -Bbuild
-DCMAKE_BUILD_TYPE=Debug
-DLLVM_ENABLE_ASSERTIONS=ON
-DPython3_FIND_VIRTUALENV=ONLY
-DLLVM_ENABLE_PROJECTS=mlir
-DLLVM_EXTERNAL_PROJECTS="torch-mlir"
-DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD"
-DMLIR_ENABLE_BINDINGS_PYTHON=ON
-DLLVM_TARGETS_TO_BUILD=host
-DTORCH_MLIR_ENABLE_LTC=ON
-DTORCH_MLIR_ENABLE_PYTORCH_EXTENSIONS=ON
-DTORCH_MLIR_ENABLE_JIT_IR_IMPORTER=ON
externals/llvm-project/llvm

[mdehling]

Thank you, I'll give it a try.

[vivekkhandelwal1]

@mdehling Are the tests still failing for you?

[mdehling]

@mdehling Are the tests still failing for you?

@yemyhdtrc6088 's compile command fixes one of them, but sparse_test.py still fails. Just tested with the latest main branch.

$ cmake -GNinja -Bbuild -DCMAKE_BUILD_TYPE=Debug -DLLVM_ENABLE_ASSERTIONS=ON -DPython3_FIND_VIRTUALENV=ONLY -DLLVM_ENABLE_PROJECTS=mlir -DLLVM_EXTERNAL_PROJECTS="torch-mlir" -DLLVM_EXTERNAL_TORCH_MLIR_SOURCE_DIR="$PWD" -DMLIR_ENABLE_BINDINGS_PYTHON=ON -DLLVM_TARGETS_TO_BUILD=host -DTORCH_MLIR_ENABLE_LTC=ON -DTORCH_MLIR_ENABLE_PYTORCH_EXTENSIONS=ON -DTORCH_MLIR_ENABLE_JIT_IR_IMPORTER=ON externals/llvm-project/llvm -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
$ cmake --build build --target check-torch-mlir
[...]
Enabling Torch v2.3+ tests
Enabling sparsity propagation tests
FAIL: TORCH_MLIR :: python/fx_importer/sparsity/sparse_test.py (99 of 115)
******************** TEST 'TORCH_MLIR :: python/fx_importer/sparsity/sparse_test.py' FAILED ********************
Exit Code: 2

Command Output (stderr):
--
/agicl-dev/m.dehling/torch-mlir/venv/bin/python3.12 /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py | FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py # RUN: at line 6
+ /agicl-dev/m.dehling/torch-mlir/venv/bin/python3.12 /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py
+ FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py
Traceback (most recent call last):
  File "/agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py", line 19, in <module>
    from torch_mlir_e2e_test.linalg_on_tensors_backends.refbackend import (
ModuleNotFoundError: No module named 'torch_mlir_e2e_test.linalg_on_tensors_backends.refbackend'
FileCheck error: '<stdin>' is empty.
FileCheck command line:  FileCheck /agicl-dev/m.dehling/torch-mlir/test/python/fx_importer/sparsity/sparse_test.py

--

********************
********************
Failed Tests (1):
  TORCH_MLIR :: python/fx_importer/sparsity/sparse_test.py


Testing Time: 78.29s

Total Discovered Tests: 115
  Passed: 114 (99.13%)
  Failed:   1 (0.87%)
[...]

[vivekkhandelwal1]

Can you please share the Torch version used by you?