CI tests silently failing (builder.sh)

[svuckovicTT]

Example job: https://github.com/tenstorrent/tt-mlir/actions/runs/19666699769/job/56326746477

Several errors appear throughout the job:

test_ttir_fusing.py::test_conv_activation_fusing[relu-1-dilation0-padding0-stride0-dtypes0-shapes0]
  /opt/ttmlir-toolchain/venv/lib/python3.11/site-packages/_pytest/runner.py:245: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
  Plugin: /__w/tt-mlir/tt-mlir/test/python/golden/conftest.py, Hook: pytest_runtest_call
  Failed: Unknown failure detected! Please address this or correctly throw a `TTBuilder*` exception instead if this is a compilation issue, runtime error, or golden mismatch. Exception: 'output_0':<class 'KeyError'>

Found matching TTNN file: ttir-builder-artifacts/ttnn/test_reduction_ops[ttnn-prod-None-False-f32-32x128x128]_ttnn.mlir.ttnn
2025-11-25 11:31:20,626 - WARNING - directory '/__w/tt-mlir/tt-mlir/ttrt-artifacts' already exists
2025-11-25 11:31:20,762 - ERROR - ERROR: test=ttir-builder-artifacts/ttnn/test_reduction_ops[ttnn-prod-None-False-f32-32x128x128]_ttnn.mlir.ttnn experienced an error with exception=buffer length (2 bytes) after offset (0 bytes) must be a multiple of element size (4)

Found matching TTNN file: ttir-builder-artifacts/ttnn/test_reshape[ttnn-ui8-0x32x128-0]_ttnn.mlir.ttnn
2025-11-25 11:40:32,944 - WARNING - directory '/__w/tt-mlir/tt-mlir/ttrt-artifacts' already exists
                 Always |  WARNING | getTensorDataBuffer: Tensor has zero volume; returning an empty data vector.
2025-11-25 11:40:33,056 - ERROR - ERROR: test=ttir-builder-artifacts/ttnn/test_reshape[ttnn-ui8-0x32x128-0]_ttnn.mlir.ttnn experienced an error with exception=both buffer length (0) and count (-1) must not be 0
2025-11-25 11:40:33,058 - WARNING - no binaries found to run - returning early
2025-11-25 11:40:33,060 - ERROR - ERROR: test case=ttir-builder-artifacts/ttnn/test_reshape[ttnn-ui8-0x32x128-0]_ttnn.mlir.ttnn

Found matching TTNN file: ttir-builder-artifacts/ttnn/test_ternary_ops[where-ttnn-f32-128x128]_ttnn.mlir.ttnn
2025-11-25 11:44:21,164 - WARNING - directory '/__w/tt-mlir/tt-mlir/ttrt-artifacts' already exists
                 Always |  WARNING | getting output tensor is not supported for DeallocateOp
                 Always |  WARNING | getting output tensor is not supported for DeallocateOp
                 Always |  WARNING | getting output tensor is not supported for DeallocateOp
2025-11-25 11:44:21,523 - ERROR - ERROR: test=ttir-builder-artifacts/ttnn/test_ternary_ops[where-ttnn-f32-128x128]_ttnn.mlir.ttnn experienced an error with exception=Failed: program-level output golden comparison failed, actual_pcc=0.49734237090912065 < expected_pcc=0.99
2025-11-25 11:44:21,525 - WARNING - no binaries found to run - returning early
2025-11-25 11:44:21,526 - ERROR - ERROR: test case=ttir-builder-artifacts/ttnn/test_ternary_ops[where-ttnn-f32-128x128]_ttnn.mlir.ttnn

[svuckovicTT]

@tapspatel @nsumrakTT

Hey folks, I saw some errors in logs, wanted to check with you if these were expected. Not sure if things are silently failing or not. Please lmk if you think someone else should be looking into this.

[nsumrakTT]

I've added @ctodTT as these are ttrt runs after builder tests.
However, I think ttrt should handle errors much better - the first thing is to exit with non-zero code after error happens, and the second would be to output exceptions with better non copy-paste messages (eg. grep "experienced an error with exception" - 7 matches, "ERROR: test=", 8 matches).

[tapspatel]

@ctodTT can you take a look?