Try abort pytest plugin

[gulsumgudukbay]

Motivation

Trying out pytest abort plugin

Technical Details

remove abort logic from test runners.

Test Plan

git clone https://github.com/ROCm/pytest-abort-plugin.git
cd pytest-abort-plugin
pip install pytest-xdist pytest-timeout
python3 -m pip install --upgrade build
python3 -m build
pip install ./dist/*.whl

export PYTEST_ABORT_CRASHED_TESTS_LOG="$PWD/logs/crashed_tests.jsonl"
python3 jax_rocm_plugin/build/rocm/run_single_gpu.py -c 2>&1 | tee JAX_SG_0.8.0_ut.log
export PYTEST_ABORT_CRASHED_TESTS_LOG="$PWD/logs/crashed_tests.jsonl"
python3 jax_rocm_plugin/build/rocm/run_multi_gpu.py -c 2>&1 | tee JAX_MG_0.8.0_ut.log

Test Result

All tests are shown in the log and final compiled report.

Single GPU:

FINAL TEST SUMMARY

Total Tests: 35287
Passed: 23013
Failed: 86
Skipped: 11960
Errors: 6
Crashed: 77

---

CRASHED TESTS (77):

• tests/lax_scipy_spectral_dac_test.py::LaxScipySpectralDacTest::test_spectral_dac_eigh4 (4.9s)
• tests/ode_test.py::ODETest::test_weird_time_pendulum_grads (2.9s)
• tests/pallas/gpu_ops_test.py::FusedAttentionInterpretTest::test_fused_attention_bwd2 (1.7s)
• tests/pallas/pallas_vmap_test.py::PallasCallVmapInterpretTest::test_small_large_vmap (0.3s)
• tests/pallas/pallas_vmap_test.py::PallasCallVmapInterpretTest::test_small_small_large_vmap (0.3s)
• tests/linalg_test.py::ScipyLinalgTest::testLuGrad9 (0.7s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=16_head_dim=64_kwargs={}_start_idx=123_kv_seq_len=None_return_residuals=False (0.6s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=16_head_dim=64_kwargs={}_start_idx=123_kv_seq_len=250_return_residuals=True (0.7s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=16_head_dim=64_kwargs={}_start_idx=None_kv_seq_len=250_return_residuals=False (0.7s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=16_head_dim=64_kwargs={}_start_idx=123_kv_seq_len=None_return_residuals=True (0.8s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=16_head_dim=64_kwargs={}_start_idx=None_kv_seq_len=250_return_residuals=True (0.9s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=4_head_dim=64_kwargs={}_start_idx=None_kv_seq_len=250_return_residuals=False (0.9s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=4_head_dim=64_kwargs={}_start_idx=None_kv_seq_len=250_return_residuals=True (0.6s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=16_head_dim=64_kwargs={}_start_idx=None_kv_seq_len=None_return_residuals=True (0.6s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=4_head_dim=64_kwargs={}_start_idx=123_kv_seq_len=None_return_residuals=True (0.9s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=32_num_kv_heads=32_head_dim=64_kwargs={}_start_idx=123_kv_seq_len=250_return_residuals=True (0.6s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=32_num_kv_heads=32_head_dim=64_kwargs={}_start_idx=123_kv_seq_len=250_return_residuals=False (0.7s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=4_head_dim=64_kwargs={}_start_idx=None_kv_seq_len=None_return_residuals=False (0.6s)
• tests/pallas/gpu_attention_test.py::DecodeAttentionInterpretTest::test_gqa_batch_size=1_seq_len=1024_num_q_heads=16_num_kv_heads=16_head_dim=64_kwargs={}_start_idx=None_kv_seq_len=None_return_residuals=False (0.7s)
• tests/lax_scipy_special_functions_test.py::LaxScipySpecialFunctionsTest::testScipySpecialFun_expn_s_s_int32_float32 (0.8s)
• tests/lax_scipy_special_functions_test.py::LaxScipySpecialFunctionsTest::testScipySpecialFun_gammaincc_s_s_float32_float32 (0.7s)
• tests/lax_scipy_special_functions_test.py::LaxScipySpecialFunctionsTest::testScipySpecialFun_gammainc_s_2x1x4_float32_float32 (1.1s)
• tests/scipy_stats_test.py::LaxBackedScipyStatsTests::testGammaLogSf7 (30.3s)
• tests/pallas/gpu_paged_attention_test.py::PagedAttentionInterpretTest::test_paged_attention2 (3581.3s)
• tests/pallas/gpu_paged_attention_test.py::PagedAttentionInterpretTest::test_paged_attention7 (0.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float16_bm_64_bn_64_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float32_bm_64_bn_128_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float16_bm_128_bn_64_bk_32 (0.6s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float16_bm_64_bn_64_bk_32 (0.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float16_bm_128_bn_64_bk_32 (3577.8s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float16_bm_64_bn_128_bk_32 (0.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float32_bm_64_bn_64_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float32_bm_128_bn_128_bk_32 (3581.8s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float16_bm_128_bn_128_bk_32 (3580.5s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float16_bm_64_bn_128_bk_32 (3580.2s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float32_bm_128_bn_64_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float16_bm_128_bn_128_bk_32 (3581.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float32_bm_128_bn_64_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float16_bm_128_bn_64_bk_32_gm_8 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float16_bm_64_bn_128_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float16_bm_64_bn_64_bk_32 (3580.8s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float32_bm_128_bn_128_bk_32_gm_8 (0.5s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float32_bm_64_bn_128_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float32_bm_64_bn_64_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_512_k_512_dtype_float16_bm_128_bn_128_bk_32 (3580.6s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_512_k_512_dtype_float16_bm_128_bn_64_bk_32 (3580.8s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float32_bm_128_bn_64_bk_32 (1.5s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_512_k_512_dtype_float32_bm_128_bn_128_bk_32 (1.0s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_512_k_512_dtype_float16_bm_64_bn_64_bk_32 (0.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_512_k_512_dtype_float32_bm_64_bn_128_bk_32 (3573.8s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float16_bm_64_bn_128_bk_32_gm_8 (3580.0s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_512_k_512_dtype_float32_bm_64_bn_64_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float16_bm_64_bn_64_bk_32_gm_8 (3580.2s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float32_bm_64_bn_128_bk_32 (3581.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float16_bm_128_bn_64_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_512_k_512_dtype_float16_bm_64_bn_128_bk_32 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float32_bm_64_bn_64_bk_32 (3581.6s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float32_bm_64_bn_64_bk_32_gm_8 (3580.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_1024_k_512_dtype_float32_bm_128_bn_128_bk_32 (3581.8s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_512_k_512_dtype_float32_bm_128_bn_128_bk_32_gm_8 (3580.1s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float32_bm_64_bn_128_bk_32_gm_8 (1.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_512_k_512_dtype_float16_bm_64_bn_64_bk_32_gm_8 (2.3s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float32_bm_128_bn_64_bk_32_gm_8 (1.1s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_512_k_512_dtype_float32_bm_64_bn_64_bk_32_gm_8 (1.0s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_512_k_512_dtype_float16_bm_128_bn_64_bk_32_gm_8 (3543.0s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_512_k_512_dtype_float32_bm_64_bn_128_bk_32_gm_8 (3579.5s)
• tests/pallas/pallas_test.py::PallasCallTest::test_split2 (0.6s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_1024_n_1024_k_512_dtype_float16_bm_128_bn_128_bk_32 (0.9s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_512_n_1024_k_512_dtype_float16_bm_128_bn_64_bk_32_gm_8 (3562.6s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_512_k_512_dtype_float32_bm_128_bn_64_bk_32_gm_8 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_512_n_1024_k_512_dtype_float32_bm_128_bn_64_bk_32_gm_8 (0.4s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_1024_n_1024_k_512_dtype_float16_bm_128_bn_128_bk_32_gm_8 (3580.5s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_512_n_512_k_512_dtype_float16_bm_128_bn_64_bk_32_gm_8 (3578.9s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_512_n_1024_k_512_dtype_float16_bm_64_bn_64_bk_32_gm_8 (0.7s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_block_spec_m_512_n_512_k_512_dtype_float32_bm_128_bn_64_bk_32 (0.5s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_512_n_512_k_512_dtype_float16_bm_64_bn_64_bk_32_gm_8 (3578.5s)
• tests/pallas/pallas_test.py::PallasCallInterpretTest::test_matmul_m_512_n_1024_k_512_dtype_float16_bm_64_bn_128_bk_32_gm_8 (0.5s)
  ======================================================================

Multi-GPU:

FINAL TEST SUMMARY

Total Tests: 51338
Passed: 30247
Failed: 212
Skipped: 20722
Errors: 6
Crashed: 1

---

CRASHED TESTS (1):

• tests/lax_control_flow_test.py::LaxControlFlowTest::testScanJVP_jit_scan=True_jit_f=True_impl=unroll2 (1.8s)

---

FAILED TEST FILES (1):

• tests/lax_control_flow_test.py (exit code: 1)
  ======================================================================

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.