Skip to content

test(Lux): migrate to ParallelTestRunner#1656

Merged
avik-pal merged 4 commits intomainfrom
ap/lux_migrate_to_paralleltestrunner
Feb 3, 2026
Merged

test(Lux): migrate to ParallelTestRunner#1656
avik-pal merged 4 commits intomainfrom
ap/lux_migrate_to_paralleltestrunner

Conversation

@avik-pal
Copy link
Member

@avik-pal avik-pal commented Feb 3, 2026

No description provided.

@gemini-code-assist
Copy link

Summary of Changes

Hello @avik-pal, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request undertakes a comprehensive refactoring of the project's testing framework. The core change involves transitioning from ReTestItems to ParallelTestRunner, which necessitated updates across the CI configuration, test dependencies, and the entire test suite's file structure. This initiative aims to enhance test execution efficiency, improve the organization of test cases, and streamline the overall development workflow.

Highlights

  • Test Runner Migration: The testing infrastructure has been migrated from ReTestItems to ParallelTestRunner, which significantly changes how tests are executed and managed.
  • Buildkite CI Configuration Update: The .buildkite/testing.yml file has been updated to reflect the new test runner, including changes to test_args format and reorganization of test groups for CUDA backend testing.
  • Test File Restructuring: Test files have been extensively reorganized into more granular and logical subdirectories (e.g., test/layers/ to test/core_layers/, test/contrib/ to test/misc/contrib/, and test/autodiff/ tests split into specific files), improving modularity and clarity.
  • Dependency Updates: The test/Project.toml file now includes AMDGPU, CUDA, LuxCUDA, ParallelTestRunner, and cuDNN as dependencies, while InteractiveUtils and ReTestItems have been removed, aligning with the new testing setup and expanded GPU support.
  • README Documentation Update: The README.md has been revised to provide updated instructions for running tests, deprecating the LUX_TEST_GROUP environment variable and ReTestItems.runtests in favor of Pkg.test with test_args.
  • Reactant Test Setup Consolidation: A new test/reactant_testsetup.jl file centralizes helper functions for Reactant-related tests, and existing Reactant tests have been refactored to utilize this shared setup, removing redundant code.
Changelog
  • .buildkite/testing.yml
    • Updated test_args format for JuliaCI/julia-test#v1 to use --BACKEND_GROUP=CUDA.
    • Reordered and changed test groups in the matrix.group section, adding autodiff, contrib, helpers, others and removing/renaming normalize_layers, extras.
    • Updated test_args format for commented-out AMDGPU tests.
  • README.md
    • Removed instructions for using LUX_TEST_GROUP environment variable and ReTestItems.runtests.
    • Added new instructions for running specific test files and test groups using Pkg.test(test_args=[...]).
  • test/Project.toml
    • Added AMDGPU, CUDA, LuxCUDA, ParallelTestRunner, and cuDNN to dependencies.
    • Removed InteractiveUtils and ReTestItems from dependencies.
  • test/autodiff/batched_jacobian_tests.jl
    • Added new file containing batched Jacobian tests.
  • test/autodiff/enzyme_tests.jl
    • Renamed from test/enzyme_tests.jl.
    • Refactored test structure to use @testset blocks instead of @testitem.
    • Removed the EnzymeTestSetup module.
  • test/autodiff/nested_autodiff_tests.jl
    • Removed file, its content was split into new, more specific files.
  • test/autodiff/nested_batched_jacobian_tests.jl
    • Renamed from test/autodiff/batched_autodiff_tests.jl.
    • Refactored test structure.
  • test/autodiff/nested_input_gradient_tests.jl
    • Added new file for nested AD input gradient/jacobian tests.
  • test/autodiff/nested_issue743_tests.jl
  • test/autodiff/nested_parameter_gradient_tests.jl
    • Added new file for nested AD parameter gradient/jacobian tests.
  • test/autodiff/nested_structured_matrix_tests.jl
    • Added new file for nested AD structured matrix tests.
  • test/autodiff/nested_vjp_jvp_tests.jl
    • Added new file for nested AD VJP & JVP tests.
  • test/core_layers/attention_tests.jl
    • Renamed from test/layers/attention_tests.jl.
    • Refactored test structure.
    • Removed Reactant-specific tests from this file.
  • test/core_layers/basic_tests.jl
    • Renamed from test/layers/basic_tests.jl.
    • Refactored test structure.
  • test/core_layers/containers_tests.jl
    • Renamed from test/layers/containers_tests.jl.
    • Refactored test structure.
    • Renamed ConcatLayer to ConcatLayerParallel and ConcatLayerBranch to avoid name clashes.
  • test/core_layers/conv_tests.jl
    • Renamed from test/layers/conv_tests.jl.
    • Refactored test structure.
  • test/core_layers/dropout_tests.jl
    • Renamed from test/layers/dropout_tests.jl.
    • Refactored test structure.
  • test/core_layers/embedding_tests.jl
    • Renamed from test/layers/embedding_tests.jl.
    • Refactored test structure.
    • Removed Reactant-specific tests from this file.
  • test/core_layers/normalize_tests.jl
    • Renamed from test/layers/normalize_tests.jl.
    • Refactored test structure.
  • test/core_layers/pooling_tests.jl
    • Renamed from test/layers/pooling_tests.jl.
    • Refactored test structure.
  • test/layers/recurrent_tests.jl
    • Removed file, its content was split into new files in test/recurrent_layers/.
  • test/misc/contrib/debug_tests.jl
    • Renamed from test/contrib/debug_tests.jl.
    • Refactored test structure.
  • test/misc/contrib/freeze_tests.jl
    • Renamed from test/contrib/freeze_tests.jl.
    • Refactored test structure.
  • test/misc/contrib/map_tests.jl
    • Renamed from test/contrib/map_tests.jl.
    • Refactored test structure.
  • test/misc/contrib/share_parameters_tests.jl
    • Renamed from test/contrib/share_parameters_tests.jl.
    • Refactored test structure.
  • test/misc/helpers/compact_tests.jl
    • Renamed from test/helpers/compact_tests.jl.
    • Refactored test structure.
  • test/misc/helpers/loss_tests.jl
    • Renamed from test/helpers/loss_tests.jl.
    • Refactored test structure.
    • Corrected to ŷ in some loss function tests.
  • test/misc/helpers/size_propagator_tests.jl
    • Renamed from test/helpers/size_propagator_tests.jl.
    • Refactored test structure.
  • test/misc/helpers/stateful_tests.jl
    • Renamed from test/helpers/stateful_tests.jl.
    • Refactored test structure.
  • test/misc/helpers/training_tests.jl
    • Renamed from test/helpers/training_tests.jl.
    • Refactored test structure.
  • test/misc/issue_tests.jl
    • Renamed from test/issue_tests.jl.
    • Refactored test structure.
  • test/misc/qa_tests.jl
    • Renamed from test/qa_tests.jl.
    • Refactored test structure.
  • test/misc/utils_tests.jl
    • Renamed from test/utils_tests.jl.
    • Refactored test structure.
  • test/reactant/alternate_precision_tests.jl
    • Added new file for Reactant alternate precision tests.
  • test/reactant/attention_tests.jl
    • Added new file for Reactant attention tests.
  • test/reactant/autodiff_tests.jl
    • Refactored to remove AMDGPU warnings and use @testset.
  • test/reactant/batchnorm_tests.jl
    • Added new file for Reactant BatchNorm tests.
  • test/reactant/dropout_tests.jl
    • Added new file for Reactant Dropout tests.
  • test/reactant/embedding_tests.jl
    • Added new file for Reactant embedding tests.
  • test/reactant/layer_tests.jl
    • Removed file, its content was split into new Reactant-specific test files.
  • test/reactant/loss_tests.jl
    • Refactored to remove AMDGPU warnings and use @testset.
    • Corrected to ŷ in some loss function tests.
  • test/reactant/recurrent_tests.jl
    • Added new file for Reactant recurrent layers tests.
  • test/reactant/rmsnorm_tests.jl
    • Added new file for Reactant RMSNorm tests.
  • test/reactant/tf_saved_model_tests.jl
    • Refactored to use @testset.
  • test/reactant/tracing_tests.jl
    • Refactored to use @testset.
  • test/reactant/training_tests.jl
    • Refactored to remove AMDGPU warnings and use @testset.
    • Corrected to ŷ in some training tests.
  • test/reactant_testsetup.jl
    • Added new file for Reactant test setup helper functions.
  • test/recurrent_layers/bidirectional_tests.jl
    • Added new file for bidirectional recurrent tests.
  • test/recurrent_layers/grucell_tests.jl
    • Added new file for GRUCell tests.
  • test/recurrent_layers/lstmcell_tests.jl
    • Added new file for LSTMCell tests.
  • test/recurrent_layers/recurrence_layer_tests.jl
    • Added new file for recurrence layer tests.
  • test/recurrent_layers/recurrent_issue_tests.jl
    • Added new file for recurrent layer issue tests.
  • test/recurrent_layers/rnncell_tests.jl
    • Added new file for RNNCell tests.
  • test/recurrent_layers/statefulrecurrentcell_tests.jl
    • Added new file for StatefulRecurrentCell tests.
  • test/runtests.jl
    • Replaced ReTestItems with ParallelTestRunner.
    • Updated argument parsing for test groups.
    • Modified environment variable settings for GPU memory and backend groups.
    • Updated logic for running integration and distributed tests based on parsed_args.positionals.
  • test/setup_modes.jl
    • Removed LUX_CURRENT_TEST_GROUP and cuda_reactant_testing logic.
    • Simplified MODES generation.
  • test/shared_testsetup.jl
    • Removed @testsetup module wrapper.
    • Removed @reexport for ComponentArrays, LuxCore, etc.
    • Removed SharedReactantLayersTestSetup module.
Ignored Files
  • Ignored by pattern: .github/workflows/** (1)
    • .github/workflows/CI.yml
Activity
  • The pull request was authored by avik-pal.
  • The changes involve a significant refactoring of the test suite, including migration to a new test runner, reorganization of test files into more logical categories, and updates to CI/CD configurations.
  • No explicit reviewer comments or progress updates were provided in the context.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a large-scale refactoring of the test suite, migrating from ReTestItems.jl to ParallelTestRunner.jl. The changes are extensive and well-executed, involving a significant restructuring of the test files into a more modular directory hierarchy, updating test definitions from @testitem to @testset, and rewriting the main test runner script. This improves the organization and maintainability of the test suite. I have one minor suggestion to update the documentation to reflect the new test groups.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 3, 2026

Benchmark Results (Julia v1.12)

Time benchmarks
main 9ebc339... main / 9ebc339...
basics/MHA 3.75 ± 0.25 μs 3.89 ± 0.23 μs 0.964 ± 0.086
basics/MHA (first run) 3.97 ± 0.34 μs 4.08 ± 0.33 μs 0.973 ± 0.12
basics/MHA reactant 0.0695 ± 0.0056 ms 0.0671 ± 0.008 ms 1.04 ± 0.15
basics/MHA reactant (comp + run) 0.204 ± 0.0045 s 0.206 ± 0.0045 s 0.994 ± 0.031
basics/conv 16.9 ± 24 μs 18.8 ± 27 μs 0.899 ± 1.8
basics/conv (first run) 18.7 ± 46 μs 18.8 ± 23 μs 0.998 ± 2.7
basics/conv reactant 0.0537 ± 0.0021 ms 0.0549 ± 0.0035 ms 0.979 ± 0.074
basics/conv reactant (comp + run) 0.147 ± 0.0024 s 0.147 ± 0.0022 s 1 ± 0.022
basics/dense 0.19 ± 0.01 μs 0.19 ± 0.001 μs 1 ± 0.053
basics/dense (first run) 0.2 ± 0.01 μs 0.2 ± 0.011 μs 1 ± 0.074
basics/dense reactant 0.0497 ± 0.0023 ms 0.0497 ± 0.0018 ms 1 ± 0.058
basics/dense reactant (comp + run) 0.119 ± 0.0023 s 0.12 ± 0.0044 s 0.992 ± 0.041
time_to_load 0.608 ± 0.0026 s 0.613 ± 0.0041 s 0.992 ± 0.0078
Memory benchmarks
main 9ebc339... main / 9ebc339...
basics/MHA 0.085 k allocs: 5.75 kB 0.087 k allocs: 5.81 kB 0.989
basics/MHA (first run) 0.085 k allocs: 5.75 kB 0.087 k allocs: 5.81 kB 0.989
basics/MHA reactant 19 allocs: 0.578 kB 19 allocs: 0.578 kB 1
basics/MHA reactant (comp + run) 18.5 k allocs: 1.84 MB 18.5 k allocs: 1.84 MB 1
basics/conv 0.039 k allocs: 4.55 kB 0.039 k allocs: 4.55 kB 1
basics/conv (first run) 0.039 k allocs: 4.55 kB 0.039 k allocs: 4.55 kB 1
basics/conv reactant 15 allocs: 0.438 kB 15 allocs: 0.438 kB 1
basics/conv reactant (comp + run) 6.76 k allocs: 1.3 MB 6.76 k allocs: 1.3 MB 1
basics/dense 2 allocs: 0.109 kB 2 allocs: 0.109 kB 1
basics/dense (first run) 2 allocs: 0.109 kB 2 allocs: 0.109 kB 1
basics/dense reactant 15 allocs: 0.422 kB 15 allocs: 0.422 kB 1
basics/dense reactant (comp + run) 6.18 k allocs: 1.27 MB 6.18 k allocs: 1.27 MB 1
time_to_load 0.145 k allocs: 11 kB 0.145 k allocs: 11 kB 1

@avik-pal avik-pal force-pushed the ap/lux_migrate_to_paralleltestrunner branch from a92eb19 to a0dc0a7 Compare February 3, 2026 13:29
@avik-pal avik-pal force-pushed the ap/lux_migrate_to_paralleltestrunner branch from a0dc0a7 to 8826220 Compare February 3, 2026 13:31
@codecov
Copy link

codecov bot commented Feb 3, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 2.11%. Comparing base (fa3c732) to head (9ebc339).

❗ There is a different number of reports uploaded between BASE (fa3c732) and HEAD (9ebc339). Click for more details.

HEAD has 49 uploads less than BASE
Flag BASE (fa3c732) HEAD (9ebc339)
50 1
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #1656       +/-   ##
==========================================
- Coverage   75.96%   2.11%   -73.86%     
==========================================
  Files         175     163       -12     
  Lines        7190    7000      -190     
==========================================
- Hits         5462     148     -5314     
- Misses       1728    6852     +5124     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@avik-pal avik-pal merged commit 61ba8ad into main Feb 3, 2026
70 of 73 checks passed
@avik-pal avik-pal deleted the ap/lux_migrate_to_paralleltestrunner branch February 3, 2026 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant