Skip to content

Limit arena size in C++ tests - #2506

Open
bdice wants to merge 8 commits into
rapidsai:mainfrom
bdice:fix/gb300-arena-test-timeout
Open

Limit arena size in C++ tests#2506
bdice wants to merge 8 commits into
rapidsai:mainfrom
bdice:fix/gb300-arena-test-timeout

Conversation

@bdice

@bdice bdice commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Use an explicit 8 GiB arena in the C++ arena MR reference tests instead of the default half of available device memory. This prevents 116 fixture constructions from each reserving roughly 125 GiB on GB300 while preserving the capacity available on the original 16 GiB test target.

Measured on DGX Spark (128GB integrated memory), total test time for the arena MR reference tests decreased from 64.41 seconds to 11.88 seconds (81.6% reduction).

Closes #2505. Follows the Python test precedent in #2273.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Aug 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@bdice bdice added bug Something isn't working tests Related to unit tests ci non-breaking Non-breaking change labels Aug 10, 2026
@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Summary by CodeRabbit

  • Tests

    • Increased test arena capacity to 8 GiB for more extensive memory-resource testing.
    • Updated continuous integration to run the nightly test matrix for the GB300 GPU configuration.
    • Added diagnostics for kernel memory initialization and NVIDIA driver settings during C++ test runs.
  • Chores

    • Streamlined continuous integration by disabling selected build, test, packaging, and telemetry jobs.
    • Updated copyright and licensing notices to include NVIDIA affiliates.

Walkthrough

Arena memory resource tests now use an explicit 8 GiB capacity. The C++ nightly workflow targets GB300 and disables selected CI jobs. The C++ test script reports kernel and NVIDIA driver allocation settings.

Changes

Arena test sizing and CI workflow

Layer / File(s) Summary
Configure explicit arena capacity
cpp/tests/mr/mr_ref_arena_tests.cpp, cpp/tests/mr/mr_ref_test.hpp
Both arena test fixtures pass an explicit 8 GiB initial size. Copyright notices include NVIDIA affiliates.
Target GB300 nightly tests
.github/workflows/pr.yaml
The conda-cpp-tests job uses the nightly matrix and filters it to gb300. Selected CI jobs are disabled with if: false.
Report allocation settings
ci/test_cpp.sh
The script reports /proc/cmdline’s init_on_alloc setting and the NVIDIA driver’s Coherent parameter. It reports explicit unavailable messages when either value cannot be read.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • rapidsai/rmm#2468: Both PRs update arena memory-resource code and related tests, but that PR adds [[nodiscard]] allocation annotations.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The workflow disables multiple unrelated CI jobs, and ci/test_cpp.sh adds diagnostics not required by issue #2505. Remove unrelated workflow changes and CI diagnostics, or link them to separate issues with clear justification.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the primary change: limiting the arena size in C++ tests.
Description check ✅ Passed The description explains the explicit arena size, the performance issue, measured improvement, and linked issue.
Linked Issues check ✅ Passed The PR passes an explicit 8 GiB arena size to both arena test fixtures, addressing issue #2505 without increasing the timeout.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@bdice
bdice force-pushed the fix/gb300-arena-test-timeout branch from 9eb6018 to 221ee4e Compare August 10, 2026 04:21
@bdice

bdice commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/pr.yaml:
- Line 59: Replace each constant if: false condition in the workflow with an
appropriate feature flag, or remove the disabled jobs and corresponding
pr-builder.needs entries. Ensure no unconditional constant-false conditions
remain so actionlint passes.
- Around line 282-283: Update the GB300 nightly matrix entry by setting its
build_workflow_name input to pr.yaml, ensuring artifact lookup targets the C++
artifacts produced by conda-cpp-build in pr.yaml while preserving the existing
matrix_type and matrix_filter values.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1415b8aa-3e59-4a55-9e04-52f2c78ac5fc

📥 Commits

Reviewing files that changed from the base of the PR and between 9eb6018 and 221ee4e.

📒 Files selected for processing (1)
  • .github/workflows/pr.yaml

Comment thread .github/workflows/pr.yaml
Comment thread .github/workflows/pr.yaml

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@ci/test_cpp.sh`:
- Line 39: Update the `/proc/cmdline` check in `ci/test_cpp.sh` to validate
readability before searching for `init_on_alloc`. Report the setting as
unavailable with a meaningful error message when the file is missing or
unreadable, and retain the existing “not set” message only when a readable file
lacks the parameter.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6e520b46-c658-4e82-8933-30214a8f73f6

📥 Commits

Reviewing files that changed from the base of the PR and between 221ee4e and 20b3dd1.

📒 Files selected for processing (1)
  • ci/test_cpp.sh

Comment thread ci/test_cpp.sh Outdated
@bdice
bdice marked this pull request as ready for review August 10, 2026 04:57
@bdice
bdice requested review from a team as code owners August 10, 2026 04:57
@bdice

bdice commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator Author

I have some temporary testing changes for GB300 so that I can figure out why the tests are so slow. I think it's an issue with missing the init_on_alloc=0 config. I'm verifying that.

@wence- wence- left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving C++ changes, which seem reasonable independent of the runner setup issues.

@bdice
bdice force-pushed the fix/gb300-arena-test-timeout branch from 7fa0e5a to 0067d3e Compare August 10, 2026 12:09
@bdice bdice self-assigned this Aug 10, 2026
@bdice bdice moved this to Review in RMM Project Board Aug 10, 2026
@bdice bdice moved this from Review to In Progress in RMM Project Board Aug 10, 2026
@bdice
bdice force-pushed the fix/gb300-arena-test-timeout branch from e128f67 to 437beed Compare August 11, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working ci non-breaking Non-breaking change tests Related to unit tests

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

GB300 C++ nightly times out in arena memory resource tests

4 participants