Skip to content

feat: add tests to validate EFA#621

Merged
mselim00 merged 1 commit into
aws:mainfrom
mselim00:efa-test
Jun 2, 2025
Merged

feat: add tests to validate EFA#621
mselim00 merged 1 commit into
aws:mainfrom
mselim00:efa-test

Conversation

@mselim00

@mselim00 mselim00 commented Jun 1, 2025

Copy link
Copy Markdown
Contributor

Issue #, if available:

Description of changes:

Adds some initial tests to validate just EFA capacity. This is useful to run as a pre-flight check before running neuron or nvidia test cases that may use or require EFA, but also for testing EFA directly and on instance types which support it but don't have accelerators.

These are mainly intended as smoke tests at the moment, but sets up for more advanced future testing. At the moment, this will:

  1. Verify that the EFA device capacity on the nodes is consistent with the expected.
  2. Verify that all the requested number of EFA devices are visible within the container.
  3. Verify that RDM and DGRAM endpoints are available for each EFA device.
  4. Verify that two different pods can ping pong traffic between each other via efa (using the libfabric fi_pingpong utility)

Sample output (verbose logs cropped):

=== RUN   TestPingPong
=== RUN   TestPingPong/pingpong
2025/06/02 18:26:19 [INFO] Using node ip-192-168-41-150.us-west-2.compute.internal (type: r6i.32xlarge), as server
2025/06/02 18:26:19 [INFO] Using node ip-192-168-55-210.us-west-2.compute.internal (type: r6i.32xlarge), as client
=== RUN   TestPingPong/pingpong/Pingpong_between_nodes_succeeds
=== NAME  TestPingPong/pingpong
    pingpong_test.go:176: Logs for server
        bytes   #sent   #ack     total       time     MB/sec    usec/xfer   Mxfers/sec
        0       10k     =10k     0           0.21s      0.00      10.38       0.10
        1       10k     =10k     19k         0.21s      0.10      10.37       0.10
        2       10k     =10k     39k         0.21s      0.19      10.36       0.10
        3       10k     =10k     58k         0.21s      0.29      10.36       0.10
        4       10k     =10k     78k         0.21s      0.39      10.35       0.10
        6       10k     =10k     117k        0.21s      0.58      10.38       0.10
        8       10k     =10k     156k        0.21s      0.77      10.35       0.10
        12      10k     =10k     234k        0.21s      1.16      10.35       0.10
        16      10k     =10k     312k        0.21s      1.54      10.36       0.10
        24      10k     =10k     468k        0.21s      2.32      10.35       0.10
        32      10k     =10k     625k        0.21s      3.09      10.36       0.10
        48      10k     =10k     937k        0.21s      4.48      10.71       0.09
        64      10k     =10k     1.2m        0.22s      5.94      10.77       0.09
        96      10k     =10k     1.8m        0.22s      8.86      10.83       0.09
        128     10k     =10k     2.4m        0.22s     11.78      10.87       0.09
        192     10k     =10k     3.6m        0.22s     17.67      10.86       0.09
        256     10k     =10k     4.8m        0.22s     23.56      10.87       0.09
        384     10k     =10k     7.3m        0.22s     35.32      10.87       0.09
        512     10k     =10k     9.7m        0.22s     47.05      10.88       0.09
        768     10k     =10k     14m         0.22s     70.21      10.94       0.09
        1k      10k     =10k     19m         0.22s     93.41      10.96       0.09
        1.5k    10k     =10k     29m         0.22s    139.30      11.03       0.09
        2k      10k     =10k     39m         0.22s    184.72      11.09       0.09
        3k      10k     =10k     58m         0.22s    273.50      11.23       0.09
        4k      10k     =10k     78m         0.23s    357.52      11.46       0.09
        6k      10k     =10k     117m        0.24s    511.76      12.01       0.08
        8k      10k     =10k     156m        0.25s    650.53      12.59       0.08
--- PASS: TestPingPong (25.25s)
    --- PASS: TestPingPong/pingpong (25.25s)
        --- PASS: TestPingPong/pingpong/Pingpong_between_nodes_succeeds (25.01s)
=== RUN   TestUnit
=== RUN   TestUnit/unit
=== RUN   TestUnit/unit/Unit_test_succeeds
=== NAME  TestUnit/unit
    unit_test.go:132: Logs for pod "efa-unit-0"
        Running test on a r6i.32xlarge
        provider: efa
            fabric: efa-direct
            domain: efa_0-rdm
            version: 201.0
            type: FI_EP_RDM
            protocol: FI_PROTO_EFA
        provider: efa
            fabric: efa
            domain: efa_0-rdm
            version: 201.0
            type: FI_EP_RDM
            protocol: FI_PROTO_EFA
        provider: efa
            fabric: efa
            domain: efa_0-dgrm
            version: 201.0
            type: FI_EP_DGRAM
            protocol: FI_PROTO_EFA
        Verified at least 1 DGRAM endpoint(s) are available (found 1)
        Verified at least 1 RDM endpoint(s) are available (found 2)
        Success!
    unit_test.go:132: Logs for pod "efa-unit-1"
        Running test on a r6i.32xlarge
        hca_id: efa_0
                transport:                      unspecified (4)
                fw_ver:                         0.0.0.0
                node_guid:                      394b:3d0e:2432:3200
                sys_image_guid:                 0000:0000:0000:0000
                vendor_id:                      0x1d0f
                vendor_part_id:                 61345
                hw_ver:                         0xEFA1
                phys_port_cnt:                  1
                        port:   1
                                state:                  PORT_ACTIVE (4)
                                max_mtu:                4096 (5)
                                active_mtu:             4096 (5)
                                sm_lid:                 0
                                port_lid:               0
                                port_lmc:               0x01
                                link_layer:             Unspecified
        
        Verified 1 EFA device(s) visible
        provider: efa
            fabric: efa-direct
            domain: efa_0-rdm
            version: 201.0
            type: FI_EP_RDM
            protocol: FI_PROTO_EFA
        provider: efa
            fabric: efa
            domain: efa_0-rdm
            version: 201.0
            type: FI_EP_RDM
            protocol: FI_PROTO_EFA
        provider: efa
            fabric: efa
            domain: efa_0-dgrm
            version: 201.0
            type: FI_EP_DGRAM
            protocol: FI_PROTO_EFA
        Verified at least 1 DGRAM endpoint(s) are available (found 1)
        Verified at least 1 RDM endpoint(s) are available (found 2)
        Success!
--- PASS: TestUnit (10.18s)
    --- PASS: TestUnit/unit (10.18s)
        --- PASS: TestUnit/unit/Unit_test_succeeds (10.02s)
PASS
ok      github.com/aws/aws-k8s-tester/test/cases/efa    56.352s

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@mselim00 mselim00 force-pushed the efa-test branch 10 times, most recently from 3ab3239 to 6b5b136 Compare June 1, 2025 21:00
@mselim00 mselim00 marked this pull request as ready for review June 1, 2025 21:01
@mselim00 mselim00 force-pushed the efa-test branch 3 times, most recently from 3beba2b to 9a25b8c Compare June 1, 2025 21:22
Comment thread test/cases/efa/commons.go Outdated
Comment thread test/cases/efa/main_test.go Outdated
Comment thread test/cases/efa/main_test.go
Comment thread test/cases/efa/main_test.go Outdated
Comment thread test/cases/efa/pingpong_test.go
Comment thread test/images/efa/Dockerfile Outdated
@Pavani-Panakanti

Copy link
Copy Markdown
Contributor

Tests looks good to me. Thanks for adding these @mselim00

Comment thread test/cases/efa/pingpong_test.go Outdated
@mselim00 mselim00 force-pushed the efa-test branch 3 times, most recently from 50b9a0c to 2a393d3 Compare June 2, 2025 20:54
@mselim00 mselim00 merged commit 10331ee into aws:main Jun 2, 2025
8 of 10 checks passed
@mselim00 mselim00 deleted the efa-test branch June 2, 2025 23:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants