Enable oneCCL v2 C API and add runtime switch

[Chao1Han]

This PR introduces the new oneCCL C API (aligned with NCCL’s API) and implements runtime switching between the oneCCL v1 API and the new v2 C API (defaulting to the legacy API, with USE_CCL_V2=1 enabling the new one). In CMake, since libccl has not yet merged the v1 and v2 shared libraries, libccl.so.2 is located separately.

• Integration of oneCCL v2 C API into the communication backend.
• Addition of runtime switch mechanisms to dynamically select CCL API implementation.
• Refactoring affected modules to support both v1 and v2 APIs.

[Copilot]

Pull Request Overview

This PR merges oneCCL v1 and v2 API support into a unified XCCL interface by creating an abstraction layer that can dynamically switch between the two API versions based on an environment variable USE_CCL_V2.

• Introduces a unified abstraction layer using C++ variants to handle both API versions
• Adds runtime switching between oneCCL v1 and v2 based on environment variable
• Consolidates duplicate code and functions into shared utilities

Reviewed Changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
src/xccl/xccl.h	New header defining unified API abstractions, data types, and utility functions for both oneCCL versions
src/xccl/xccl.cpp	Implementation of unified collective operations that dispatch to appropriate API version
src/xccl/ProcessGroupXCCL.hpp	Updated to use new unified types and removed duplicate utility functions
src/xccl/ProcessGroupXCCL.cpp	Refactored to use unified API calls and removed version-specific implementations
cmake/XCCL.cmake	Added linking to v2.0 library
cmake/Modules/FindXCCL.cmake	Added discovery of v2.0 library file

Comments suppressed due to low confidence (1)

src/xccl/ProcessGroupXCCL.cpp:1818

• Missing 'opts.asyncOp' parameter in the collective call. This should be the fourth parameter before the profiling title.

            outputSplitSizes, output, &recv_lengths, &recv_offsets);

[guangyey]

Suggested change

void* syclQueue;

I think syclQueue is a redundant member.

[guangyey]

It always can be fetched from xpuStream

[Chao1Han]

You’re right — previously, converting an XPU stream to a CCL stream incurred significant overhead, so it was stored in a map. Since sycl::queue is essentially an alias for the XPU stream, I’ll make the necessary modification.