Refactor parallel communicators, Introduce module_parallel and migrate rhog_io to domain-aware interfaces

AGENTS.md

@@ -26,6 +26,15 @@ rules. Read the complete governance document before making or reviewing changes:
   8. Declare one variable per line; do not use comma-separated declarations.
   9. Do not call MPI routines directly; use the internally-guarded wrappers
      (e.g., `Parallel_Reduce::reduce_*`, `Parallel_Common::bcast_*`) instead.
+  10. Do not write new `#define private public` or `#define protected public`
+      access hacks in test files. If a unit test needs to inspect internal
+      state, either promote the member visibility explicitly or add a
+      public test-only accessor.
+  11. New unit test source files shall be named `test_<module_name>.cpp`,
+      matching the source file they exercise. For example, the test for
+      `rhog_io.cpp` shall be `test_rhog_io.cpp`. This naming keeps the
+      file-to-test relationship discoverable and consistent across the
+      repository. Historical tests are not required to be renamed.
 - Use LF line endings for text files. Only `.bat` and `.cmd` files may use CRLF.
 - Keep source file additions deterministic: update the relevant `CMakeLists.txt`
   or explain why the file is generated or included indirectly.

README20260902

@@ -0,0 +1,65 @@
+# para* 重构计划（分支 2026-09-02-b）
+
+## 背景与起点
+
+- **分支**：`2026-09-02-b`，起点 commit `2a7696f5d`（step-0 cleanup source_base parallel_*）
+- **基线状态**：`source_base/` 下有 14 个 `parallel_*` 文件（2d/cell/comm/common/device/global/grid/reduce），**没有** `module_parallel/` 目录，**没有** ParallelPartition——从 0 开始
+- 之前分支上尝试过 ParallelPartition（8 个裸 MPI_Comm 成员）和 ParaTag enum 两套方案，都已推倒
+
+## 核心设计（已定）
+
+**两个类，放在 `source/source_base/module_parallel/` 下：**
+
+### 1. `ParaWorld` —— 单个通信域
+- 内容：`tag`（字符串常量）+ `comm`（MPI_Comm，串行下不存在）+ `rank` + `size`
+- 把原本散在 GlobalV 的并行参数（NPROC_IN_POOL、RANK_IN_POOL 等）收进对应域对象
+- 方法：`tag()` / `rank()` / `size()` / `comm()`（仅 __MPI）/ `valid()` / `static serial(tag)` 安全退化（size=1, rank=0）
+- 串行编译：`comm()` 用 `#ifdef __MPI` 包住，`rank()`/`size()`/`tag()` 总可用
+- 域特有参数（如 npw_per_proc、2D 网格行列）**不放进类**，由函数按需另外传
+
+### 2. `ParaCollection` —— 全域容器
+- 内容：`std::vector<ParaWorld>`
+- 查找：`find(tag)` 按字符串 tag 线性查找，**找不到返回静态空域（安全退化，不抛异常）**
+- tag 用常量（避免裸字符串拼写错误运行时才暴露）
+
+### 3. `ParaTag` —— 域标签常量
+- 8 大域：`pw` / `kmesh` / `bsame_kdiff` / `bdiff_ksame` / `rgrid` / `diag` / `matrix` / `atom`
+- 对应原全局：POOL_WORLD / KP_WORLD / INT_BGROUP / BP_WORLD / GRID_WORLD / DIAG_WORLD / matrix / atom
+
+## 目标
+
+- 函数通过**注入** `const ParaWorld&` 或 `const ParaCollection&` 获取通信域，不再读裸全局 POOL_WORLD/GlobalV
+- wrapper（Parallel_Common::bcast_* / Parallel_Reduce::reduce_*）加 `ParaWorld` 重载，`#ifdef __MPI` 收进 wrapper 内部，调用点无 `#ifdef`、无 MPI_Comm，串行并行都能编译跑
+- 测试用 `ParaWorld::serial(tag)` 或一行工厂构造，**去掉 GlobalV/divide_pools/set_global_partition 样板**
+
+## 分步计划（每步一个 commit，确认合理再进下一步）
+
+| 步骤 | 内容 | commit 信息 |
+|---|---|---|
+| **step 1** | 建 `module_parallel/` 目录 + `ParaWorld` 类（tag 常量 + comm/rank/size + `serial()` + `valid()`），含单元测试 + CMake/Makefile.Objects 接线 | `feat(parallel): add ParaWorld comm-domain value type` |
+| **step 2** | `ParaCollection`（`vector<ParaWorld>` + `find(tag)` 安全退化返回静态空域），含单元测试 | `feat(parallel): add ParaCollection domain container` |
+| **step 3** | 用 `ParaWorld`/`ParaCollection` 表达 8 大域装配（替代旧 divide_pools 全局写法），接进 driver 初始化 | `feat(parallel): assemble domains into ParaCollection at driver` |
+| **step 4** | `Parallel_Common::bcast_bool` 加 `ParaWorld` 重载（`#ifdef __MPI` 收进 wrapper，串行 no-op，旧签名保留） | `feat(parallel): bcast_bool overload taking ParaWorld` |
+| **step 5** | rhog_io.cpp 打样：注入 `ParaWorld`，`bcast_bool(error, pw)` 一行无 `#ifdef`；read_rhog_test 改一行构造去 GlobalV | `refactor(io): inject ParaWorld into read_rhog` |
+
+## 命名与规范约束
+
+- 文件名小写+下划线：`para_world.h/.cpp`、`para_collection.h/.cpp`
+- C++11，4 空格缩进，大括号独占一行，不用 `using namespace std`，注释用英文 doxygen 格式
+- 不加默认参数，不用全局变量（ParaCollection 通过注入传递，不做全局单例）
+- include guard 用短名，与同目录其它文件一致
+- 不用 goto，不用宏做域替换，struct 不裸露公有成员
+- 函数参数带校验（指针非空、int 范围合理）
+
+## 验证方式
+
+- 编译目录：`/home/510Group/6_abacus_mc/abacus-mc/build_max_para_test`，命令 `make -j 30`
+- 测试：`OMP_NUM_THREADS=1 ctest -V -R <pattern>`
+- 注意：沙箱内 MPI 测试会因 `/dev/nvidiactl` 受限误报崩溃，需看 ctest 日志实际结果
+- 每步 commit 前确认编译 0 错误 + 相关测试通过
+
+## 待确认细节（开工前）
+
+1. 文件路径 `source/source_base/module_parallel/para_world.h/.cpp` 是否 OK
+2. `ParaWorld` 串行下 `comm()` 不存在（`#ifdef __MPI`），`rank()`/`size()` 返回 0/1，`tag()` 总可用——是否 OK
+3. 从 step 1 开始，还是想先调整步骤划分

cmake/Testing.cmake

@@ -32,9 +32,17 @@ endif()
 
  function(AddTest) # function for UT
     cmake_parse_arguments(UT "DYN" "TARGET"
-                          "LIBS;DYN_LIBS;STATIC_LIBS;SOURCES;DEPENDS" ${ARGN})
+                          "LIBS;DYN_LIBS;STATIC_LIBS;SOURCES;DEPENDS;KEEP_FEATURE_DEFINITIONS" ${ARGN})
     add_executable(${UT_TARGET} ${UT_SOURCES})
 
+    # Let this target keep feature definitions (e.g. __MPI) that its source
+    # directory disables via abacus_disable_feature_definitions(). Needed by
+    # tests that genuinely exercise the feature.
+    if(UT_KEEP_FEATURE_DEFINITIONS)
+      set_property(TARGET ${UT_TARGET} PROPERTY
+        ABACUS_KEPT_FEATURE_DEFINITIONS ${UT_KEEP_FEATURE_DEFINITIONS})
+    endif()
+
     if(ENABLE_COVERAGE)
       add_coverage(${UT_TARGET})
     endif()

source/CMakeLists.txt

@@ -417,6 +417,15 @@ define_property(
   BRIEF_DOCS "Additional ABACUS feature definitions for targets in this directory"
   FULL_DOCS "Additional feature definitions for targets created in this directory.")
 
+define_property(
+  TARGET
+  PROPERTY ABACUS_KEPT_FEATURE_DEFINITIONS
+  BRIEF_DOCS "Feature definitions this target keeps despite a directory-level disable"
+  FULL_DOCS "Feature definitions that must not be stripped from this target even "
+    "when its source directory disables them via "
+    "abacus_disable_feature_definitions(). Used by tests that genuinely need a "
+    "feature (e.g. __MPI) inside a directory that otherwise disables it.")
+
 function(abacus_disable_feature_definitions)
   abacus_normalize_definitions(_defs ${ARGN})
   set_property(DIRECTORY APPEND PROPERTY
@@ -448,7 +457,12 @@ function(abacus_apply_build_options target)
   set(_defs "${_abacus_feature_definitions}")
   get_property(_disabled DIRECTORY "${_source_dir}" PROPERTY ABACUS_DISABLED_FEATURE_DEFINITIONS)
   get_property(_local DIRECTORY "${_source_dir}" PROPERTY ABACUS_LOCAL_FEATURE_DEFINITIONS)
+  get_target_property(_kept "${target}" ABACUS_KEPT_FEATURE_DEFINITIONS)
 
+  if(_kept)
+    # A target may opt back into definitions its directory disables.
+    list(REMOVE_ITEM _disabled ${_kept})
+  endif()
   if(_disabled)
     # Filter after conditional definitions have been evaluated.
     string(JOIN "|" _disabled_regex ${_disabled})

source/Makefile.Objects

@@ -22,6 +22,7 @@ VPATH=./src_global:\
 ./source_base:\
 ./source_base/kernels:\
 ./source_base/module_external:\
+./source_base/module_parallel:\
 ./source_base/module_container/base/core:\
 ./source_base/module_container/ATen/core:\
 ./source_base/module_container/ATen/kernels:\
@@ -625,7 +626,6 @@ OBJS_IO=module_parameter/input_conv.o\
     output.o\
     module_output/print_info.o\
     module_output/read_cube.o\
-    module_chgpot/rhog_io.o\
     module_wf/read_wfc_pw.o\
     module_wf/read_wf2rho_pw.o\
     module_restart/restart.o\
@@ -647,10 +647,8 @@ OBJS_IO=module_parameter/input_conv.o\
     module_output/write_pao.o\
     module_wf/write_wfc_pw.o\
     module_output/write_cube.o\
-    module_chgpot/write_elecstat_pot.o\
     module_elf/write_elf.o\
     module_dipole/write_dipole.o\
-    module_chgpot/write_init.o\
     module_current/td_current_io.o\
     module_current/td_current_io_comm.o\
     td_efield_io.o\
@@ -791,7 +789,18 @@ OBJS_PARALLEL=parallel_common.o\
     parallel_grid.o\
     parallel_kpoints.o\
     parallel_reduce.o\
-    parallel_device.o
+    parallel_device.o\
+    para_world.o\
+    para_collection.o\
+    para_kmesh_world.o\
+    para_pw_world.o\
+    para_diag_world.o\
+    para_rgrid_world.o\
+    para_bgroup_world.o\
+    para_matrix_world.o\
+    para_mpi_func.o\
+    para_setup.o\
+    para_bridge.o
 
 OBJS_SRCPW=h_ewald_pw.o\
     dnrm2.o\
@@ -815,6 +824,9 @@ OBJS_SRCPW=h_ewald_pw.o\
     mix_precond.o\
     charge_mixing_rho.o\
     charge_mixing_uspp.o\
+    rhog_io.o\
+    write_elecstat_pot.o\
+    write_init.o\
     fp_energy.o\
     setup_pot.o\
     setup_pwrho.o\

source/source_base/CMakeLists.txt

@@ -73,6 +73,17 @@ add_library(
     module_mixing/plain_mixing.cpp
     module_mixing/pulay_mixing.cpp
     module_mixing/broyden_mixing.cpp
+    module_parallel/para_world.cpp
+    module_parallel/para_collection.cpp
+    module_parallel/para_kmesh_world.cpp
+    module_parallel/para_pw_world.cpp
+    module_parallel/para_diag_world.cpp
+    module_parallel/para_rgrid_world.cpp
+    module_parallel/para_bgroup_world.cpp
+    module_parallel/para_matrix_world.cpp
+    module_parallel/para_mpi_func.cpp
+    module_parallel/para_setup.cpp
+    module_parallel/para_bridge.cpp
     ${LIBM_SRC}
 )
 
@@ -95,6 +106,7 @@ if(BUILD_TESTING)
     add_subdirectory(module_mixing/test)
     add_subdirectory(module_device/test)
     add_subdirectory(module_grid/test)
+    add_subdirectory(module_parallel/test)
     if (ENABLE_ABACUS_LIBM)
       add_subdirectory(libm/test)
     endif()

source/source_base/module_parallel/para_bgroup_world.cpp

@@ -0,0 +1,22 @@
+#include "para_bgroup_world.h"
+
+namespace Parallel
+{
+
+ParaBgroupWorld::ParaBgroupWorld()
+    : ParaWorld("bdiff_ksame"), my_bndgroup_(0), nbndgroup_(1)
+{
+}
+
+#ifdef __MPI
+ParaBgroupWorld::ParaBgroupWorld(const MPI_Comm& intra_comm, const MPI_Comm& inter_comm, int nbndgroup)
+    : ParaWorld("bdiff_ksame", intra_comm), inter_comm_(inter_comm), nbndgroup_(nbndgroup)
+{
+    if (inter_comm != MPI_COMM_NULL)
+    {
+        MPI_Comm_rank(inter_comm, &my_bndgroup_);
+    }
+}
+#endif
+
+} // namespace Parallel

source/source_base/module_parallel/para_bgroup_world.h

@@ -0,0 +1,67 @@
+#ifndef PARA_BGROUP_WORLD_H
+#define PARA_BGROUP_WORLD_H
+
+#include "para_world.h"
+
+namespace Parallel
+{
+
+/**
+ * @brief bgroup parallel domain: band group communication topology.
+ *
+ * Self-contained replacement for INT_BGROUP + BP_WORLD +
+ * GlobalV::MY_BNDGROUP/NPROC_IN_BNDGROUP/RANK_IN_BPGROUP.
+ *
+ * The band group domain has two communicators:
+ *   - intra: INT_BGROUP (same band group, different k/pw)
+ *   - inter: BP_WORLD (different band groups, same k)
+ *
+ * Tests only need this header.
+ */
+class ParaBgroupWorld : public ParaWorld
+{
+public:
+    /**
+     * @brief Construct a serial bgroup domain (single band group).
+     */
+    ParaBgroupWorld();
+
+#ifdef __MPI
+    /**
+     * @brief Construct a bgroup domain from intra and inter communicators.
+     *
+     * @param[in] intra_comm  intra-group communicator (e.g. INT_BGROUP)
+     * @param[in] inter_comm  inter-group communicator (e.g. BP_WORLD)
+     * @param[in] nbndgroup   number of band groups
+     */
+    ParaBgroupWorld(const MPI_Comm& intra_comm, const MPI_Comm& inter_comm, int nbndgroup);
+#endif
+
+    /// Band group index of this process.
+    int my_bndgroup() const { return my_bndgroup_; }
+
+    /// Number of band groups.
+    int nbndgroup() const { return nbndgroup_; }
+
+    /// Rank within the band group (alias for rank()).
+    int rank_in_bpgroup() const { return rank(); }
+
+    /// Number of processes in the band group (alias for size()).
+    int nproc_in_bndgroup() const { return size(); }
+
+#ifdef __MPI
+    /// Inter-group communicator (BP_WORLD equivalent).
+    MPI_Comm inter_comm() const { return inter_comm_; }
+#endif
+
+private:
+    int my_bndgroup_ = 0;
+    int nbndgroup_ = 1;
+#ifdef __MPI
+    MPI_Comm inter_comm_ = MPI_COMM_NULL;
+#endif
+};
+
+} // namespace Parallel
+
+#endif // PARA_BGROUP_WORLD_H

source/source_base/module_parallel/para_bridge.cpp

@@ -0,0 +1,22 @@
+#include "para_bridge.h"
+#include "para_tag.h"
+
+#ifdef __MPI
+#include "source_base/parallel_comm.h"
+#endif
+
+namespace Parallel
+{
+
+// Temporary bridge: construct a pw-domain ParaWorld from the old globals.
+// Delete this file once ParaCollection is wired into driver initialization.
+ParaWorld make_pw_world()
+{
+#ifdef __MPI
+    return ParaWorld::make_mpi(ParaTag::pw, POOL_WORLD);
+#else
+    return ParaWorld::serial(ParaTag::pw);
+#endif
+}
+
+} // namespace Parallel

source/source_base/module_parallel/para_bridge.h

@@ -0,0 +1,21 @@
+#ifndef PARA_BRIDGE_H
+#define PARA_BRIDGE_H
+
+#include "para_world.h"
+
+namespace Parallel
+{
+
+/**
+ * @brief Temporary bridge: construct a pw-domain ParaWorld from the old
+ * global POOL_WORLD (MPI) or as a serial domain (non-MPI).
+ *
+ * Hides the #ifdef __MPI from call sites so they stay one-liner. Delete
+ * this function (and this file) once ParaCollection is wired into driver
+ * initialization and callers receive a ParaWorld& from above.
+ */
+ParaWorld make_pw_world();
+
+} // namespace Parallel
+
+#endif // PARA_BRIDGE_H

source/source_base/module_parallel/para_collection.cpp

@@ -0,0 +1,31 @@
+#include "para_collection.h"
+
+namespace Parallel
+{
+
+void ParaCollection::add(std::unique_ptr<ParaWorld> world)
+{
+    for (const auto& existing : worlds_)
+    {
+        if (existing->tag() == world->tag())
+        {
+            return;
+        }
+    }
+    worlds_.push_back(std::move(world));
+}
+
+const ParaWorld& ParaCollection::find(const std::string& tag) const
+{
+    for (const auto& world : worlds_)
+    {
+        if (world->tag() == tag)
+        {
+            return *world;
+        }
+    }
+    static const ParaWorld empty = ParaWorld::serial("");
+    return empty;
+}
+
+} // namespace Parallel