Skip to content

Commit ee450d9

Browse files
mohanchenabacus_fixer
andauthored
Refactor parallel communicators, Introduce module_parallel and migrate rhog_io to domain-aware interfaces (#7899)
* refactor(parallel): step-0 cleanup source_base parallel_* Scope: 18 files changed, +75 -134 (net -59 lines). No behavior change. Build: verified with cmake --build build (exit=0), abacus_basic_para built. Parallel common (模板化去重) - parallel_common.cpp: collapse the 6 per-type bcast_* copy-paste wrappers (int / double / complex<double>, scalar + array) into one template <typename T> bcast_world_impl backed by Parallel_Reduce's existing MPI_Type<T> traits. Keep bcast_bool/string/char as bespoke helpers and drop the redundant extra assignment in bcast_bool. - test_parallel/CMakeLists.txt (MODULE_BASE_ParaCommon): list the transitive objects the standalone test now needs - parallel_reduce, parallel_comm, parallel_global, tool_quit, global_file, global_function, memory_recorder, timer - because parallel_common.cpp references Parallel_Reduce::MPI_Type<T>::value, which depends on the six global MPI_Comm in parallel_comm.cpp, which calls Parallel_Global::divide_mpi_groups. Parallel 2D (收紧头文件依赖, rule 3) - parallel_2d.h: drop the unused #include "source_base/parallel_comm.h" (parallel_2d.h only needed <mpi.h> for MPI_Comm); this was the single largest conduit that pulled POOL_WORLD/KP_WORLD/INT_BGROUP/BP_WORLD/ GRID_WORLD/DIAG_WORLD declarations into every user of Parallel_2D / Parallel_Orbitals. - Explicitly include source_base/parallel_comm.h in the 13 consumers that were relying on transitive include to reference the global communicators: write_hs.h, hsolver_lrtd.hpp, sto_iter.cpp / sto_tool.cpp / sto_dos.cpp / sto_elecond.cpp, chgmixing.cpp, hsolver_pw_sdft.cpp, esolver_sdft_pw.cpp, diago_bpcg_test.cpp, test_hsolver_sdft.cpp. Parallel grid (重复逻辑收敛 + 现代C++清理) - parallel_grid.h/cpp: merge zpiece_to_all and zpiece_to_stogroup into one zpiece_distribute(zpiece, iz, rho, is_sdft). The only difference between the two (~130 lines each) is the choice of communicator (MPI_COMM_WORLD vs INT_BGROUP) and the root rank used in the non-pool-0 receive path (MY_RANK vs RANK_IN_BPGROUP); both are selected with two local variables so the four send/recv branches (pool0 root copy, other-rank recv, pool-root multicast, other-pool recv) share one implementation. Also rename duplicate "case 2" labels into "case 2 / case 3". - parallel_grid.cpp::z_distribution: replace raw new int[KPAR] / delete[] startp with std::vector<int> startp(KPAR) and remove five blocks of commented-out debug output. Misc dead-code / include cleanup - parallel_reduce.h: remove the dead declaration bool check_if_equal(double& v) - never defined, never referenced anywhere in the repo. - parallel_global.cpp: drop two unused includes (parallel_common.h, parallel_reduce.h) left over from earlier refactors. Governance notes: - GlobalV budget: PR total added=3 GlobalV refs, removed=19, net_delta = -16. The 3 new refs are inside the merged zpiece_distribute function (it uses the same GlobalV::MY_POOL etc. as the original two functions, they just appear on new lines in the diff). Remaining GlobalV usage in Parallel_Grid stays for Step 1 (ProcessTopology injection). - Added header includes: parallel_2d.h now includes <mpi.h> directly (it previously got MPI_Comm via parallel_comm.h); write_hs.h and hsolver_lrtd.hpp now include parallel_comm.h because the implementations reference DIAG_WORLD and POOL_WORLD respectively and were previously hiding that dependency behind the Parallel_Orbitals -> Parallel_2D -> parallel_comm transit. - No INPUT / documentation change required: all public APIs keep the same signatures and semantics (bcast, grid reduce, Parallel_2D). * refactor(parallel): add module_parallel with ParaWorld, ParaCollection, ParaKmeshWorld Step 1-2: establish module_parallel/ as the new home for parallel domain abstractions. - ParaWorld: base class with tag + rank + size + comm, virtual destructor for polymorphism, protected constructors, make_serial factory for ParaCollection::add() - ParaTag: 8 domain tag string constants (pw/kmesh/bsame_kdiff/bdiff_ksame/ rgrid/diag/matrix/atom) - ParaCollection: unique_ptr<ParaWorld> container with find(tag) and find_as<T>() for safe downcast; missing tag returns static empty domain - ParaKmeshWorld: first domain subclass, extracts Parallel_Kpoints logic (k-point distribution, pool mapping, cross-pool collection, gather_kvec) into a self-contained class; tests only include para_kmesh_world.h - All tests moved to module_parallel/test/ with both serial and MPI variants - Old para_world tests in test/ and test_parallel/ removed - CMakeLists.txt and Makefile.Objects updated * refactor(parallel): add ParaPwWorld for pw domain Extract pool-level parallel parameters (poolnproc, poolrank, npw, npw_per, npwtot) from PW_Basis into a self-contained domain class. The actual FFT-based distribution algorithm (method1/method2) stays in PW_Basis; ParaPwWorld only holds the result: how many plane waves each process gets. Tests only include para_pw_world.h, no PW_Basis or parallel_comm.h needed. * refactor(parallel): add ParaDiagWorld for diag domain Extract DIAG_WORLD + GlobalV::DRANK/DSIZE/DCOLOR into a self-contained domain class with drank()/dsize()/dcolor() aliases. Used by PEXSI solver and LCAO HS IO modules. * refactor(parallel): add ParaRgridWorld for rgrid domain Extract GRID_WORLD + GlobalV::GRANK/GSIZE + Parallel_Grid's z-distribution tables (numz/startz/whichpro) into a self-contained domain class. Owns grid dimensions and per-process z-plane allocation. Cross-pool operations (reduce_across_pools, bcast, reduce) will be added later with explicit communicator parameters, breaking the cross-domain dependency on KP_WORLD/INT_BGROUP. * refactor(parallel): add ParaBgroupWorld for bgroup domain Extract INT_BGROUP + BP_WORLD + GlobalV::MY_BNDGROUP/NPROC_IN_BNDGROUP/ RANK_IN_BPGROUP into a self-contained domain class with both intra and inter group communicators. Used by SDFT band parallel and BPCG diagonalization. * refactor(parallel): add ParaMatrixWorld for matrix domain Extract the process-grid part of Parallel_2D (dim0/dim1/coord) into a self-contained domain class. The actual ScaLAPACK descriptor and BLACS context management stay in Parallel_2D; this class only holds the process grid dimensions and coordinates, computed automatically from the communicator size. * feat(module_parallel): add para_comm communication wrappers Add para_comm.h/.cpp providing domain-aware communication primitives (bcast, reduce, gather, min/max) that accept const ParaWorld& instead of hardcoding MPI_COMM_WORLD or POOL_WORLD. Serial mode is a no-op (except gather_int which copies locally). Invalid worlds are skipped. Tests only need para_comm.h + para_world.h, no dependency on the old parallel_common.h or parallel_reduce.h. * feat(module_parallel): add cross-domain operations on ParaRgridWorld Add reduce_across_pools, bcast_data, and reduce_data methods to ParaRgridWorld. These replace Parallel_Grid::reduce_across_pools, Parallel_Grid::bcast, and Parallel_Grid::reduce respectively. Key difference from old code: communicators are passed explicitly as const ParaWorld& parameters, eliminating GlobalV::KPAR/MY_POOL/ RANK_IN_POOL and global POOL_WORLD/KP_WORLD/INT_BGROUP dependencies. Serial mode: reduce_across_pools is no-op, bcast_data/reduce_data copy local slabs directly. Invalid worlds are skipped safely. * feat(module_parallel): add para_setup domain initialization/split tools Add para_setup.h/.cpp providing: - divide_mpi_groups: utility to split nproc into num_groups (serial+MPI) - split_pools: split WORLD into k-pools + band groups, returns ParaWorld objects for pw/kmesh/bsame_kdiff/bdiff_ksame domains - split_diag_world: split for DIAG_WORLD - split_grid_world: split for GRID_WORLD - setup_para_worlds: top-level function assembling all 8 domains into a ParaCollection Also adds make_mpi/make_mpi_ptr factory methods to ParaWorld for constructing domains from MPI communicators (protected ctor stays protected, factories are the public API). Replaces Parallel_Global::divide_pools, split_diag_world, split_grid_world, and divide_mpi_groups without touching old code. * refactor(module_parallel): rename para_comm to para_mpi_func The name "para_comm" was ambiguous (communicator? communication?). Rename to para_mpi_func to match the file's role: a collection of domain-aware MPI functions (bcast/reduce/gather/min/max). Files renamed: - para_comm.h/.cpp -> para_mpi_func.h/.cpp - para_comm_test.cpp -> para_mpi_func_test.cpp - para_comm_mpi_test.cpp/.sh -> para_mpi_func_mpi_test.cpp/.sh Targets renamed accordingly (MODULE_BASE_para_mpi_func[_mpi]); gtest suites renamed to ParaMpiFuncTest/ParaMpiFuncMpiTest. No behavior change, 16/16 tests pass. * feat(module_parallel): add esolver/images domains and parent-comm splits Add two top-level parallel domains for multi-image calculations (e.g. NEB replicas with independent unit cells): - para_esolver_world [color = image_id]: all processes of one esolver instance; every existing solver domain is now derived from it instead of being hard-wired to MPI_COMM_WORLD. - para_images_world [color = rank_in_esolver]: cross-image communicator connecting corresponding ranks; MPI_COMM_NULL for nimage == 1 or uneven splits, following the KP_WORLD convention. split_pools/split_diag_world/split_grid_world now take a parent MPI_Comm, and setup_para_worlds takes nimage and builds the full hierarchy: WORLD -> images split -> esolver domain -> kmesh/pw/ bgroup/diag/rgrid/matrix. The hierarchy tree is documented in para_setup.h. New MPI test (mpirun -np 4) verifies single-image degradation, 2-image split ranks/sizes, derived-domain sizes, and kpar=2 inside images; multi-image tests skip under single-rank direct execution. * Migrate rhog_io to domain-aware parallel interface End-to-end pilot migration of read_rhog/write_rhog from the old POOL_WORLD + GlobalV::RANK_IN_POOL + #ifdef __MPI pattern to the new ParaWorld-based interface. Changes: - Add barrier() to para_mpi_func (required by write_rhog's 6 barriers) - Add para_bridge.h/.cpp: temporary factory make_pw_world() that wraps POOL_WORLD into a ParaWorld, hiding #ifdef __MPI from call sites. Delete this file once ParaCollection is wired into driver init. - Migrate rhog_io.cpp: all MPI_Bcast(POOL_WORLD) -> Parallel::bcast_*, all MPI_Barrier(POOL_WORLD) -> Parallel::barrier, all GlobalV::RANK_IN_POOL -> pw_world.rank(). Zero #ifdef __MPI in the file body. - Change read_rhog signature: add const ParaWorld& parameter. - Change write_rhog signature: replace ipool/irank/nrank with const ParaWorld&. Move "only pool 0 writes" check to caller. - Update call sites: charge_init.cpp (2 read_rhog calls), esolver_fp.cpp (2 write_rhog calls), read_rhog_test.cpp. - Add barrier tests to para_mpi_func_test and para_mpi_func_mpi_test. Verification: make -j30 0 errors; ctest read_rhog 4/4 pass, module_parallel 18/18 pass. Old parallel_* test failures are pre-existing (missing .sh scripts). * fix: add module_parallel to VPATH in Makefile.Objects Without this VPATH entry, make cannot locate para_*.cpp source files for the OBJS_PARALLEL targets, causing build failure: No rule to make target 'build/obj/para_world.o' * refactor(rhog_io): remove PARAM dependency, pass nspin as parameter - Add explicit nspin parameter to read_rhog() signature, making it consistent with write_rhog() which already has nspin - Remove #include of parameter.h and global_variable.h from rhog_io.cpp - No longer reads PARAM.inp.nspin anywhere - Update charge_init.cpp call sites to pass the existing local nspin var - Remove dead PARAM.input.nspin assignments and #define private public hack from read_rhog_test.cpp (rhog_io no longer consumes PARAM) * refactor(read_rhog_test): replace raw new/delete with std::vector - PW_Basis* rhopw (new+delete) -> value member PW_Basis rhopw - new complex*[1] + new complex[1471] (double-delete chain) -> vector<vector<complex>> rhog_data + vector<complex*> rhog - Remove TearDown() entirely (RAII handles cleanup) - Update call sites to use &rhopw and rhog.data() * refactor(rhog_io): inject warning stream, remove GlobalV dependency - Add std::ostream* os_warning parameter to read_rhog() and write_rhog() (no default value; callers must pass explicitly) - Replace ModuleBase::WARNING/WARNING_QUIT with a local warn() helper that writes to the injected stream on rank 0 - Remove indirect GlobalV dependencies: drop TITLE, timer, WARNING, WARNING_QUIT calls; drop timer.h include - write_rhog: replace WARNING_QUIT with warn() + return false (same flow) - Update all call sites to pass &GlobalV::ofs_warning: - charge_init.cpp (2 read_rhog calls) - esolver_fp.cpp (2 write_rhog calls) - read_rhog_test.cpp (4 read_rhog calls) - read_rhog_test.cpp main(): replace GlobalV parallel vars with locals * refactor: migrate rhog_io, write_elecstat_pot, write_init to source_estate Move three file pairs from source_io/module_chgpot to source_estate root: - rhog_io.h/cpp - write_elecstat_pot.h/cpp - write_init.h/cpp Also migrate read_rhog_test.cpp to source_estate/test/. Update all include paths, CMakeLists.txt (source_io and source_estate, including test configs), and Makefile.Objects (OBJS_IO -> OBJS_SRCPW). Add AGENTS.md rules 10-11: forbid #define private public access hacks in new tests, and require test_<module>.cpp naming for new unit tests. * refactor: rename read_rhog_test.cpp to test_rhog_io.cpp * test(rhog_io): add write_rhog tests and remove GlobalV from test file - Add 3 new tests: WriteRoundTrip, WriteFileFail, WriteRoundTripNspin2 (write_rhog was previously 0% covered by tests) - Add OsNullptrSilent test for nullptr warning stream safety - Remove GlobalV::ofs_warning usage, use local std::ofstream + fixture helpers - Extract setup_pw_basis() fixture method to avoid duplication * refactor(rhog_io): use namespace elecstate, replace ZEROS with std::fill, add input validation - Rename namespace ModuleIO -> elecstate to match source_estate convention (rhog_io was the only source_estate file still using ModuleIO) - Replace 3x ModuleBase::GlobalFunc::ZEROS with std::fill, remove global_function.h dependency, add <algorithm> - Add defensive parameter checks at the top of read_rhog and write_rhog: - pw_basis null check - rhog null check - nspin range check (1-4) - read_rhog: nx/ny/nz > 0 check - Update all call sites in esolver_fp.cpp, charge_init.cpp, test_rhog_io.cpp - Update test expected warning strings to match new namespace * fix(rhog_io): use whitelist for nspin validation to reject nspin=3 * fix(test_rhog_io): keep __MPI per-target so the MPI test initializes MPI The test aborted with "MPI_Comm_rank() called before MPI_INIT" because the directory-level abacus_disable_feature_definitions(__MPI) in source_estate/test stripped __MPI from every target there. With __MPI gone, the test's main() never called MPI_Init, while the linked base/planewave libraries still issue real MPI calls. Add a target-level escape hatch: a new ABACUS_KEPT_FEATURE_DEFINITIONS target property (exposed as AddTest's KEEP_FEATURE_DEFINITIONS) lets a single target keep definitions its directory disables. Use it for this test, link planewave instead of planewave_serial, and copy the support/ data at configure time (install() is skipped by plain make+ctest). Also fix test bugs hidden while __MPI was stripped: drop const from kpar (passed by reference to divide_pools), call setup_pw_basis() in the three tests that need a valid grid, and set npwtot=1000 in InconsistentGammaOnly to trigger the expected "planewaves not used" warning. Verified: make MODULE_ESTATE_test_rhog_io && ctest -R MODULE_ESTATE_test_rhog_io -> 10/10 passed. * refactor(source_estate): fold charge_mpi_test into test/, drop test_mpi/ charge_mpi_test is a real MPI test (its main() calls MPI_Init) but does not need a separate directory now that AddTest supports KEEP_FEATURE_DEFINITIONS. Move it into test/ alongside the other module tests, keep __MPI for that target, and register its mpirun -np 4 variant there. Remove the now-empty test_mpi/ subdirectory and its add_subdirectory() call. Verified: ctest -R "MODULE_ESTATE_charge_mpi_test|MODULE_ESTATE_test_rhog_io" -> 3/3 passed (test_rhog_io 10/10, charge_mpi_test, charge_mpi_test_4np 4/4). --------- Co-authored-by: abacus_fixer <mohanchen@pku.eud.cn>
1 parent a4be870 commit ee450d9

82 files changed

Lines changed: 4195 additions & 785 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

AGENTS.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,15 @@ rules. Read the complete governance document before making or reviewing changes:
2626
8. Declare one variable per line; do not use comma-separated declarations.
2727
9. Do not call MPI routines directly; use the internally-guarded wrappers
2828
(e.g., `Parallel_Reduce::reduce_*`, `Parallel_Common::bcast_*`) instead.
29+
10. Do not write new `#define private public` or `#define protected public`
30+
access hacks in test files. If a unit test needs to inspect internal
31+
state, either promote the member visibility explicitly or add a
32+
public test-only accessor.
33+
11. New unit test source files shall be named `test_<module_name>.cpp`,
34+
matching the source file they exercise. For example, the test for
35+
`rhog_io.cpp` shall be `test_rhog_io.cpp`. This naming keeps the
36+
file-to-test relationship discoverable and consistent across the
37+
repository. Historical tests are not required to be renamed.
2938
- Use LF line endings for text files. Only `.bat` and `.cmd` files may use CRLF.
3039
- Keep source file additions deterministic: update the relevant `CMakeLists.txt`
3140
or explain why the file is generated or included indirectly.

README20260902

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# para* 重构计划(分支 2026-09-02-b)
2+
3+
## 背景与起点
4+
5+
- **分支**:`2026-09-02-b`,起点 commit `2a7696f5d`(step-0 cleanup source_base parallel_*)
6+
- **基线状态**:`source_base/` 下有 14 个 `parallel_*` 文件(2d/cell/comm/common/device/global/grid/reduce),**没有** `module_parallel/` 目录,**没有** ParallelPartition——从 0 开始
7+
- 之前分支上尝试过 ParallelPartition(8 个裸 MPI_Comm 成员)和 ParaTag enum 两套方案,都已推倒
8+
9+
## 核心设计(已定)
10+
11+
**两个类,放在 `source/source_base/module_parallel/` 下:**
12+
13+
### 1. `ParaWorld` —— 单个通信域
14+
- 内容:`tag`(字符串常量)+ `comm`(MPI_Comm,串行下不存在)+ `rank` + `size`
15+
- 把原本散在 GlobalV 的并行参数(NPROC_IN_POOL、RANK_IN_POOL 等)收进对应域对象
16+
- 方法:`tag()` / `rank()` / `size()` / `comm()`(仅 __MPI)/ `valid()` / `static serial(tag)` 安全退化(size=1, rank=0)
17+
- 串行编译:`comm()` 用 `#ifdef __MPI` 包住,`rank()`/`size()`/`tag()` 总可用
18+
- 域特有参数(如 npw_per_proc、2D 网格行列)**不放进类**,由函数按需另外传
19+
20+
### 2. `ParaCollection` —— 全域容器
21+
- 内容:`std::vector<ParaWorld>`
22+
- 查找:`find(tag)` 按字符串 tag 线性查找,**找不到返回静态空域(安全退化,不抛异常)**
23+
- tag 用常量(避免裸字符串拼写错误运行时才暴露)
24+
25+
### 3. `ParaTag` —— 域标签常量
26+
- 8 大域:`pw` / `kmesh` / `bsame_kdiff` / `bdiff_ksame` / `rgrid` / `diag` / `matrix` / `atom`
27+
- 对应原全局:POOL_WORLD / KP_WORLD / INT_BGROUP / BP_WORLD / GRID_WORLD / DIAG_WORLD / matrix / atom
28+
29+
## 目标
30+
31+
- 函数通过**注入** `const ParaWorld&` 或 `const ParaCollection&` 获取通信域,不再读裸全局 POOL_WORLD/GlobalV
32+
- wrapper(Parallel_Common::bcast_* / Parallel_Reduce::reduce_*)加 `ParaWorld` 重载,`#ifdef __MPI` 收进 wrapper 内部,调用点无 `#ifdef`、无 MPI_Comm,串行并行都能编译跑
33+
- 测试用 `ParaWorld::serial(tag)` 或一行工厂构造,**去掉 GlobalV/divide_pools/set_global_partition 样板**
34+
35+
## 分步计划(每步一个 commit,确认合理再进下一步)
36+
37+
| 步骤 | 内容 | commit 信息 |
38+
|---|---|---|
39+
| **step 1** | 建 `module_parallel/` 目录 + `ParaWorld` 类(tag 常量 + comm/rank/size + `serial()` + `valid()`),含单元测试 + CMake/Makefile.Objects 接线 | `feat(parallel): add ParaWorld comm-domain value type` |
40+
| **step 2** | `ParaCollection`(`vector<ParaWorld>` + `find(tag)` 安全退化返回静态空域),含单元测试 | `feat(parallel): add ParaCollection domain container` |
41+
| **step 3** | 用 `ParaWorld`/`ParaCollection` 表达 8 大域装配(替代旧 divide_pools 全局写法),接进 driver 初始化 | `feat(parallel): assemble domains into ParaCollection at driver` |
42+
| **step 4** | `Parallel_Common::bcast_bool` 加 `ParaWorld` 重载(`#ifdef __MPI` 收进 wrapper,串行 no-op,旧签名保留) | `feat(parallel): bcast_bool overload taking ParaWorld` |
43+
| **step 5** | rhog_io.cpp 打样:注入 `ParaWorld`,`bcast_bool(error, pw)` 一行无 `#ifdef`;read_rhog_test 改一行构造去 GlobalV | `refactor(io): inject ParaWorld into read_rhog` |
44+
45+
## 命名与规范约束
46+
47+
- 文件名小写+下划线:`para_world.h/.cpp`、`para_collection.h/.cpp`
48+
- C++11,4 空格缩进,大括号独占一行,不用 `using namespace std`,注释用英文 doxygen 格式
49+
- 不加默认参数,不用全局变量(ParaCollection 通过注入传递,不做全局单例)
50+
- include guard 用短名,与同目录其它文件一致
51+
- 不用 goto,不用宏做域替换,struct 不裸露公有成员
52+
- 函数参数带校验(指针非空、int 范围合理)
53+
54+
## 验证方式
55+
56+
- 编译目录:`/home/510Group/6_abacus_mc/abacus-mc/build_max_para_test`,命令 `make -j 30`
57+
- 测试:`OMP_NUM_THREADS=1 ctest -V -R <pattern>`
58+
- 注意:沙箱内 MPI 测试会因 `/dev/nvidiactl` 受限误报崩溃,需看 ctest 日志实际结果
59+
- 每步 commit 前确认编译 0 错误 + 相关测试通过
60+
61+
## 待确认细节(开工前)
62+
63+
1. 文件路径 `source/source_base/module_parallel/para_world.h/.cpp` 是否 OK
64+
2. `ParaWorld` 串行下 `comm()` 不存在(`#ifdef __MPI`),`rank()`/`size()` 返回 0/1,`tag()` 总可用——是否 OK
65+
3. 从 step 1 开始,还是想先调整步骤划分

cmake/Testing.cmake

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,9 +32,17 @@ endif()
3232

3333
function(AddTest) # function for UT
3434
cmake_parse_arguments(UT "DYN" "TARGET"
35-
"LIBS;DYN_LIBS;STATIC_LIBS;SOURCES;DEPENDS" ${ARGN})
35+
"LIBS;DYN_LIBS;STATIC_LIBS;SOURCES;DEPENDS;KEEP_FEATURE_DEFINITIONS" ${ARGN})
3636
add_executable(${UT_TARGET} ${UT_SOURCES})
3737

38+
# Let this target keep feature definitions (e.g. __MPI) that its source
39+
# directory disables via abacus_disable_feature_definitions(). Needed by
40+
# tests that genuinely exercise the feature.
41+
if(UT_KEEP_FEATURE_DEFINITIONS)
42+
set_property(TARGET ${UT_TARGET} PROPERTY
43+
ABACUS_KEPT_FEATURE_DEFINITIONS ${UT_KEEP_FEATURE_DEFINITIONS})
44+
endif()
45+
3846
if(ENABLE_COVERAGE)
3947
add_coverage(${UT_TARGET})
4048
endif()

source/CMakeLists.txt

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -417,6 +417,15 @@ define_property(
417417
BRIEF_DOCS "Additional ABACUS feature definitions for targets in this directory"
418418
FULL_DOCS "Additional feature definitions for targets created in this directory.")
419419

420+
define_property(
421+
TARGET
422+
PROPERTY ABACUS_KEPT_FEATURE_DEFINITIONS
423+
BRIEF_DOCS "Feature definitions this target keeps despite a directory-level disable"
424+
FULL_DOCS "Feature definitions that must not be stripped from this target even "
425+
"when its source directory disables them via "
426+
"abacus_disable_feature_definitions(). Used by tests that genuinely need a "
427+
"feature (e.g. __MPI) inside a directory that otherwise disables it.")
428+
420429
function(abacus_disable_feature_definitions)
421430
abacus_normalize_definitions(_defs ${ARGN})
422431
set_property(DIRECTORY APPEND PROPERTY
@@ -448,7 +457,12 @@ function(abacus_apply_build_options target)
448457
set(_defs "${_abacus_feature_definitions}")
449458
get_property(_disabled DIRECTORY "${_source_dir}" PROPERTY ABACUS_DISABLED_FEATURE_DEFINITIONS)
450459
get_property(_local DIRECTORY "${_source_dir}" PROPERTY ABACUS_LOCAL_FEATURE_DEFINITIONS)
460+
get_target_property(_kept "${target}" ABACUS_KEPT_FEATURE_DEFINITIONS)
451461

462+
if(_kept)
463+
# A target may opt back into definitions its directory disables.
464+
list(REMOVE_ITEM _disabled ${_kept})
465+
endif()
452466
if(_disabled)
453467
# Filter after conditional definitions have been evaluated.
454468
string(JOIN "|" _disabled_regex ${_disabled})

source/Makefile.Objects

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ VPATH=./src_global:\
2222
./source_base:\
2323
./source_base/kernels:\
2424
./source_base/module_external:\
25+
./source_base/module_parallel:\
2526
./source_base/module_container/base/core:\
2627
./source_base/module_container/ATen/core:\
2728
./source_base/module_container/ATen/kernels:\
@@ -625,7 +626,6 @@ OBJS_IO=module_parameter/input_conv.o\
625626
output.o\
626627
module_output/print_info.o\
627628
module_output/read_cube.o\
628-
module_chgpot/rhog_io.o\
629629
module_wf/read_wfc_pw.o\
630630
module_wf/read_wf2rho_pw.o\
631631
module_restart/restart.o\
@@ -647,10 +647,8 @@ OBJS_IO=module_parameter/input_conv.o\
647647
module_output/write_pao.o\
648648
module_wf/write_wfc_pw.o\
649649
module_output/write_cube.o\
650-
module_chgpot/write_elecstat_pot.o\
651650
module_elf/write_elf.o\
652651
module_dipole/write_dipole.o\
653-
module_chgpot/write_init.o\
654652
module_current/td_current_io.o\
655653
module_current/td_current_io_comm.o\
656654
td_efield_io.o\
@@ -791,7 +789,18 @@ OBJS_PARALLEL=parallel_common.o\
791789
parallel_grid.o\
792790
parallel_kpoints.o\
793791
parallel_reduce.o\
794-
parallel_device.o
792+
parallel_device.o\
793+
para_world.o\
794+
para_collection.o\
795+
para_kmesh_world.o\
796+
para_pw_world.o\
797+
para_diag_world.o\
798+
para_rgrid_world.o\
799+
para_bgroup_world.o\
800+
para_matrix_world.o\
801+
para_mpi_func.o\
802+
para_setup.o\
803+
para_bridge.o
795804

796805
OBJS_SRCPW=h_ewald_pw.o\
797806
dnrm2.o\
@@ -815,6 +824,9 @@ OBJS_SRCPW=h_ewald_pw.o\
815824
mix_precond.o\
816825
charge_mixing_rho.o\
817826
charge_mixing_uspp.o\
827+
rhog_io.o\
828+
write_elecstat_pot.o\
829+
write_init.o\
818830
fp_energy.o\
819831
setup_pot.o\
820832
setup_pwrho.o\

source/source_base/CMakeLists.txt

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,17 @@ add_library(
7373
module_mixing/plain_mixing.cpp
7474
module_mixing/pulay_mixing.cpp
7575
module_mixing/broyden_mixing.cpp
76+
module_parallel/para_world.cpp
77+
module_parallel/para_collection.cpp
78+
module_parallel/para_kmesh_world.cpp
79+
module_parallel/para_pw_world.cpp
80+
module_parallel/para_diag_world.cpp
81+
module_parallel/para_rgrid_world.cpp
82+
module_parallel/para_bgroup_world.cpp
83+
module_parallel/para_matrix_world.cpp
84+
module_parallel/para_mpi_func.cpp
85+
module_parallel/para_setup.cpp
86+
module_parallel/para_bridge.cpp
7687
${LIBM_SRC}
7788
)
7889

@@ -95,6 +106,7 @@ if(BUILD_TESTING)
95106
add_subdirectory(module_mixing/test)
96107
add_subdirectory(module_device/test)
97108
add_subdirectory(module_grid/test)
109+
add_subdirectory(module_parallel/test)
98110
if (ENABLE_ABACUS_LIBM)
99111
add_subdirectory(libm/test)
100112
endif()
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#include "para_bgroup_world.h"
2+
3+
namespace Parallel
4+
{
5+
6+
ParaBgroupWorld::ParaBgroupWorld()
7+
: ParaWorld("bdiff_ksame"), my_bndgroup_(0), nbndgroup_(1)
8+
{
9+
}
10+
11+
#ifdef __MPI
12+
ParaBgroupWorld::ParaBgroupWorld(const MPI_Comm& intra_comm, const MPI_Comm& inter_comm, int nbndgroup)
13+
: ParaWorld("bdiff_ksame", intra_comm), inter_comm_(inter_comm), nbndgroup_(nbndgroup)
14+
{
15+
if (inter_comm != MPI_COMM_NULL)
16+
{
17+
MPI_Comm_rank(inter_comm, &my_bndgroup_);
18+
}
19+
}
20+
#endif
21+
22+
} // namespace Parallel
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
#ifndef PARA_BGROUP_WORLD_H
2+
#define PARA_BGROUP_WORLD_H
3+
4+
#include "para_world.h"
5+
6+
namespace Parallel
7+
{
8+
9+
/**
10+
* @brief bgroup parallel domain: band group communication topology.
11+
*
12+
* Self-contained replacement for INT_BGROUP + BP_WORLD +
13+
* GlobalV::MY_BNDGROUP/NPROC_IN_BNDGROUP/RANK_IN_BPGROUP.
14+
*
15+
* The band group domain has two communicators:
16+
* - intra: INT_BGROUP (same band group, different k/pw)
17+
* - inter: BP_WORLD (different band groups, same k)
18+
*
19+
* Tests only need this header.
20+
*/
21+
class ParaBgroupWorld : public ParaWorld
22+
{
23+
public:
24+
/**
25+
* @brief Construct a serial bgroup domain (single band group).
26+
*/
27+
ParaBgroupWorld();
28+
29+
#ifdef __MPI
30+
/**
31+
* @brief Construct a bgroup domain from intra and inter communicators.
32+
*
33+
* @param[in] intra_comm intra-group communicator (e.g. INT_BGROUP)
34+
* @param[in] inter_comm inter-group communicator (e.g. BP_WORLD)
35+
* @param[in] nbndgroup number of band groups
36+
*/
37+
ParaBgroupWorld(const MPI_Comm& intra_comm, const MPI_Comm& inter_comm, int nbndgroup);
38+
#endif
39+
40+
/// Band group index of this process.
41+
int my_bndgroup() const { return my_bndgroup_; }
42+
43+
/// Number of band groups.
44+
int nbndgroup() const { return nbndgroup_; }
45+
46+
/// Rank within the band group (alias for rank()).
47+
int rank_in_bpgroup() const { return rank(); }
48+
49+
/// Number of processes in the band group (alias for size()).
50+
int nproc_in_bndgroup() const { return size(); }
51+
52+
#ifdef __MPI
53+
/// Inter-group communicator (BP_WORLD equivalent).
54+
MPI_Comm inter_comm() const { return inter_comm_; }
55+
#endif
56+
57+
private:
58+
int my_bndgroup_ = 0;
59+
int nbndgroup_ = 1;
60+
#ifdef __MPI
61+
MPI_Comm inter_comm_ = MPI_COMM_NULL;
62+
#endif
63+
};
64+
65+
} // namespace Parallel
66+
67+
#endif // PARA_BGROUP_WORLD_H
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#include "para_bridge.h"
2+
#include "para_tag.h"
3+
4+
#ifdef __MPI
5+
#include "source_base/parallel_comm.h"
6+
#endif
7+
8+
namespace Parallel
9+
{
10+
11+
// Temporary bridge: construct a pw-domain ParaWorld from the old globals.
12+
// Delete this file once ParaCollection is wired into driver initialization.
13+
ParaWorld make_pw_world()
14+
{
15+
#ifdef __MPI
16+
return ParaWorld::make_mpi(ParaTag::pw, POOL_WORLD);
17+
#else
18+
return ParaWorld::serial(ParaTag::pw);
19+
#endif
20+
}
21+
22+
} // namespace Parallel
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#ifndef PARA_BRIDGE_H
2+
#define PARA_BRIDGE_H
3+
4+
#include "para_world.h"
5+
6+
namespace Parallel
7+
{
8+
9+
/**
10+
* @brief Temporary bridge: construct a pw-domain ParaWorld from the old
11+
* global POOL_WORLD (MPI) or as a serial domain (non-MPI).
12+
*
13+
* Hides the #ifdef __MPI from call sites so they stay one-liner. Delete
14+
* this function (and this file) once ParaCollection is wired into driver
15+
* initialization and callers receive a ParaWorld& from above.
16+
*/
17+
ParaWorld make_pw_world();
18+
19+
} // namespace Parallel
20+
21+
#endif // PARA_BRIDGE_H

0 commit comments

Comments
 (0)