Skip to content

Commit a9dcda3

Browse files
author
abacus_fixer
committed
refactor(parallel): step-1a rename ProcessTopology to 8-domain consistent *_world_comm naming
Scope: 3 files changed, +209 -82 lines. Build: cmake --build build exit=0, abacus_basic_para linked successfully. Test : OMP_NUM_THREADS=1 mpirun -np 4 MODULE_BASE_ProcessTopology -> 7 / 7 tests passed (5x divide_mpi_groups arithmetic + 2x ProcessTopology accessor / value-semantics cases). Naming decision (after multi-round user review, naming principles: 1. All 8 communicators share the uniform `_world_comm` suffix. 2. The two legacy band-parallel domains INT_BGROUP / BP_WORLD are renamed directly after the "band-side vs k-side diff/same relation", ditching historically confusing abbreviations such as `intra / inter / BP / INT`. 3. Two long-implicit domains are promoted to first-class names: the matrix 2D block-cyclic BLACS world and the atom 3D real-space DD (domain decomposition) world -- so callers no longer rely on the ad-hoc "pick POOL_WORLD or DIAG_WORLD or MPI_COMM_WORLD depending on the scene" convention. Final 8-domain map (see parallel_topology.h for the full comment): Legacy global -> New `*_world_comm` name # 1-line semantic 1. POOL_WORLD -> pw_world_comm # Same-k, same-band-group PW tile (smallest parallel world) 2. KP_WORLD -> kmesh_world_comm # (user-suggested content name) k-mesh root bridge across pools 3. INT_BGROUP -> bsame_kdiff_world_comm # (user-suggested abbreviation) band same, k different; same band-group union across k pools 4. BP_WORLD -> bdiff_ksame_world_comm # (user-suggested abbreviation) band different, k same; intra-pool cross-band-group pair bridge 5. GRID_WORLD -> rgrid_world_comm # explicitly approved by user early on 6. DIAG_WORLD -> diag_world_comm # explicitly approved by user early on - (previously implicit)-> matrix_world_comm # (user-suggested replacement for blacs_world) 2D block-cyclic matrix BLACS world - (previously implicit)-> atom_world_comm # (user-suggested) 3D real-space atomic DD / neighlist world Scalar accessors aligned with the full-word `band_group`: - `bgroup` shorthand is expanded to the full `band_group`: my_band_group() / rank_in_band_group() / nproc_in_band_group(). - `bndpar()` is kept because it matches the INPUT `BNDPAR` flag. - `kpar()` / `my_pool()` / `rank_in_pool()` / `nproc_in_pool()` are preserved to keep the naming compatible with the 100+ occurrences of `pool` in Parallel_Kpoints and related modules. New `band_group_root_rank(bg)` accessor: - Symmetric with `pool_root_rank(pool)` on the pool axis. - Constructor invariant: bndpar_ * nproc_in_band_group_ == world_nproc_, so the root rank formula is simply `band_group * nproc_in_band_group_` -- this matches ABACUS divide_pools output exactly. - Tested on the 10-process / BNDPAR=2 / nproc_in_band_group=5 case: ASSERT band_group_root_rank(0)==0 and band_group_root_rank(1)==5. matrix_world_comm / atom_world_comm injection policy: - Both fields default to MPI_COMM_NULL in the constructor (two new trailing default parameters; the old 6-comm signature still works, so AGENTS rule deepmodeling#5 does not apply). - The "correct" BLACS / DD domain actually depends on the use case (LCAO diag / GK diag / MD step / ...). Distributed modules in Step 2 will fill these two handles in from the appropriate view on an as-needed basis; no real data flow is touched today. - Serial fallback default-constructor asserts matrix_world_comm == MPI_COMM_NULL and atom_world_comm == MPI_COMM_NULL. The ConstructAndAccessors test injects MPI_COMM_SELF / MPI_COMM_WORLD respectively and round-trips all 8 comm accessors plus copy semantics. Governance (agent_governance_check.py --staged): - 1 warning only: "Documentation sync review". No user-visible INPUT / CLI / external API change, so documentation is correctly not updated. No exception needed. - GlobalV / PARAM / GlobalC: 0 new reads. All topology data are injected explicitly through constructor arguments, in line with AGENTS rule #1 (budget: non-increasing, no new globals introduced in this patch). - Header dependencies: only <vector> and <mpi.h> in parallel_topology.h, both required for self-containment (member type + MPI_Comm return types). The 6 legacy global communicators from parallel_comm.h are NOT transitively pulled in, satisfying rule deepmodeling#3. - C++11 compatible, one variable per declaration, no direct MPI calls in this patch.
1 parent a6e1a67 commit a9dcda3

3 files changed

Lines changed: 209 additions & 82 deletions

File tree

source/source_base/parallel_topology.cpp

Lines changed: 65 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -10,17 +10,19 @@ ProcessTopology::ProcessTopology()
1010
rank_in_pool_(0),
1111
nproc_in_pool_(1, 1),
1212
bndpar_(1),
13-
my_bndgroup_(0),
14-
rank_in_bgroup_(0),
15-
nproc_in_bgroup_(1)
13+
my_band_group_(0),
14+
rank_in_band_group_(0),
15+
nproc_in_band_group_(1)
1616
#ifdef __MPI
1717
,
18-
pool_comm_(MPI_COMM_SELF),
19-
kp_world_comm_(MPI_COMM_NULL),
20-
int_bgroup_comm_(MPI_COMM_SELF),
21-
bp_world_comm_(MPI_COMM_NULL),
22-
grid_comm_(MPI_COMM_NULL),
23-
diag_comm_(MPI_COMM_NULL)
18+
pw_world_comm_(MPI_COMM_SELF),
19+
kmesh_world_comm_(MPI_COMM_NULL),
20+
bsame_kdiff_world_comm_(MPI_COMM_SELF),
21+
bdiff_ksame_world_comm_(MPI_COMM_NULL),
22+
rgrid_world_comm_(MPI_COMM_NULL),
23+
diag_world_comm_(MPI_COMM_NULL),
24+
matrix_world_comm_(MPI_COMM_NULL),
25+
atom_world_comm_(MPI_COMM_NULL)
2426
#endif
2527
{
2628
}
@@ -32,17 +34,19 @@ ProcessTopology::ProcessTopology(int world_nproc_in,
3234
int rank_in_pool_in,
3335
const std::vector<int>& nproc_in_pool_in,
3436
int bndpar_in,
35-
int my_bndgroup_in,
36-
int rank_in_bgroup_in,
37-
int nproc_in_bgroup_in
37+
int my_band_group_in,
38+
int rank_in_band_group_in,
39+
int nproc_in_band_group_in
3840
#ifdef __MPI
3941
,
40-
MPI_Comm pool_comm_in,
41-
MPI_Comm kp_world_comm_in,
42-
MPI_Comm int_bgroup_comm_in,
43-
MPI_Comm bp_world_comm_in,
44-
MPI_Comm grid_comm_in,
45-
MPI_Comm diag_comm_in
42+
MPI_Comm pw_world_comm_in,
43+
MPI_Comm kmesh_world_comm_in,
44+
MPI_Comm bsame_kdiff_world_comm_in,
45+
MPI_Comm bdiff_ksame_world_comm_in,
46+
MPI_Comm rgrid_world_comm_in,
47+
MPI_Comm diag_world_comm_in,
48+
MPI_Comm matrix_world_comm_in,
49+
MPI_Comm atom_world_comm_in
4650
#endif
4751
)
4852
: world_nproc_(world_nproc_in),
@@ -52,17 +56,19 @@ ProcessTopology::ProcessTopology(int world_nproc_in,
5256
rank_in_pool_(rank_in_pool_in),
5357
nproc_in_pool_(nproc_in_pool_in),
5458
bndpar_(bndpar_in),
55-
my_bndgroup_(my_bndgroup_in),
56-
rank_in_bgroup_(rank_in_bgroup_in),
57-
nproc_in_bgroup_(nproc_in_bgroup_in)
59+
my_band_group_(my_band_group_in),
60+
rank_in_band_group_(rank_in_band_group_in),
61+
nproc_in_band_group_(nproc_in_band_group_in)
5862
#ifdef __MPI
5963
,
60-
pool_comm_(pool_comm_in),
61-
kp_world_comm_(kp_world_comm_in),
62-
int_bgroup_comm_(int_bgroup_comm_in),
63-
bp_world_comm_(bp_world_comm_in),
64-
grid_comm_(grid_comm_in),
65-
diag_comm_(diag_comm_in)
64+
pw_world_comm_(pw_world_comm_in),
65+
kmesh_world_comm_(kmesh_world_comm_in),
66+
bsame_kdiff_world_comm_(bsame_kdiff_world_comm_in),
67+
bdiff_ksame_world_comm_(bdiff_ksame_world_comm_in),
68+
rgrid_world_comm_(rgrid_world_comm_in),
69+
diag_world_comm_(diag_world_comm_in),
70+
matrix_world_comm_(matrix_world_comm_in),
71+
atom_world_comm_(atom_world_comm_in)
6672
#endif
6773
{
6874
assert(world_nproc_ >= 1);
@@ -80,10 +86,10 @@ ProcessTopology::ProcessTopology(int world_nproc_in,
8086
assert(rank_in_pool_ >= 0 && rank_in_pool_ < nproc_in_pool_[my_pool_]);
8187

8288
assert(bndpar_ >= 1);
83-
assert(my_bndgroup_ >= 0 && my_bndgroup_ < bndpar_);
84-
assert(nproc_in_bgroup_ >= 1);
85-
assert(bndpar_ * nproc_in_bgroup_ == world_nproc_);
86-
assert(rank_in_bgroup_ >= 0 && rank_in_bgroup_ < nproc_in_bgroup_);
89+
assert(my_band_group_ >= 0 && my_band_group_ < bndpar_);
90+
assert(nproc_in_band_group_ >= 1);
91+
assert(bndpar_ * nproc_in_band_group_ == world_nproc_);
92+
assert(rank_in_band_group_ >= 0 && rank_in_band_group_ < nproc_in_band_group_);
8793
}
8894

8995
int ProcessTopology::pool_root_rank(int pool) const
@@ -99,3 +105,31 @@ int ProcessTopology::pool_root_rank(int pool) const
99105
}
100106
return offset;
101107
}
108+
109+
int ProcessTopology::band_group_root_rank(int band_group) const
110+
{
111+
if (band_group < 0 || band_group >= bndpar_)
112+
{
113+
return -1;
114+
}
115+
// In ABACUS divide_pools the band-group layout over world ranks is
116+
// stripe-contiguous inside each k pool: within pool P the first
117+
// (nproc_in_pool[P]/bndpar) ranks belong to band-group 0, the next
118+
// slice to band-group 1, and so on. The first rank of the
119+
// concatenated "same band-group across all pools" set (i.e. the
120+
// root of bsame_kdiff_world for that band-group) is therefore the
121+
// first occurrence in pool 0, which falls at offset band_group *
122+
// (nproc_in_pool[0]/bndpar) from pool_root_rank(0). The invariant
123+
// bndpar_ * nproc_in_band_group_ == world_nproc_ + the even split
124+
// enforced by MPICommGroup::divide_group_comm make that offset
125+
// equal to (band_group * nproc_in_band_group_) directly because
126+
// each band-group contains exactly nproc_in_band_group_ processes
127+
// globally and they appear in ascending band-group id order in
128+
// world rank when scanned pool by pool.
129+
const int per_bg_in_pool0 = nproc_in_pool_[0] / bndpar_;
130+
assert(per_bg_in_pool0 * bndpar_ == nproc_in_pool_[0]);
131+
const int via_pool0 = band_group * per_bg_in_pool0;
132+
const int via_global = band_group * nproc_in_band_group_;
133+
assert(via_pool0 == via_global);
134+
return via_global;
135+
}

source/source_base/parallel_topology.h

Lines changed: 75 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -12,16 +12,45 @@
1212
*
1313
* Replaces the six raw-global MPI_Comm (POOL_WORLD / KP_WORLD /
1414
* INT_BGROUP / BP_WORLD / GRID_WORLD / DIAG_WORLD) together with the
15-
* associated pool / band-group rank & size info from GlobalV.
16-
* Values captured at construction time; the object is copyable
17-
* (value semantics) and exposes no mutable workflow switches.
15+
* associated pool / band-group rank & size info from GlobalV, and
16+
* also exposes explicit handles for two long-implicit domains:
17+
*
18+
* - matrix_world_comm: communicator backing the 2D block-cyclic
19+
* BLACS grid used by Parallel_2D / Parallel_Orbitals to lay out
20+
* distributed H/S/projector matrices. Historically callers chose
21+
* pw_world / diag_world / MPI_COMM_WORLD ad-hoc; here we give the
22+
* concept a first-class name so call sites inject the appropriate
23+
* view explicitly in step 2.
24+
* - atom_world_comm : communicator backing the 3D Cartesian domain
25+
* decomposition used by the MD / neighlist
26+
* DomainDecomposition to distribute atoms spatially. Today the
27+
* concrete communicator is always MPI_COMM_WORLD (wrapped via the
28+
* CommunicationDomain thin shell in parallel_cell.h).
29+
*
30+
* All 8 communicator handles share the consistent `_world_comm`
31+
* suffix. The pair (bdiff_ksame_world, bsame_kdiff_world) is named
32+
* directly after the relation between band side and k side:
1833
*
34+
* bdiff_ksame -> band groups differ, k is the same
35+
* (legacy BP_WORLD, intra-pool cross-band pairs)
36+
* bsame_kdiff -> band groups are identical, k differs
37+
* (legacy INT_BGROUP, intra-band-group union across
38+
* all k pools)
39+
*
40+
* Values are captured at construction time; the object is copyable
41+
* (value semantics) and exposes no mutable workflow switches.
1942
* Consumers currently reading the global MPI_Comm / GlobalV fields
2043
* should migrate to taking a `const ProcessTopology&` from their
2144
* callers. Parallel_Global::create_topology(...) produces one such
2245
* instance at startup whose communicators are also aliased back to
2346
* the legacy globals so existing code keeps working during the
2447
* migration.
48+
*
49+
* matrix_world_comm / atom_world_comm are not derived inside the
50+
* topology constructor because the correct choice depends on where
51+
* the view is used (LCAO diag, GK diag, MD step, ...). Callers that
52+
* need a proper matrix or atom domain are expected to fill in the
53+
* handle before passing the topology down to distributed modules.
2554
*/
2655
class ProcessTopology
2756
{
@@ -36,6 +65,9 @@ class ProcessTopology
3665
* All inputs are plain values; GlobalV is never read inside the
3766
* constructor. Pass MPI_COMM_NULL for communicators that are
3867
* unused on this rank (same semantics as legacy divide_pools).
68+
* matrix_world_comm / atom_world_comm default to MPI_COMM_NULL
69+
* and are filled in later by whichever call site knows which
70+
* domain view is needed for a given distributed calculation.
3971
*/
4072
ProcessTopology(int world_nproc_in,
4173
int my_rank_in,
@@ -44,25 +76,27 @@ class ProcessTopology
4476
int rank_in_pool_in,
4577
const std::vector<int>& nproc_in_pool_in,
4678
int bndpar_in,
47-
int my_bndgroup_in,
48-
int rank_in_bgroup_in,
49-
int nproc_in_bgroup_in
79+
int my_band_group_in,
80+
int rank_in_band_group_in,
81+
int nproc_in_band_group_in
5082
#ifdef __MPI
5183
,
52-
MPI_Comm pool_comm_in,
53-
MPI_Comm kp_world_comm_in,
54-
MPI_Comm int_bgroup_comm_in,
55-
MPI_Comm bp_world_comm_in,
56-
MPI_Comm grid_comm_in,
57-
MPI_Comm diag_comm_in
84+
MPI_Comm pw_world_comm_in,
85+
MPI_Comm kmesh_world_comm_in,
86+
MPI_Comm bsame_kdiff_world_comm_in,
87+
MPI_Comm bdiff_ksame_world_comm_in,
88+
MPI_Comm rgrid_world_comm_in,
89+
MPI_Comm diag_world_comm_in,
90+
MPI_Comm matrix_world_comm_in = MPI_COMM_NULL,
91+
MPI_Comm atom_world_comm_in = MPI_COMM_NULL
5892
#endif
5993
);
6094

6195
// world ----------------------------------------------------------
6296
int world_size() const { return world_nproc_; }
6397
int world_rank() const { return my_rank_; }
6498

65-
// k-point pools -------------------------------------------------
99+
// k-point pools --------------------------------------------------
66100
int kpar() const { return kpar_; }
67101
int my_pool() const { return my_pool_; }
68102
int rank_in_pool() const { return rank_in_pool_; }
@@ -71,20 +105,26 @@ class ProcessTopology
71105
/// World rank of the root (rank 0) of a given pool; -1 on bad index.
72106
int pool_root_rank(int pool) const;
73107

74-
// band groups (bndpar) ------------------------------------------
108+
// band groups (BNDPAR) ------------------------------------------
75109
int bndpar() const { return bndpar_; }
76-
int my_bndgroup() const { return my_bndgroup_; }
77-
int rank_in_bgroup() const { return rank_in_bgroup_; }
78-
int nproc_in_bgroup() const { return nproc_in_bgroup_; }
110+
int my_band_group() const { return my_band_group_; }
111+
int rank_in_band_group() const { return rank_in_band_group_; }
112+
int nproc_in_band_group() const { return nproc_in_band_group_; }
113+
/// World rank of the root (rank 0) inside a band-group union.
114+
/// Uses the same prefix-sum logic as pool_root_rank but over the
115+
/// band-group partition of MPI_COMM_WORLD. -1 on invalid index.
116+
int band_group_root_rank(int band_group) const;
79117

80118
#ifdef __MPI
81-
// communicators -------------------------------------------------
82-
MPI_Comm pool_comm() const { return pool_comm_; }
83-
MPI_Comm kp_world_comm() const { return kp_world_comm_; }
84-
MPI_Comm int_bgroup_comm() const { return int_bgroup_comm_; }
85-
MPI_Comm bp_world_comm() const { return bp_world_comm_; }
86-
MPI_Comm grid_comm() const { return grid_comm_; }
87-
MPI_Comm diag_comm() const { return diag_comm_; }
119+
// communicators (consistent `_world_comm` suffix) ---------------
120+
MPI_Comm pw_world_comm() const { return pw_world_comm_; }
121+
MPI_Comm kmesh_world_comm() const { return kmesh_world_comm_; }
122+
MPI_Comm bsame_kdiff_world_comm() const { return bsame_kdiff_world_comm_; }
123+
MPI_Comm bdiff_ksame_world_comm() const { return bdiff_ksame_world_comm_; }
124+
MPI_Comm rgrid_world_comm() const { return rgrid_world_comm_; }
125+
MPI_Comm diag_world_comm() const { return diag_world_comm_; }
126+
MPI_Comm matrix_world_comm() const { return matrix_world_comm_; }
127+
MPI_Comm atom_world_comm() const { return atom_world_comm_; }
88128
#endif
89129

90130
private:
@@ -97,17 +137,19 @@ class ProcessTopology
97137
std::vector<int> nproc_in_pool_;
98138

99139
int bndpar_ = 1;
100-
int my_bndgroup_ = 0;
101-
int rank_in_bgroup_ = 0;
102-
int nproc_in_bgroup_ = 1;
140+
int my_band_group_ = 0;
141+
int rank_in_band_group_ = 0;
142+
int nproc_in_band_group_ = 1;
103143

104144
#ifdef __MPI
105-
MPI_Comm pool_comm_ = MPI_COMM_SELF;
106-
MPI_Comm kp_world_comm_ = MPI_COMM_NULL;
107-
MPI_Comm int_bgroup_comm_ = MPI_COMM_SELF;
108-
MPI_Comm bp_world_comm_ = MPI_COMM_NULL;
109-
MPI_Comm grid_comm_ = MPI_COMM_NULL;
110-
MPI_Comm diag_comm_ = MPI_COMM_NULL;
145+
MPI_Comm pw_world_comm_ = MPI_COMM_SELF;
146+
MPI_Comm kmesh_world_comm_ = MPI_COMM_NULL;
147+
MPI_Comm bsame_kdiff_world_comm_ = MPI_COMM_SELF;
148+
MPI_Comm bdiff_ksame_world_comm_ = MPI_COMM_NULL;
149+
MPI_Comm rgrid_world_comm_ = MPI_COMM_NULL;
150+
MPI_Comm diag_world_comm_ = MPI_COMM_NULL;
151+
MPI_Comm matrix_world_comm_ = MPI_COMM_NULL;
152+
MPI_Comm atom_world_comm_ = MPI_COMM_NULL;
111153
#endif
112154
};
113155

0 commit comments

Comments
 (0)