Description
Summary:
Allow more partition patterns in MPI jobs, instead of single equally-divided job allocation pattern.
Description:
map_rect
only allows equally-divided pattern for MPI job allocation. That is, in mpi_map_chunks
the number of jobs in each node is calculated as total_number_of_jobs/total_number_of_nodes
. This is assuming each job is small enough to be efficiently handled by a single node. For many problems this assumption is not met, such examples include user-supplied PDE solution routine. For such an application, we may only have a single job for each chain, but that job requires N
nodes to process, where N>>1
. Current mpi_map_chunks
function will map this single job to a single node, and leave the rest N-1
node idle, instead of delegating the whole N
nodes to the task(how the N
nodes are used to perform the task depends on application).
A related issue is #940 , where we allow specifying communicator to construct mpi_cluster
, so that a group of dedicated nodes can be used in a task-specific communication group.
For example, we may look at an input like the following.
functions {
real[] user_function_based_on_metis(...);
}
}
parameters {
real theta;
...
}
....
model {
....
target += sum(map_rect(user_function_based_on_metis, beta, theta, xs, ys));
}
Since there is only one theta
, no matte how many nodes we throw in, only the root will be used to perform the task of evaluating user_function_based_on_metis
, the heavy-lifting function using theta
to solve a problem using MPI partition produced by METIS. What SHOULD happen is ALL the nodes issued are used to evaluate this function.
TL;DR
Current mpi_map_chunks
pattern is too rigid, we need fix loosen it.
Current Version:
v2.18.0