Skip to content

Increase flexiblility of MPI task partition patterns #948

Open
@yizhang-yiz

Description

@yizhang-yiz

Summary:

Allow more partition patterns in MPI jobs, instead of single equally-divided job allocation pattern.

Description:

map_rect only allows equally-divided pattern for MPI job allocation. That is, in mpi_map_chunks the number of jobs in each node is calculated as total_number_of_jobs/total_number_of_nodes. This is assuming each job is small enough to be efficiently handled by a single node. For many problems this assumption is not met, such examples include user-supplied PDE solution routine. For such an application, we may only have a single job for each chain, but that job requires N nodes to process, where N>>1. Current mpi_map_chunks function will map this single job to a single node, and leave the rest N-1 node idle, instead of delegating the whole N nodes to the task(how the N nodes are used to perform the task depends on application).

A related issue is #940 , where we allow specifying communicator to construct mpi_cluster, so that a group of dedicated nodes can be used in a task-specific communication group.

For example, we may look at an input like the following.

functions {
   real[] user_function_based_on_metis(...);
  }
}
parameters {
real theta;
...
}
....
model {
  ....
  target += sum(map_rect(user_function_based_on_metis, beta, theta, xs, ys));
}

Since there is only one theta, no matte how many nodes we throw in, only the root will be used to perform the task of evaluating user_function_based_on_metis, the heavy-lifting function using theta to solve a problem using MPI partition produced by METIS. What SHOULD happen is ALL the nodes issued are used to evaluate this function.

TL;DR
Current mpi_map_chunks pattern is too rigid, we need fix loosen it.

Current Version:

v2.18.0

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions