Potential Use Cases

[eleon]

Courtesy of @adammoody.

As a concrete use case, we might have a situation like:

• an application uses SCR and MPI
• SCR spawns a background thread per application process for data copies from node-local storage to parallel file system
• MPI spawns a background thread per application process for async progress

Ideally, those background threads would run on different cores than the main application thread to avoid contention. However, they could run on the same core as the main app thread if there are no spare cores available. The background threads could run on the same core together, since they are likely not CPU intensive.

Does the Quo Vadis interface provide a way to specify a situation like that?

We can use hints within qv_scope_create to accomodate this.

// worker_scope has been defined by the app

// Get the system cores
qv_scope_get(ctx, QV_SCOPE_SYSTEM, &sys_scope);

// Check if there are available PUs
qv_scope_nobjs_avail(ctx, sys_scope, QV_HW_OBJ_PU, &pus);

if (pus > 0)
   qv_scope_create(ctx, sys_scope, QV_HW_OBJ_PU, 1,
                   QV_SCOPE_ATTR_INCLUSIVE,
                   &sub_scope);
else
   sub_scope = worker_scope;

// Launch the thread(s)
qv_pthread_create(thread, attr, start_routine, arg,
                    subscope, ctx);

What we need to implement:

• QV_SCOPE_SYSTEM
• qv_pthread_create
• qv_scope_nobjs_avail
• Hints: Implement (and define) the hints that qv_scope_create can take. The INCLUSIVE (or shared) hint means that other workers may be running on the same resource (the opposite of exclusive). By default we should place threads using a BFS strategy and then fill up the cores if multiple hardware threads are available.

I guess both SCR and MPI would need to make QV calls?

Yes, the more components that use QV, the better placement and coordination.

[GuillaumeMercier]

What we need to implement:
* qv_pthread_create

I'm working on it.

[GuillaumeMercier]

* MPI spawns a background thread per application process for async progress

Interesting. It means then that we have to support the hybrid MPI + OpenMP + Pthreads model. I'm not sure that the current OpenMP-based implementation in QV supports this.

[samuelkgutierrez]

* MPI spawns a background thread per application process for async progress

Interesting. It means then that we have to support the hybrid MPI + OpenMP + Pthreads model. I'm not sure that the current OpenMP-based implementation in QV supports this.

Fine-grained MPI + OpenMP + Pthreads support would be fantastic. I'm aware of other use cases that would benefit from this capability, too.

[GuillaumeMercier]

One thing I'm thinking about it the ability to share a contex and/or a scope between OpenMP threads and Pthreads.
This is more or less the same issue as in #35 I guess.

[GuillaumeMercier]

I think we need functions in the like of qv_init/qv_finalize to support this.

[samuelkgutierrez]

I think we need functions in the like of qv_init/qv_finalize to support this.

Could you please elaborate on this?

[samuelkgutierrez]

Also, something to consider: would splitting up OpenMP and Pthread support help in any way?

[GuillaumeMercier]

Could you please elaborate on this?

Yes, I can: as we already discussed, MPI and OpenMP feature some kind of runtime system that can be relied upon and queried to. This is not the case with pthread and I will need to introduce some shared memory space that can contain some information the Pthread implementation will need. In the case of multi-paradigms programs (eg. MPI + OpenMP + Pthreads) then this shared space should be accessible by all "paradigms". My thinking is that having three separate supports/implemenations that are not aware of the others is not going to work.

[GuillaumeMercier]

Thus, having a generic qv_init call would allow for setting up this shared space regardless of the programming model.
But maybe it's a call that would need to be made only in the case of hybrid cases?

[samuelkgutierrez]

Could you please elaborate on this?

Yes, I can: as we already discussed, MPI and OpenMP feature some kind of runtime system that can be relied upon and queried to. This is not the case with pthread and I will need to introduce some shared memory space that can contain some information the Pthread implementation will need. In the case of multi-paradigms programs (eg. MPI + OpenMP + Pthreads) then this shared space should be accessible by all "paradigms". My thinking is that having three separate supports/implemenations that are not aware of the others is not going to work.

In that case, could one implement an internal abstraction that provides such mechanisms for Pthreads?

[GuillaumeMercier]

Yes, that is what I'm planning to do. But this internal abstraction would have to be shared eventually, wouldn't it?

[samuelkgutierrez]

Thus, having a generic qv_init call would allow for setting up this shared space regardless of the programming model. But maybe it's a call that would need to be made only in the case of hybrid cases?

Could we accomplish the same goal by implementing the missing machinery internally to QV?

[GuillaumeMercier]

Maybe. But how do you detect hybrid cases? And enable the support in these cases?

[samuelkgutierrez]

Yes, that is what I'm planning to do. But this internal abstraction would have to be shared eventually, wouldn't it?

Shared across tasks, yes; but I'm not convinced that we have to expose those details to the user.

[GuillaumeMercier]

And MPI + OpenMP is different from OpenMP + Pthreads IMHO.