[internal] New execution interface and development plan

[llaniewski]

As things are getting hard to maintain I propose a change of overall approach of configuration and execution of TCLB.

These are notes for a possible direction of development

• Integrate TCLB_cluster into the main repo
• Make a common wrapper script for running all calculations
  • The model selection would be based on something line <CLBConfig model="...">
  • It would run both native and slurm
  • It would run the code coupling (eg. ESYS/TCLB) based on best practices.
  • Could generate batch script (without submitting it to the queue).
• Make machine specific alternation to configuration in a mechanism similar to the one done in TCLB_cluster
• Allow for easier execution of tests on different machines/architectures
• Make configuration more persistent (now you have to remember the ./configure options that you used on specific machine) (cmake has a similar mechanism, and can be considered as a solution)

The plan would be to:

• Close V6.3 including DEM and all nice proposed features (VTK for subregion #186 Make half-way bounce back easy to use #198)
• Begin V6.4 or V7.0 with Refactoring of all models #157 and maybe Make half-way bounce back easy to use #198 + Cleanup Geometry #125
• make the wrappers for configuration and execution and do Clean up configure #124 at the same time.

[mdzik]

I would vote for outer layer be done in python/bash/ruby as those are designed for it. It would also make integration into something "bigger" easier?

I've successfully created docker.io image with test suit for TCLB.
https://cloud.docker.com/repository/registry-1.docker.io/mdzik/tclb_testenv

so docker pull mdzik/tclb_testenv should work

[llaniewski]

I done some docker images here: https://hub.docker.com/r/cfdgo/tclb
If we want to make docker an option, we should do it properly. It's not easy as in docker there is no multiple-inheritance. This means that one cannot make something that inherist from both (example) cuda image and R image.
I did a configuration which works with the official nvidia cuda image some time ago, but I don't know if it is very useful. It doesn't work with GPU on windows. And to work with GPU on linux you have to have a non-standard installation of docker.

And on the side note non of the clusters work with docker. So I think docker is generally good, but only for testing. But testing GPU will still be a problem.

[llaniewski]

The dockerfiles are at feature/docker branch.

[mdzik]

Multiple inheritances is (i think) done by docker-compose.
Using docker images might be useful if one intends to do something inside Amazon AWS, but I only intend to create an environment for local tests evaluation, hence travis inheritance. I would like to finally pull csf model, but tests are holding me back ;)

[mdzik]

(nvidia-docker is workable with AWS)

[mdzik]

I've done some digging

One of the "proper" ways to distribute could be singularity image (https://sylabs.io/singularity/). It is somehow close to docker, but in scientific software in mind.

• it is supported on the Prometheus, it seems that it supports nvidia/cuda
• it is supported (and encouraged) at the ICM
• it could be nice to have integrated TCLB+*MPI+FlyingBalls as a package, with sorted out dependencies and inside preconfigured env. Build elsewhere

questions:

• GPU performance (i could check that on K20@Cyfronet and V100@ICM)
• MPMD performance ?
• how does it help? :)
• other?

[llaniewski]

@mdzik Could you make a proof-of-concept Singularity container with TCLB code? Let's say: OpenMPI+GCC, no fancy stuff. Let's start without CUDA, and progress from that.

[mdzik]

I'll try

The goal is to have (as far as I feel syntax):

$ module load singularity
$ singularity pull somehost/tclb.sif
$ singularity run tclb-gcc.sif  d2q9 ~/test/karman.xml

[mdzik]

Nie summary:
https://tin6150.github.io/psg/blogger_container_hpc.html

[mdzik]

hay, i've got this

singularity pull library://mdzik/tclb/tclb:latest
singularity verify ./tclb_latest.sif

then to use build-in TCLB:

singularity exec --nv  ./tclb_latest.sif /opt/TCLB_gpu/TCLB/CLB/d2q9/main  /opt/TCLB_gpu/TCLB/example/flow/2d/karman.xml

or to use in-container shell, with all Rtools etc, to build local tclb:

singularity -nv shell ./tclb_latest.sif

How do we proceed? :)