Mapping of MPMD hybrid MPI+OpenMP threads and MPI-only tasks efficiently onto cores on different architectures.¶
To get the best performance from a HPC system, each task or thread needs to be assigned to a specific core. Since EC-EARTH consists of three applications, of which NEMO and OASIS are pure MPI and IFS is MPI+OpenMP that are all running within the same MPI_COMM_WORLD communicator. No scheduler can automatically assign processes and threads in an optimal way for this complicated setup. Therefore, the user needs to do this manually. Unfortunately, there is no standardized method for this, so each platform requires a separate solution.
- Cray XE6:
- Curie Bullx:
- IBM Power6/7:
IBM provides unofficially the affinity tools 'launch' (for MPI-only programs) and 'hybrid_launch' (for MPI+OpenMP programs). These tools can be used to assign any combination of programs efficiently to a set of computing cores. The disadvantage is that the tool cannot automatically detect the use of 'launch' and 'hybrid_launch' on one node at the same time and set the affinity accordingly (but it can be done manually). However, Uwe mentioned that EC-EARTH is only run with only one binary per node and this situation is therefore not (yet) relevant. LoadLeveler allows for any assignment of MPI-tasks to nodes using the task_geometry keyword.
- IBM BlueGene/P: BG/P nodes do not have an OS and therefore processes and threads are assigned to cores at the start
and will not move around.
This work depends mostly on having ported EC-EARTH to all the relevant platforms. It is difficult as to how long this will take. It is hoped that the porting to the Power6 platform can be finished before the f2f meeting. Once that is done, the work needed is about one week for each platform. it is expected that initial work for the Power6, Curie and maybe Cray can be done during summer 2011. The other platforms will be done before December 2011 and reporting will be done before the end of 2011.* "Scaling, OpenMP, I/O
- results of EC-EARTH@T159-ORCA1 for meeting 7 dec 2011 (John Donners)":https://redmine.dkrz.de/collaboration/attachments/814/progress7dec2011.pdf