Using MPICH-PM on  Workstation/PC Clusters in the SCore Environment

Introduction

The version of MPI is called MPICH-PM and is based on MPICH version 1.0.11. It runs under the SCore and SCore-D operating systems on the workstation and PC clusters at RWCP. The underlying message passing system is PM running on top of Myrinet.

If your MPI program works under standard MPICH then it is easy to run under MPICH-PM.

Prerequisites: You need to know how to log in to and run programs on your PC cluster or Workstation Cluster, and use the Message Board, and SCore-S (or SCore-D).

MPICH-PM should have been compiled and installed properly when you installed the whole system. If there was trouble configuring it, see Configuring MPICH-PM.

Here are the steps required to compile run your program

  1. Compile and link with mpicc or mpif77
  2. Run with mpirun

1. Compiling the program

1.1 C programs

Use mpicc to both compile and link. This will link the correct SCore libraries for you. If your program will not link, see the portability note below.

1.2 C++ programs

Use mpic++ to both compile and link. This will link the correct SCore libraries for you.

1.3 FORTRAN 77 programs

Use mpif77 to both compile and link. This will link the correct SCore libraries for you.

Note that mpif77 is a shell wrapper which uses the f2c FORTRAN to C compiler. Currenly, MPICH-PM cannot use a native f77 compiler. F2c must be installed on your system.

2. Running the program

2.1 Environment

All programs must be run on the cluster under the SCore operating system. Therefore you must set up your environment as explained under SCore.

2.2 mpirun command

The mpirun command has been modified to run programs under the SCore envirionment. Shell scripts which use mpirun are portable between MPICH and MPICH-PM.

SCore-S:

Log on a cluster node and run the program with mpirun. For example, if your program needs 8 nodes:

SCore-D:

Log on to any appropriate node, and specify

where machinename is the name of the processor controlling SCore-D.

2.3 Without mpirun

SCore-S:

Log on a cluster node and run the program just like a normal SCore program. For example, if your program needs 8 nodes:

SCore-D:

Log on to any appropriate node, and specify

where machinename is the name of the processor controlling SCore-D. Note that there is just a comma, no space, between -score arguments.

2.4 Using the "Zero Copy" protocol

On some cluster hardware, such as the i386 under netbsd but not the Sun4 architecture, PM supports a Remote Memory Access (RMA) feature. RMA can be used by MPI to improve bandwidth with a so-called zero copy protocol (the hardware copies between user MPI buffers without any extra buffering or memory copies). This provides much higher point to point bandwidth and improved performance for some applications.

Specify the zero copy option in the following way:

or equivalently

You may also specify the mpizerocopy option in the SCORE_OPTIONS environment variable.

-score mpizerocopy is shorthand for -score mpizerocopy=on

You can prevent the use of the zero copy protocol by the use of -score mpizerocopy=off (or just by not specifying it).

The default is mpizerocopy=off

3. Portability Note

The file containing main() must #include <mpi.h> or the program will not link.

The SCore runtime libraries already use main(), but most MPI C and C++ programs under UNIX use main() as the start of execution. Therefore, the MPICH-PM library has been modified to call the function mpi_main() to start your program. For compatibility, the file mpi.h contains the lines

4. Bugs

If the linker cannot find the mpi_main routine, see above.

MPI can not be used within an MPC++ program.

SEE ALSO

mpic++(1), mpicc(1), mpif77(1), MPI_Init(3)


ocarroll@rwcp.or.jp Wed Jun 10 17:20:51 JST 1998