Debugging MPI applications can be more challenging than debugging sequential applications. This chapter presents methods for debugging MPI applications.
By default, the SGI MPI implementation does not check the arguments to some performance-critical MPI routines such as most of the point-to-point and collective communication routines. You can force MPI to always check the input arguments to MPI functions by setting the MPI_CHECK_ARGS environment variable. However, setting this variable might result in some degradation in application performance, so it is not recommended that it be set except when debugging.
 | Note: The ProDev WorkShop debugger (also known as CVD) is available on IRIX systems only. |
Recent versions of the ProDev WorkShop debugger work well with MPI jobs running within a single host. You can use Debugger to debug MPI applications that make use of MPI-2 spawn functions. To use the Debugger, perform the following steps:
Procedure 4-1. Steps for Using the Debugger
Use the following command to bring up the Debugger:
% cvd /usr/bin/mpirun
When the Debugger comes, up click on the Admin menu and select Multiprocess View.
When Multiprocess View appears, click on the Config menu, then the Preferences menu.
When the Preferences menu appears, check the first two unchecked boxes and click on OK. So that you do not need to set these menus the next time you bring up the Debugger, you can click on the Save button.
In the Debugger command window (bottom of the main window), enter the following commands:
cvd> set $pendingtraps=true cvd> stop pgrp all in MPI_SGI_init
(You can also use a function in your a.out) file.
In the command window (the top of the main window), enter the mpirun command with arguments, as in the following example:
/usr/bin/mpirun -np 2 a.out
Then click on the Run button.
Watch the Multiprocess View window as it forks processes. Eventually, it stops in MPI_SGI_init (or your function) in your program and the Debugger focuses on it. If you compiled with the -g option, it shows the source.
If you did not compile with the -g option, you can execute a file command to select a certain file and see the source, as in the following example:
file ep.f
For complete details about using the Debugger, see the ProDev WorkShop: Debugger User's Guide.
The pgrp attribute on the stop command (see Step 5 above) indicates the setting of the breakpoint for any processes in the Multiprocess View window (including ones that will be spawned as slaves). You can set breakpoints by clicking just to the left of the line, but by default, they are for that particular process, not all processes in the Multiprocess View window.
You can change the process to add pgrp by clicking on the Traps menu in the Debugger and selecting both of the unchecked boxes. Note that when Group Trap Default is set, the pgrp attribute is added and when Stop All Default is set, the all attribute is added. The all attribute stops all processes in the Multiprocess View window when any process hits this breakpoint.
To find various windows, use the Views menu. Call Stack and Trap Manager windows are very useful. You can also type dbx commands in the Debugger command window at the bottom of the main window.
The syntax for running SGI MPI with Etnus' TotalView is as follows:
% totalview mpirun -a -np 4 a.out |
Note that TotalView is not expected to operate with MPI processes started via the MPI_Comm_spawn or MPI_Comm_spawn_multiple functions.
Because the dbx and gdb debuggers are designed for sequential, non-parallel applications, they are generally not well suited for use in MPI program debugging and development. However, the use of the MPI_SLAVE_DEBUG_ATTACH environment variable makes these debuggers more usable.
If you set the MPI_SLAVE_DEBUG_ATTACH environment variable to a global rank number, the MPI process sleeps briefly in startup while you use dbx or gdb to attach to the process. A message is printed to the screen, telling you how to use dbx to attach to the process.
Similarly, if you want to debug the MPI daemon, setting MPI_DAEMON_DEBUG_ATTACH sleeps the daemon briefly while you attach to it. Both of these environment variables are available on IRIX and Linux.