Chapter 4. Debugging MPI Applications
Prev |  | Next

## Chapter 4. Debugging MPI Applications

Debugging MPI applications can be more challenging than debugging sequential applications. This chapter presents methods for debugging MPI applications.

## MPI Routine Argument Checking

By default, the SGI MPI implementation does not check the arguments to some performance critical MPI routines such as most of the point-to-point and collective communication routines. You can force MPI to always check the input arguments to MPI functions by setting the `MPI_CHECK_ARGS` environment variable. However, setting this variable might result in some degradation in application performance, so it is not recommended that it be set except when debugging.

## Using the ProDev (CVD) Debugger with MPI Programs

| Note: The ProDev CASEvision debugger (CVD) is available on IRIX systems only.

Recent versions of the ProDev (CVD) debugger work well with MPI jobs running within a single host. You can use CVD to debug MPI applications that make use of MPI-2 spawn functions. To use the CVD debugger, perform the following steps:

Procedure 4-1. Steps for Using CVD

1. Use the following command to bring up the CVD debugger :

       cvd /usr/bin/mpirun

2. When CVD comes, up click on the Admin menu and select Multiprocess View.

3. When Multiprocess View appears, click on the Config menu, then the Preferences menu.

4. When the Preferences menu appears, check the first two unchecked boxes and click on OK. So that you do not need to set these menus the next time you bring up CVD, you can click on the Save button.

5. In the CVD command window (bottom of the CVD window), enter the following commands:

       cvd> set $pendingtraps=true
       cvd> stop pgrp all in MPI_SGI_init

   (You can also use a function in your `a.out`) file.

6. In the CVD command window (top of the CVD window), enter the `mpirun` command with arguments, as in the following example:

       /usr/bin/mpirun -np 2 a.out

   Then click on the Run button.

7. Watch the Multiprocess View window as it forks processes. Eventually, it stops in `MPI_SGI_init` (or your function) in your program and CVD focuses on it. If you compiled with the `-g` option, it shows the source.

8. If you did not compile with the `-g` option, you can execute a `file` command to select a certain file and see the source, as in the following example:

       file ep.f

### Setting Breakpoints

The `pgrp` attribute on the `stop` command (see Step 5 above) indicates the setting of the breakpoint for any processes in the Multiprocess View window (including ones that will be spawned as slaves). You can set breakpoints by clicking just to the left of the line, but by default, they are for that particular process, not all processes in the Multiprocess View window. You can change the process to add `pgrp` by clicking on the Traps menu in CVD and selecting both of the unchecked boxes. Note that when Group Trap Default is set, the `pgrp` attribute is added and when Stop All Default is set, the `all` attribute is added. The `all` attribute stops all processes in the Multiprocess View window when any process hits this breakpoint.

### Finding Windows

To find various windows, use the Views menu. Call Stack and Trap Manager windows are very useful. You can also type dbx commands in the `cvd` command window at the bottom of the CVD window.

### Continuing and Stepping Processes

The buttons in the Multiprocess View window cause all processes to continue or step. Typically, you will want to use these. The buttons in the CVD main window are for a single process, unless a button indicates All. You can set up the viewing of line numbers from the Display menu.

### Rerunning a Process

If you want to rerun the process, simply click on the Run button again. To temporarily turn off breakpoints, use the Traps menu in the Trap Manager window.

## Using TotalView with MPI programs

The syntax for running SGI MPI with Etnus' TotalView is as follows:

    totalview mpirun -a -np 4 a.out

Note that TotalView is not expected to operate with MPI processes started via the `MPI_Comm_spawn` or `MPI_Comm_spawn_multiple` functions.

| Note: The TotalView debugger for Linux does not work with SGI MPI.

## Using dbx and gdb with MPI programs

Because the dbx and gdb debuggers are designed for sequential, non-parallel applications, they are generally not well suited for use in MPI program debugging and development. However, the use of the `MPI_SLAVE_DEBUG_ATTACH` environment variable makes these debuggers more usable. If you set the `MPI_SLAVE_DEBUG_ATTACH` environment variable to a global rank number, the MPI process sleeps briefly in startup while you use dbx or gdb to attach to the process. A message is printed to the screen, telling you how to use dbx to attach to the process. Similarly, if you want to debug the MPI daemon, setting `MPI_DAEMON_DEBUG_ATTACH` sleeps the daemon briefly while you attach to it. Both of these environment variables are available on IRIX and Linux.

Prev | Table of Contents | Next
Chapter 3. Programming with SGI MPI |  | Chapter 5. Profiling MPI Applications