* Open an xterm window and ssh into the cluster's head node with X11-forwarding enabled. Depending on your settings, you may need to invoke the ssh client with "ssh -Y".
* On the cluster's head node, in your .cshrc or .bashrc file:
o Set up your module environment (e.g., module load mpich-debug)
o Set the environmental variable TVDSVRLAUNCHCMD to 'ssh'
o Add totalview (/opt/export/toolworks/totalview/bin) to your PATH environmental variable.
* Build your application as you normally would, but add the -g option to the mpicc command line for debugging.
* Create a PBS script file like the one you would use to run your MPI program, but which does not have an mpirun/mpiexec command in it. This will be used to allocate the nodes/processors.
* Type 'qsub -I -X
* When you get a compute node prompt, type:
mpirun -tv -machinefile ${PBS_NODEFILE} -np `wc -l <${PBS_NODEFILE}` \
* The TotalView windows should now be displayed by your X server.
o The code should be stopped at the beginning of your main program.
o You should see your source code, and can enter breakpoints, single-step through it, etc.
If you see assembly code, you most likely forgot to use the -g option when building your program.
o When you click on 'Go' for the first time, you will halt at a permanent MPIR_Breakpoint deep in the MPI_Init() code. Just click on 'Go' again to hit your first breakpoint.
* When done debugging, click on File | Exit to close the TotalView windows.
* Log out of the compute node session to end the PBS job.
