Table 4-1 shows the possible experiments you can perform using the SpeedShop tools and the reasons why you might want to choose a specific experiment. The Clues column shows when you might use an experiment. The Data Collected column indicates performance data collected by the experiment. For detailed information on the experiments, see the relevant section in the remainder of this chapter.

A floating-point exception trace collects each floating-point exception with the exception type and the call stack at the time of the exception. Floating-point exception tracing experiments should incur a slowdown in execution of the program of no more than 15%. These measurements are exact, not statistical.

The prof command generates a report that shows inclusive and exclusive floating-point exception counts.

If you are running a heap trace experiment (heap) on a multiprocessor application, you will get an experiment file for each process and an additional experiment file for the master process. Each process experiment file can either contain a sample of the data from the whole application or its own data only, as follows:

In the SpeedShop hardware counter experiments, overflows of a particular hardware counter are recorded. Each hardware counter is configured to count from zero to a number designated as the overflow value. When the counter reaches the overflow value, the system resets it to zero and increments the number of overflows at the present program instruction address. Each experiment provides two possible overflow values; the values are prime numbers, so any profiles that seem the same for both overflow values should be statistically valid.

The experiments described in this section are available for systems that have hardware counters (R10000, R12000, R14000, and R16000 class machines). Hardware counters allow you to count various types of events, such as cache misses and counts of issued and graduated instructions.

A hardware counter works as follows: for each event, the appropriate hardware counter is incremented on the processor clock cycle. For example, when a floating-point instruction is graduated in a cycle, the graduated floating-point instruction counter is incremented by 1.

These experiments are detailed by nature. They return information gathered at the hardware level. You probably want to run a higher level experiment first. Once you have narrowed the scope, you can use hardware counter experiments to pinpoint the area to be tuned.

The bbcounts experiment displays an estimated time based on linear basic block counting.

Data is measured by counting the number of executions of each basic block and calculating an estimated time for each function. This involves instrumenting the program to divide the code into basic blocks, which are consecutive sequences of instructions with a single entry point, a single exit point, and no branches into or out of the sequence. Instrumentation also records a count of all dynamic (function-pointer) calls.

Because an exact count of every instruction in your program is recorded, you can also use the bbcounts experiment to determine the efficiency of your algorithm and identify any code that is not executed.

The I/O trace experiment shows you the level of I/O activity in your program by tracing various I/O system calls, for example read(2) and write(2).

The prof output of an I/O trace experiment yields the following information:

The following ssrun command creates an I/O trace experiment file from the executable file generic:

% ssrun -io generic

The mpi trace experiment traces calls to various MPI routines and generates a file that is viewable in prof. For a list of the routines that are traced, see the ssrun man page.

The mpi_trace experiment traces calls to various MPI routines and generates a file that is viewable in the cvperf Performance Analyzer window. For a list of the routines that are traced, see the ssrun man page (this experiment will be removed in a future release).

For more details about MPI experiments, see “Running Experiments on MPI Programs” in Chapter 6.

The NUMA profiler operates on a statistical basis by periodically interrupting the running application. During each interrupt the application's memory accesses are examined. Interrupts are triggered periodically by waiting for a particular number of CPU hardware performance counter events to occur. For example, the default setting is to interrupt the running application after approximately 100 secondary data cache misses have occurred.

During each interrupt, the profiler begins at the interrupted program counter location and finds the nearest memory access for which it can accurately calculate a target address. This address is then used to determine the ccNUMA node that contains the memory being accessed. The profiler also determines which ccNUMA node is executing the interrupted application thread.

Each time a sample is taken, the following information is stored in the experiments data file:

This sampling process is repeated continuously until the application terminates.

The generated data can then be used to locate those lines of application code that generate the largest number of remote memory accesses ('remote' refers to the situation in which a node is performing an access to memory that does not lie on that node). By using facilities already present in IRIX (namely CPU sets and memory locality domains) the application engineer can attempt to minimize these remote accesses.

Applications running on a ccNUMA system do not see the same memory latency for every memory access - even after accounting for all cache effects. Accessing memory located on the same node as that on which you run your code is faster than accessing memory on other nodes. By reducing the number of remote memory accesses the application's performance is enhanced.

The pcsamp experiment estimates the actual CPU time for each source code line, machine code line, and function in your program. The prof listing of this experiment shows exclusive PC sampling time. This experiment is a lightweight, high-speed operation that makes use of the operating system.

CPU time is calculated by multiplying the number of times an instruction or function appears in the PC by the interval specified for the experiment (either 1 or 10 milliseconds).

To collect the data, the operating system regularly stops the process, increments a counter corresponding to the current value of the PC, and resumes the process. The default sample interval is 10 milliseconds. If you specify the optional f prefix to the experiment, a sample interval of 1 millisecond is used. (See “A pcsamp Experiment” in Chapter 2, for an example.)

By default, the experiment uses 16-bit counters. If the optional x suffix is used, a 32-bit counter size will be used. Using a 32-bit bin provides more accurate information, but requires additional memory and disk space. See “Example Using the pcsampx Experiment” in Chapter 6, for an example.

PC sampling runs should slow the execution time of the program down no more than 5 percent. The measurements are statistical in nature, meaning they exhibit variance inversely proportional to the running time.

The usertime and the totaltime experiments are useful experiments to start your performance analysis. The usertime experiment returns CPU time for each function while your program runs and the totaltime experiment returns real time for each function.

These expeirments use statistical call stack profiling to measure inclusive and exclusive user time. They take a sample every 30 milliseconds. Data is measured by periodically sampling the callstack. The program's callstack data is used to do the following:

The time spent in a procedure is determined by multiplying the number of times an instruction for that procedure appears in the stack by the sampling time interval between call stack samples. Call stacks are gathered when the program is running; hence, the time computed represents user time, not time spent when the program is waiting for a CPU. User time shows both the time the program itself is executing and the time the operating system is performing services for the program, such as I/O.

The usertime experiment should incur a program execution slowdown of no more than 15%. Data from a usertime experiment is statistical in nature and shows some variance from run to run.

---

Note: For this experiment, o32 executables must explicitly link with -lexc.

---