The timex utility is a useful troubleshooting tool when you are running a single application. For example:

timex -s application

The -s option reports total system activity (not just that due to the application) that occurred during the execution interval of application. To redirect timex output to a file, (assuming you use the Bourne shell, ( sh(1)) enter:

timex -s application 2> file

The same command, entered using the C shell, looks like this:

timex -s application > file

The sar utility is a useful troubleshooting tool when you are running many programs and processes and/or have a multiuser system such as a server. You can take a sample of the operating system counters over a period of time (for a day, a few days, or a week).

Depending on your needs, you can choose the way in which you examine system activity. You can monitor the system:

You can set up the system so that sar automatically collects system activity data and puts it into files for you. Use the chkconfig command to turn on sar's automatic reporting feature, which generates a sar -A listing. A crontab entry instructs the system to sample the system counters every 20 minutes during working hours and every hour at other times for the current day (data is kept for the last 7 days). To enable this feature, type:

/etc/chkconfig sar on

The collected data is put in /var/adm/sa in the form sann and sarnn, where nn is the date of the report (sarnn is in ASCII format). You can use the sar(1M) command to output the results of system activity.

You can use sar to generate consecutive reports about the current state of the system. On the command line, specify a time interval and a count. For example:

sar -u 5 8

This prints information about CPU use eight times at five-second intervals.

You may find it useful to take a snapshot of the system activity counters before and after running an application (or after running several applications concurrently). To take a snapshot of system activity, instruct sadc (the data collector) to dump its output into a file. Then run the application(s) either under normal system load or restricted load, and when you are ready to stop recording, take another snapshot of system activity. Then compare results to see what happened.

Following is an example of commands that samples the system counters before and after the application:

/usr/lib/sa/sadc 1 1 file

Run the application(s) or perform any work you want to monitor, then type:

/usr/lib/sa/sadc 1 1 file 
sar -f file

If file does not exist, sadc creates it. If it does exist, sadc appends data to it.

Often you want to examine system activity during the execution of a command or set of commands. For example, to examine all system activity while running nroff(1), type:

/usr/lib/sa/sadc 1 1 sa.out 
nroff -mm file.mm > file.out 
/usr/lib/sa/sadc 1 1 sa.out 
sar -A -f sa.out 

By using timex, you can do the same thing with one line of code:

timex -s nroff -mm file.mm > file.out

Note that the timex also includes the real, user, and system time spent executing the nroff request.

There are two minor differences between timex and sar. The sar program can limit its output (such as, the -u option reports only CPU activity), while timex always prints the -A listing. Also, sar works in a variety of ways, but timex only works by executing a command—however, the command can be a shell file.

If you are interested in system activity during the execution of two or more commands running concurrently, put the commands into a shell file and run timex -s on the file. For example, suppose the file nroff.sh contained the following lines:

nroff -mm file1.mm > file1.out &
nroff -mm file2.mm > file2.out &
wait

To get a report of all system activity after both of the nroff requests (running concurrently) finish, invoke timex as follows:

timex -s nroff.sh

You can use par in much as you use sar:

See the par(1) man page for specifics on usage.

Use par instead of sar when you want a finer look at a suspect or problem process. Instead of simply telling you how much total time was used while your process was executing, like timex, par breaks down the information so you can get a better idea of what parts of the process are consuming time. In particular, use the following command options:

When tracing system calls, par prints a report showing all system calls made by the subject processes complete with arguments and return values. In this mode, par also reports all signals delivered to the subject processes. In schedule tracing mode, par prints a report showing all scheduling events taking place in the system during the measurement period. The report shows each time a process is put on the run queue, started on a processor, and unscheduled from a processor, including the reason that the process was unscheduled. The events include timestamps. You can set up the system so par automatically collects system activity data and puts it into files for you.

The par utility works by processing the output of the padc(1)man page. This can be done in two ways:

The par utility can provide different types of reports from a given set of padc data depending on the reporting options that are specified. This is a reason why it is sometimes desirable to run the data collection as a separate step.

Now that you have learned when and how to use par, sar, and timex, you can choose one of these utilities to monitor the operating system. Then examine the output and try to determine what is causing performance degradation. Look for numbers that show large fluctuation or change over a sustained period; do not be too concerned if numbers occasionally go beyond the maximum.

The first thing to check is how the system is handling the disk I/O process. After that, check for excessive paging/swapping. Finally look at CPU use and memory allocation.

The following sections assume that the system you are tuning is active (with applications/benchmark executing).

If you are going to run a large application or have a heavy system load, the system benefits from disk I/O tuning. Run sar -A or timex -s and look at the %busy, %rcache, %wcache, and %wio fields. To see if your disk subsystem needs tuning, check your output of sar -A against the figures in Table 10-4. (Note that in this table, the right column lists the sar option that prints only selected output, for example, output for disk usage (sar -d) or CPU activity (sar -u).)

Table 10-4 lists sar results that indicate an I/O-bound system.

Notice that for the %wio figures (indicates the percentage of time the CPU is idle while waiting for disk I/O), there are examples of two types of systems:

There are many other factors to consider when you tune for maximum I/O performance. You may also be able to increase performance by:

By using logical volumes, you can improve disk I/O:

Striping works best on disks that are on different controllers. Logical volumes give you more space without remaking the first filesystem. Disk striping gives you more space with increased performance potential, but you run the risk that if you lose one of the disks with striped data, you lose all the data on the filesystem, since the data is interspersed across all the disks.

Contiguous logical volumes fill up one disk, and then write to the next. Striped logical volumes write to both disks equally, spreading each file across all disks in the volume. It is impossible to recover from a bad disk if the data is striped, but it is possible if the data is in a contiguous logical volume. For information on creating a striped disk volume, see IRIX Admin: Disks and Filesystems .

There are obvious ways to increase your system's throughput, such as limiting the number of programs that can run at peak times, shifting processes to non-peak hours (run batch jobs at night), and shifting processes to another system. You can also set up partitions on separate disks to redistribute the disk load or add disks.

Before continuing with the discussion about partitions, look at how a program uses a disk as it executes. Table 10-5 shows various reasons why an application may need to access the disk.

You can maximize I/O performance by using separate partitions on different disks for some of the disk access areas. In effect, you are spreading out the application's disk access routines, which speeds up I/O.

By default, disks are partitioned to allow access in one of two ways:

On the system disk, partition 0 is for root and partition 1 is for swap.

---

Note: On older systems, disks may have three partitions: partitions 0, 1, and 6. On the system disk, partition 0 is for root, 1 is for swap, and 6 is for /usr. If there is one large partition, it encompasses the three smaller partitions.

---

For each additional disk, decide if you want a number of partitions or one large one and the filesystems (or swap) you want on each disk and partition. It is best to distribute filesystems in the disk partitions so that different disks are being accessed concurrently.

The configuration depends on how you use the system; so it helps to look at a few examples.

If you have a very large application and have three disks, consider using partitions on the second and third disks for the application's executables (/bin and /usr/bin) and for data files, respectively. Next, consider a system that mostly runs as a compile-engine.

In this case, it might be best to place the /tmp directory on a disk separate from the source code being compiled. Make sure that you check and mount the filesystem before creating any files on it. (If this is not feasible, you can instruct the compiler to use a directory on another disk for temporary files. Just set the TMPDIR environment variable to the new directory for temporary files.) Now, look at a system that mainly runs many programs at the same time and does a lot of swapping.

In this case, it might be best to distribute the swap area in several partitions on different disks.

If improved I/O performance still does not occur after you have tuned your system, you may want to consider adding more hardware: disks, controllers, or memory.

If you are going to add more hardware to your system, how do you know which disk or controller to add? You can compare hardware specifications for currently supported disks and controllers by looking up the system specifications in your hardware owner's guide. By using this information, you can choose the right disk or controller to suit your particular needs.

By balancing the most active filesystems across controllers/disks, you can speed up disk access.

Another way to reduce the number of reads and writes that go out to the disk is to add more memory. This reduces swapping and paging.

When the IRIX system is multiprocessing a large number of processes, the amount of this swapping and paging activity can dominate the performance of the system. You can use the sar command to detect this condition and other tools to deal with it.

Determining whether your system is overloaded with paging and swapping requires some knowledge of a baseline. You need to use sar under various conditions to determine a baseline for your specific implementation. For example, you can boot your system and run some baseline tests with a limited number of processes running, and then again during a period of light use, a period of heavy networking activity, and then especially when the load is high and you are experiencing poor performance. Recording the results in your system log book can help you in making these baseline measurements.

Table 10-6 shows indicators of excessive paging and swapping on a smaller system.

You can use the following sar options to determine if poor system performance is related to swap I/O or to other factors:

If the %vswp number is 0 or very low, and vflt/s does not increase with the onset of poor performance, the performance problem is not primarily due to swap I/O.

However, when swap I/O may be the cause, there are several possible actions you can take:

Refer to “Multiple Page Sizes” for information on dynamic tuning of page size.

A CPU can execute only one process at any given instant. If the CPU becomes overloaded, processes have to wait instead of executing. You cannot change the speed of the CPU (although you may be able to upgrade to a faster CPU or add CPU boards to your system if your hardware allows it), but you can monitor CPU load and try to distribute it. Table 10-7 shows the fields to check for indications that a system is CPU bound.

You can also use the top(1) or gr_top(1) commands to display processes having the highest CPU usage. For each process, the output lists the user, process state flags, process ID and group ID, CPU cycles used, processor currently executing the process, process priority, process size (in pages), resident set size (in pages), amount of time used by the process, and the process name. For more information, see the top(1) or gr_top(1) man page.

To increase CPU performance, make the following modifications:

“Paging and Swapping” describes what happens when you do not have enough physical (main) memory for processes. This section discusses a different problem—what happens when you do not have enough available memory (sometimes called virtual memory), which includes both physical memory and logical swap space.

The IRIX virtual memory subsystem allows programs that are larger than physical memory to execute successfully. It also allows several programs to run even if the combined memory needs of the programs exceed physical memory. It does this by storing the excess data on the swap device(s).

The allocation of swap space is done after program execution has begun. This allows programs with large a virtual address to run as long as the actual amount of virtual memory allocated does not exceed the memory and swap resources of the machine.

Usually it is evident when you run out of memory, because a message is sent to the console that begins:

Out of logical swap space...

If you see this message these are the possible causes:

You can add virtual swap space to your system at any time. See “Swap Space” in Chapter 6 to add more swap space. You need to add physical swap space, though, if you see the message:

Process killed due to insufficient memory

The following system calls return EAGAIN if there is insufficient available memory: exec, fork, brk, sbrk (called by malloc), mpin, and plock. Applications should check the return status and exit gracefully with a useful message.

To check the size (in pages) of a process that is running, execute ps -el (you can also use top). The SZ:RSS field shows very large processes.

By checking this field, you can determine the amount of memory the process is using. A good strategy is to run very large processes at less busy times.

To see the amount of main memory, use the hinv(1) command. It displays data about your system's configuration. For example:

Main memory size: 64 Mb

To increase the amount of virtual memory, increase the amount of real memory and/or swap space. Note that most of the paging/swapping solutions are also ways to conserve available memory. These include:

However, the most dramatic way to increase the amount of virtual memory is to add more swap space. See “Swap Space” in Chapter 6.

Before you reconfigure the system by changing kernel parameters, it is a good idea to have a current and complete backup of the system. See IRIX Admin: Backup, Security, and Accounting .

---

Caution: Always back up the entire system before tuning.

---

After determining the parameter you need to change (for example, you need to increase nproc because you have a large number of users), you must first back up the system and the kernel. Give the command:

cp /unix /unix.save

This command creates a safe copy of your kernel. Through the rest of this example, this is called your old saved kernel. If you make this copy, you can always go back to your old saved kernel if you are not satisfied with the results of your tuning.

Once your backups are complete, you can execute the systune command. Note that you can present new values to systune in either hexadecimal or decimal notation. Both values are printed by systune.

An invocation of systune to increase nproc looks something like this:

systune -i
Updates will be made to running system and /unix.install
systune-> nproc
        nproc = 400 (0x190)
systune-> nproc = 500
        nproc = 400 (0x190)
        Do you really want to change nproc to 500 (0x1f4)? (y/n) y
In order for the change in parameter nproc to become effective /unix.install must be moved to /unix and the system rebooted
systune-> quit

Then reboot your system. Also, be sure to document the parameter change you made in your system log book.

---

Caution: When you issue the reboot command, the system overwrites the current kernel (/unix) with the kernel you have just created ( /unix.install). This is why you should always copy the current kernel to a safe place before rebooting.

---

The systune command creates a new kernel automatically. However, if you changed parameters without using systune, or if you have added new system hardware (such as a new CPU board on a multiprocessor system), you must use autoconfig to generate a new kernel.

The autoconfig command uses some environment variables. These variables are described in detail in the autoconfig(1M) man page. If you have any of the following variables set, you may need to unset them before running autoconfig:

To build a new kernel after reconfiguring the system, follow the steps in Procedure 10-4:

An autoconfiguration script, found in /etc/rc2.d/S95autoconfig, runs during system startup. This script asks you if you would like to build a new kernel under the following conditions:

If any of these conditions is true, the system prompts you during startup to reconfigure the operating system:

Automatically reconfigure the operating system? y

If you answer y to the prompt, the script runs lboot and generates /unix.install with the new image.You can disable the autoconfiguration script by renaming /etc/rc2.d/S95autoconfig to something else that does not begin with the letter S, for example, /etc/rc2.d/wasS95autoconfig.