The symmon standalone debugger resides in the volume header of a disk—not in a normal IRIX filesystem. The volume header is disk partition 8. It always contains a label record (sgilabel). On a bootable disk, the volume header contains the standalone shell sash that manages the bootstrap operation. Some bootable disks may also contain the ide program, a PROM-level diagnostic program. If symmon is to be available, it, too, must be placed in the volume header.

Normally you acquire symmon by installing the debugging kernel feature (eoe.sw.kdebug) in the IRIX Developer Option software distribution. You can verify that this feature has been installed by executing the command

versions eoe.sw.kdebug

The response should confirm the presence of this component (it does not show symmon by name). When you install the kernel debug feature, the symmon program file is copied to the volume header of the current boot disk automatically.

You can verify the presence of symmon in the volume header through the use of dvhtool (described in the dvhtool(1) reference page). The results should be similar to the display in Example 10-1. The response to the “l” (list) command shows that the volume header of this disk contains sgilabel, ide, sash, and symmon.

# dvhtool -v list /dev/rvh 
Current contents:
        File name        Length     Block #
        sgilabel            512           2
        ide              281600         278
        sash             281600         828
        symmon           248320        1378

In the event you need to install symmon in the volume header of a disk without using the software manager, you can copy the standalone program to the volume header using dvhtool. However, you first need to get a copy of the program in the form of a UNIX file.

Starting from a volume that currently has a copy of symmon (verified as in Example 10-1), use dvhtool to extract a copy of symmon into a convenient spot.

dvhtool -v g symmon /var/tmp/symmon.IPxx 

There is a unique version of symmon for each CPU module, so it is a good idea to qualify the filename with the CPU module type. Once the program is available as a normal file, you can use dvhtool to install it in the volume header of some other disk.

In the event there is not enough room in partition 0 (the volume header) of the target disk, it is safe to use dvhtool to delete the ide program from the volume header. The ide application can be booted manually from a CDROM if it is ever required.

In order to make debugging symbols available in the kernel, you must make two changes, one required and one optional, in the file /var/sysgen/system/irix.sm. As superuser, make a hard link to the file /var/sysgen/system/irix.sm as irix.sm.nondebug. This enables you to return easily to a nondebugging kernel.

Including Symbols in the Kernel Image

Edit /var/sysgen/system/irix.sm. Near the end, note the lines that resemble the following:

* Compilation and load flags *  To load a kernel that can be co-resident with symmon *  (for breakpoint debugging) replace LDOPTS *  with the following. You must also INCLUDE prf and idbg. * *LDOPTS: -non_shared -N -e start -G 8 -elf -woff 84 -woff 47 -woff 17 -mips2 -o32 -nostdlib -T 88069000

The active LDOPTS statement (the one without an initial asterisk) appears a few lines later. Remove the asterisk from the front of the debugging LDOPTS to make it active. Insert an asterisk to convert the original LDOPTS into a comment.

---

Tip: Despite the residual comment in the irix.sm file, you need not include module prf in a debugging kernel. It is only used for kernel profiling.

---

*
* Kernel debugging tools (see profiler(1M) and idbg(1M))
*
EXCLUDE: idbg
EXCLUDE: dmiidbg, grioidbg, xfsidbg, xlvidbg, cachefsidbg, mloadidbg

* Required kernel modules
...
* ksync - kernel synchronization routines (mutex_lock, sv_wait, psema...)
*   or
*   ksync_metered  - metered kernel synchronization routines
...
*
KERNEL: kernel
INCLUDE: os, disp, mem, zero
INCLUDE: ksync
EXCLUDE: ksync_metered

INCLUDE hardlocks

INCLUDE dhardlocks

Run the autoconfig command to generate a new kernel that will reflect the changes made in irix.sm. The result is a new kernel file, /unix.install, that will be renamed to /unix and used when the system is booted. This kernel can support idbg but is not yet ready for standalone debugging with symmon.

The setsym command copies the symbol table from a kernel file and stores it as data within the kernel, so that symmon can find it. After autoconfig has created /unix.install, apply the setsym command to it, as follows:

#setsym /unix.install

If this command returns an error message about “symbol table overflow,” it means you have neglected to activate the debugging LDOPTS statement in /var/sysgen/irix.sm.

---

Tip: You can use setsym with the -d option to generate a list of all symbols in the kernel being modified. The list is very long; direct it to a file for later reference.

---

At this time, you may wish to create a link to the current, nondebugging kernel so you can retrieve it easily. You can also return to a nondebugging kernel by restoring the original irix.sm file and running autoconfig again.

In order to use the standalone debugger, you must have an ASCII terminal as a separate system console device. Install a terminal next to the system or workstation and connect it to the first serial port (of a workstation) or the system console serial port (of a server).

You may have to modify the file /etc/inittab so that the line for the alternate console is active (see the inittab(4) reference page). Alternatively, you can use the System Manager application from the 4D desktop. Select the icon for Port Setup. Select the port and click Connect. You can then configure the port for baud rate and terminal type interactively.

Verify the terminal's operation by logging in to the system. When you know the terminal works, use the nvram command to change the nonvolatile RAM variable console from a letter “g” to a letter “d,” as follows:

# nvram console
g
# nvram console d
# nvram console
d

The nvram command is used to report and change the contents of the nonvolatile RAM variables used by the boot PROM and standalone shell (see the nvram(1) reference page).

After performing the preceding steps, restart the system. Messages from sash appear on the attached terminal, rather than on the graphics screen. If symmon is present, it announces itself on the console terminal also.

To verify operation of idbg, issue the idbg command and display the process list:

# idbg
idbg> plist
active process list:
34:672:"xdm" pri(60) SLEEP flags: load uload siglck recalc sv 
0:0:"sched" ndpri(39) SLEEP flags: sys nwake load uload sv 
31:193:"inetd" pri(60) SLEEP flags: load uload siglck recalc sv 
...

To verify operation of symmon, press control-A at the console terminal. The prompt string DBG: should appear. At this time the system is frozen and no longer responds to mouse or keyboard input. Type the letter c (for continue) and press return (in a multiprocessor, use c all). The system returns to life.

The details of cmn_err() usage are in the cmn_err(D3) reference page. The function prototype and the constant values it uses are declared in sys/cmnerr.h.

In summary, cmn_err() takes two or more arguments:

The first character of the message string specifies the destination of the message, either an in-memory buffer or the system log, or both.

#ifdef DEBUG
#define DBGMSG0(s) cmn_err(CE_DEBUG,s)
#define DBGMSG1(s,x) cmn_err(CE_DEBUG,s,x)
#define DBGMSG2(s,x,y) cmn_err(CE_DEBUG,s,x,y)
#define DBGMSG3(s,x,y,z) cmn_err(CE_DEBUG,s,x,y,z)
#else
#define DBGMSG0(s)
#define DBGMSG1(s,x)
#define DBGMSG2(s,x,y)
#define DBGMSG3(s,x,y,z)
#endif 

You can call the printf() function from a kernel module. The kernel version of printf() is basically a call to cmn_err() with severity CE_CONT. In general it is better to use cmn_err() explicitly.

The assert() macro is familiar to many C programmers; it terminates a program with a message if its argument evaluates to false (see the assert(3X) reference page). This normal assert() macro does not work in a kernel module because the normal C library is not available. However, a similar function is available as the ASSERT() macro in the header file sys/debug.h.

The ASSERT() macro compiles to null code unless the compiler variable DEBUG is not only defined, but defined as YES. When it compiles to executable code, ASSERT() tests its argument. If the argument evaluates to false, a kernel panic is forced.

Clearly ASSERT() must be used with care, testing conditions that are truly essential to the integrity of the system. When reporting conditions that are merely operational errors, use a call to cmn_err() with the CE_WARN option.

When the system boots a debugging kernel with symmon installed, control can pass into the debug monitor under several different circumstances:

When symmon gains control, it displays its “DBG:” prompt at the console terminal and waits for a command.

To resume execution at the point of interruption, enter the c (continue) command.

The exact set of commands supported by symmon changes from release to release and from CPU model to CPU model. Many symmon commands are useful only to SGI engineers who are debugging hardware and kernel problems. For a complete list of commands, see the symmon(1M) reference page, or enter symmon and give the help command. You can use control-S and control-Q on the console terminal to pause the scrolling display.

The commands described in this section are generally useful and are available on all CPU models under IRIX 6.2. These commands can be grouped into the following categories:

The symmon commands all have the same form: a keyword, usually followed by one or more arguments separated by spaces.

Many commands take an address value. An address argument value can have one of the following forms:

Some commands accept a range of addresses. A range can be written in one of two ways:

The register names that symmon accepts and shows in various displays are the conventional names used in MIPS assembly language programming. Refer to the MIPSpro Assembly Language Programmer's Guide and the processor manuals listed under “Additional Reading”.

The commands summarized in Table 10-1 are used to convert between symbolic names and their corresponding addresses.

---

Note: When symmon displays an address it normally shows a full 64 bits. In a 32-bit kernel, the most-significant 32 bits of a kernel virtual address are all-binary-1, from extension of the sign bit of the 32-bit address—as shown in the example of hx in Table 10-1. When you enter an address to a command in a 32-bit system, you only need to type the significant 32-bit value.

---

The commands summarized in Table 10-2 stop, start, and single-step kernel execution.

---

Tip: One way to force a memory dump from symmon is the command call dumpsys.

---

Following a break or a watchpoint, use the bt command to display the stack history and use printreg to display the registers (see “Commands to Display Memory”).

The hardware watchpoint used by the wpt command uses hardware registers in the MIPS R4000 and R10000 processors (the R8000 does not support the watchpoint registers). When a read or write access is addressed to any byte in the doubleword specified by the physical address, symmon gains control and displays the instruction that is attempting the access on the console terminal.

The argument of wpt must be a physical memory address and a multiple of 8. Use tlbvtop to get the physical equivalent of an address in a user address space (see “Commands to Manage Virtual Memory”). In a 32-bit kernel, the physical equivalent of an address in kernel space is obtained by changing the most significant hex digit to 0.

The commands summarized in Table 10-3 are used to display and manage the virtual memory translation system.

The commands summarized in Table 10-4 are used to display memory or variables.

The display routines available to the kp command are discussed under “Using idbg”. The names that idbg accepts as commands are all available under symmon through the kp command.

Use the dump command under symmon. Under idbg, use the hd command for the same purpose.

The commands in Table 10-5 are used to display the contents of the hwgraph (see “Hardware Graph” in Chapter 2).

The commands summarized in Table 10-6 are general-purpose utilities.

Superuser privilege is required to invoke idbg, because it maps kernel memory. The command is ineffective unless its support modules have been made part of the kernel. This can be done permanently by changing the irix.sm file (see “Including idbg in the Kernel Image”). Alternatively, you can load the needed modules dynamically using the ml command, as follows:

# ml ld -i /var/sysgen/boot/idbg.o

Dynamic loading is discussed at more length in the idbg(1M) and ml(1M) reference pages.

When the support modules are loaded, idbg can be invoked in three styles.

# idbg
idbg> plist 187
pid 187 is in proc slot 31
idbg> quit
#

# idbg -r /var/tmp/idbg.save
idbg> plist 187
pid 187 is in proc slot 31
idbg> proc 31
proc: slot 31 addr 0x8832db30 pid 187 ppid 1 uid 0 abi IRIX5 
 SLEEP flags: load uload siglck recalc sv 
...
idbg> ^D
# cat /var/tmp/idbg.save
pid 187 is in proc slot 31
proc: slot 31 addr 0x8832db30 pid 187 ppid 1 uid 0 abi IRIX5 
 SLEEP flags: load uload siglck recalc sv 
...
#

# idbg plist | fgrep -c tcsh
3
#

Almost all idbg commands are concerned with displaying kernel memory data in different ways. There are commands to display almost every type of kernel data.

The vocabulary of commands changes from release to release, and can change within releases by software patches. Also, the commands available depend on which support modules are loaded; for example lock and semaphore meters cannot be displayed unless the ksynch_meter module is loaded (see “Including Lock Metering in the Kernel Image”). Only a few commands are listed in the idbg(1M) reference page.

The commands summarized in this book are generally useful and available on all platforms in the current release of IRIX. For a complete (but cursory) list, use the command itself.

# idbg help | lp

In general, commands take zero or one argument. Typically the argument is a number, which can be any of the following:

The number is interpreted in the context of the command: sometimes it represents a process ID (pid), sometimes a process “slot” number or a buffer number. Often commands treat positive numbers as slot numbers or table indexes, while negative numbers are treated as addresses in kernel space.

The commands summarized in Table 10-7 are used to display memory based on specific addresses or symbols, and to display the addresses for kernel symbols.

When you display the circular buffer, there is no special indication to show which line is the newest. You have to deduce the boundary between the newest and oldest lines from the content.

The commands summarized in Table 10-8 are concerned with displaying the status of processes. Processes are recorded in an array of “slots.” The plist command gives the slot number for a given process ID. Many other commands take process addresses.

The commands summarized in Table 10-9 display the state of semaphores and locks of different kinds, including metering information when the metered-lock module is included in the kernel.

The commands summarized in Table 10-10 can be used to display the status of an I/O device or driver.

The commands summarized in Table 10-11 are used to display the state of buf_t objects and the queue of buf_t objects maintained by the kernel.

The commands summarized in Table 10-12 are concerned with displaying STREAMS data structures such as message buffers.

The commands summarized in Table 10-13 display data structures that are related in one way or another to networking and network device drivers.