This section describes the basic steps for using cpusets and the cpuset(1) command. For a detailed example, see “Cpuset System Tutorial”.

To install the Cpuset System software, see “Installing the Cpuset System”.

To use cpusets, perform the following steps:

The cpuset(1) command is used to create and destroy cpusets, to retrieve information about existing cpusets, and to attach a process and all of its children to a cpuset. The syntax of the cpuset command is as follows:

cpuset [-q cpuset_name[,cpuset_name_dest] [-A command]|[-c -f filename]|
[-d]|[-i]|[-l][-m]|[-M]|[-Q]|[-p]]|[-T]| -C | -Q | -h

The cpuset command accepts the following options:

For more information on using cpusets, see the cpuset(1) man page, “Restrictions on CPUs within Cpusets”, and “Cpuset System Tutorial”.

The following restrictions apply to CPUs belonging to cpusets:

For a description of cpuset command arguments and additional information, see the cpuset(1), cpuset(4), and cpuset(5) man pages.

This section gives a detailed example of how to divide a system using cpusets. It contains a simple procedure to follow to divide the example system into cpusets with references to additional explanatory information.

Figure 4-1, shows a block diagram of a system with 16 processors and three cpusets. This section provides examples of configuration files and the commands used to create a boot cpuset containing half of the system's CPUs for normal system usage, and two cpusets named Green and Blue, respectively, for specified purposes. The Green cpuset specifies a closed cpuset restricted to a specific application to be executed by members of group artists. The Blue cpuset specifies a second closed cpuset restricted to a specific application to be executed by members of group writers.

Figure 4-1. Dividing a System Using Cpusets

Perform the following steps to divide a system with 16 processors into three cpusets, as shown in Figure 4-1:

The boot_cpuset.so(4) library provides a method for containing the init(1M) process and all of its descendents within a cpuset. Because all standard processes are descendents of the init process, this means that all standard processes on the system such as daemons, interactive or background processing, scripts, and so on, are confined to this cpuset. This cpuset is named boot.

---

Note: The boot_cpuset.so library is provided only on SGI 2000, SGI Origin 300, and SGI Orgin 3000 series of systems, that is, systems that are based on ccNUMA or NUMAflex architecture. The SGI Origin 3000 series of servers uses the NUMAflex interconnect fabric and modular components, or "bricks," to isolate the CPU and memory, I/O, and storage into separate bricks. A CPU brick, called a C-brick, contains four CPUs and up to 8 Gbytes of local memory. The SGI 2000 series of servers uses the earlier ccNUMA interconnect fabric. The smallest building block of the scalable ccNUMA architecture is the node board, consisting of two CPUs with associated cache and memory. The description of cpusets in this manual applies to both the NUMAflex and ccNUMA architectures.

---

The boot_cpuset.so library is located in the /lib32 directory and its behavior is controlled by the following files:

Use chkconfig(1M) command to create the /etc/config/boot_cpuset file as follows:

chkconfig -f boot_cpuset on

You can use the chkconfig(1M) command to configure the boot_cpuset.so(4) library on or off. If the library is configured on by init during system startup, the boot_cpuset.so library is loaded and executed and the cpuset is created. If the library is configured off, the library will exit and init will resume normal processing.

The /etc/config/boot_cpuset.config file is the configuration file specifying the cpuset. It follows the same conventions as the cpuset(4) configuration file.

The following example shows a boot_cpuset.config file that would divide half of an eight CPU system for normal system usage:

# the boot_cpuset
MEMORY_LOCAL
MEMORY_MANDATORY

CPU 0
CPU 1
CPU 2
CPU 3

---

Note: CPU 0 cannot belong to an EXCLUSIVE cpuset. For restrictions that apply to CPUs belonging to cpusets, see “Restrictions on CPUs within Cpusets”.

---

The second configuration file shows a cpuset that could be dedicated to a specific application:

# the cpuset dedicated to a specific application
EXCLUSIVE
MEMORY_LOCAL
MEMORY_MANDATORY

CPU 4
CPU 5
CPU 6
CPU 7

For more information, see “Cpuset Command and Configuration File” and the cpuset(4) man page.

This section describes the cpuset(1) command and the cpuset configuration file.

cpuset Command

The cpuset(1) command is used to define and manage a set of CPUs called a cpuset. A cpuset is a named set of CPUs, which may be defined as restricted or open. The cpuset command creates and destroys cpusets, retrieves information about existing cpusets, and attaches a process to a cpuset. Attachment to a cpuset is inherited across the fork(2) system call. Consequently, all processes that are children of an attached process will also be attached to the same cpuset.

---

Note: The cpuset command does not require the use of the Miser batch processing system.

---

A restricted cpuset allows only processes that are attached to the cpuset to run on the set of CPUs. An open cpuset allows any process to run on its CPUs, but a process that is attached to the cpuset can run only on the CPUs belonging to the cpuset.

For the SGI 2000, SGI Origin 300, and SGI Origin 3000 series of systems-- systems that are based on ccNUMA architecture--the administrator can restrict memory allocation to the nodes that contain the CPUs defined in the cpuset. For more information, see the MEMORY_MANDATORY flag description that follows and the cpuset(4) man page.

Cpuset Configuration File

A cpuset is defined by a cpuset configuration file and a name. See the cpuset(4) man page for a definition of the file format. The cpuset configuration file is used to list the CPUs that are members of the cpuset. It also contains any additional arguments required to define the cpuset. A cpuset name is between 3 and 8 characters long; names of 2 or fewer characters are reserved. You can designate one or more CPUs or a range of CPUs as part of a cpuset on a single line in the cpuset configuration file. CPUs in a cpuset do not have to be specified in a particular order. Each cpuset on your system must have a separate cpuset configuration file.

---

Note: In a cluster environment, the cpuset configuration file should reside on the root file system. If the cpuset configuration file resides on a file system other than the root file system and you attempt to unmount the file system, the vnode for the cpuset remains active and the unmount command fails. For more information, see the mount(1M) man page.

---

The file permissions of the configuration file define access to the cpuset. When permissions need to be checked, the current permissions of the file are used. It is therefore possible to change access to a particular cpuset without having to tear it down and recreate it, simply by changing the access permission. Read access allows a user to retrieve information about a cpuset, while execute permission allows a user to attach a process to the cpuset.

By convention, CPU numbering on SGI systems ranges between zero and the number of processors on the system minus one. The mpadmin -n command reports which processors are physically configured on a system. You can also use the hinv -vm command to show the hardware configuration of your system. For more information on the CPU naming convention and system hardware configuration, see Chapter 4, “Configuring the IRIX Operating System”, in the IRIX Admin: System Configuration and Operation manual and the mpadmin(1) and hinv(1) man pages.

The following is a sample configuration file that describes an exclusive cpuset containing three CPUs:

# cpuset configuration file EXCLUSIVE MEMORY_LOCAL MEMORY_EXCLUSIVE CPU 1 CPU 5 CPU 10

This specification will create a cpuset containing three CPUs. When the EXCLUSIVE flag is set, it restricts those CPUs to running threads that have been explicitly assigned to the cpuset. When the MEMORY_LOCAL flag is set, the jobs running on the cpuset will use memory from the nodes containing the CPUs in the cpuset. When the MEMORY_EXCLUSIVE flag is set, jobs running on other cpusets or on the global cpuset will normally not use memory from these nodes.

When the MEMORY_MANDATORY flag is set, the jobs running on the cpuset can only use memory from nodes containing the CPUs in this cpuset. The MEMORY_LOCAL flag is only an advisory but the MEMORY_MANDATORY flag is enforced by the kernel.

---

Note: On a system with both Miser and cpuset configured, conflicts may occur between a CPU that a Miser queue is using and a CPU assigned to a cpuset. Miser does not have access to CPUs that belong to a cpuset configured with the EXCLUSIVE flag set. Avoid running Miser and cpusets on the same system.

---

The following is a sample configuration file that describes an exclusive cpuset containing seven CPUs:

# cpuset configuration file EXCLUSIVE MEMORY_LOCAL MEMORY_EXCLUSIVE CPU 16 CPU 17-19, 21 CPU 27 CPU 25

Commands are newline terminated; characters following the comment delimiter, #, are ignored; case matters; and tokens are separated by whitespace, which is ignored.

The valid tokens are as follows:

Valid tokens	Description
EXCLUSIVE	Defines the CPUs in the cpuset to be restricted. It can occur anywhere in the file. Anything else on the line is ignored.
MEMORY_LOCAL	Threads assigned to the cpuset will attempt to assign memory only from nodes within the cpuset. Assignment of memory from outside the cpuset will occur only if no free memory is available from within the cpuset. No restrictions are made on memory assignment to threads running outside the cpuset.
MEMORY_EXCLUSIVE	Threads not assigned to the cpuset will not use memory from within the cpuset unless no memory outside the cpuset is available. When a cpuset is created and memory is occupied by threads that are already running on the cpuset nodes, no attempt is made to explicitly move this memory. If page migration is enabled, the pages will be migrated when the system detects the most references to the pages that are nonlocal.
MEMORY_KERNEL_AVOID	The kernel avoids allocating memory from nodes contained in this cpuset. If kernel memory requests cannot be satisfied from outside this cpuset, this option is ignored and allocations occur from within the cpuset. Currently, this option prevents only the system buffer cache from being placed on the specified nodes. Caution: Most sites running cpusets should not use this option. The use of this option can degrade system performance because kernel memory allocations become concentrated on the remaining system nodes. This option is effective only for certain workload patterns and can cause severe performance penalties in other situations. Do not use this option unless it is indicated by SGI support staff. This option was introduced in the IRIX 6.5.7 release.
MEMORY_MANDATORY	The kernel will limit all memory allocations to nodes that are contained in this cpuset. If memory requests cannot be satisfied, the allocating process will sleep until memory is available. The process will be killed if no more memory can be allocated.
POLICY_PAGE	Requires the MEMORY_MANDATORY token. This is the default policy if no policy is specified. This policy will cause the kernel to move user pages to the swap file (see swap(1M)) to free physical memory on the nodes contained in this cpuset. If swap space is exhausted, the process will be killed.
POLICY_KILL	Requires the MEMORY_MANDATORY token. The kernel will attempt to free as much space as possible from kernel heaps, but will not page user pages to the swap file. If all physical memory on the nodes contained in this cpuset are exhausted, the process will be killed.
CPU	Specifies that a CPU will be part of the cpuset.

Although the Cpuset System is functionally separate from the Miser batch processing system, the current Cpuset System was developed in conjunction with the software development of Miser. The Cpuset System software is contained within the Miser subsystem software. To install the Cpuset System software, see “Enabling or Disabling Miser” in Chapter 3.

The cpuset -q cpuset_name -p command allows you to see the various properties associated with a particular cpuset as follows:

The cpusetGetProperties(3x) function in the cpuset library is used retrieve various properties of the specified cpuset. The cpusetFreeProperties(3x) function is used to release memory used by the cpuset_Properties_t structure. For more information, see “Retrieval Functions” in Appendix A, and “Clean-up Functions” in Appendix A, and the cpusetGetProperties(3x) and cpusetFreeProperties(3x) man pages.

This section describes how to run cpusets in a Trusted IRIX environment.

The file permissions of the configuration file define access to the cpuset. When permissions need to be checked, the current permissions of the file are used.

Read access allows a user to retrieve information about a cpuset and execute permission allows the user to attach a process to the cpuset.

Cpusets on IRIX require two user classes: root and user. The root class creates, destroys, moves a process, and adds a process to the cpuset. The user class is governed by the file permissions of the configuration file for the given cpuset.

Given a configuration file with the following characteristics:

Permissions     Owner   Group   Size    Filename
---------------------------------------------------------

 -rwxr-----      root    cpuset  512     cpuset.test

Group read permission allows a user belonging to the group cpuset to list all cpusets in the cpuset defined by the cpuset.test file and get a listing of all processes in this cpuset. In order for the user to add processes to the cpuset governed by the cpuset.test file, you would need to change the permissions as follows:

Permissions     Owner   Group   Size    Filename
---------------------------------------------------------

-rwxr-x---      root    cpuset  512     cpuset.test

In a Trusted IRIX environment, permissions are governed by the /etc/capability file. See the capability(4) and capabilities(4) man pages for more information on the capability mechanism that provides fine grained control over the privileges of a process. Each user in the capability file has a set of minimum and maximum permissions. Consequently, root does not have any special abilities except to be able to use the suattr(1M) call so that it may assume any capabilities and permissions. Capabilities and permissions are also narrowed by the use of mandatory access control (MAC) labels and access control lists (ACLs).

In Trusted IRIX, to allow a user belonging to the group cpuset to list all cpusets in the cpuset defined by the cpuset.test file and get a listing of all processes in this cpuset, you must perform the following:

You cannot assign a user all capabilities with effective, inherited, and permissive rights (+eip) added. If you add +eip, the user will gain more privileges than allowed by the Cpuset System.

A Trusted IRIX user with a cpuuser1:all=:all= entry in the /etc/capability file has the same permissions as the user class in IRIX.

The root class in Trusted IRIX must have the CAP_SCHED_MGT+eip capability to create and destroy cpusets and to move process out of the cpuset.

In Trusted IRIX, you can use ACLs to control group permissions. With ACLs, you can easily select which users in the group can add a process to the cpuset. You can use ACLs to control a user's access to a cpuset without that user belonging to the group owner of the configuration file.

The cpuset library provides interfaces that allow a programmer to create and destroy cpusets, retrieve information about existing cpusets, obtain the properties associated with an existing cpuset, and to attach a process and all of its children to a cpuset.

For information on using the Cpuset Library, see “Application Programming Interface for the Cpuset System” in Appendix A.

This section describes the following topics:

Using the cpusetAttachPID and cpusetDetachPID Functions

The cpusetAttachPID(3x) function in the cpuset library allows a specific process, identified by its PID value, to be attached to a cpuset. The new cpusetDetachPID function allows a specific process, identified by its PID value, to be detached from a cpuset. The ability to attach and detach specific processes to or from a cpuset is controlled by the permissions of the cpuset configuration file and the ownership of the processes involved. For more information on the cpuset configuration file, see “Cpuset Configuration File”.

The cpusetAttachPID(3x) and cpusetDetachPID(3x) functions should not be used with the MEMORY_MANDATORY flag set to avoid memory latency problems. Because a cpuset will use memory only from the original compute nodes, use the cpusetAttachPID and cpusetDetachPID functions as follows:

Figure 4-2, shows several jobs running in two cpusets each containing four CPUs. A prime job requires a new cpuset using all eight CPUs. To create the new cpuset, perform the following steps:

1. Use the cpusetDetachPID function to move all jobs out of cpuset A and cpuset B.

2. Suspend the jobs running on cpuset A and B.

3. Use the cpusetDestroy(3x) function to destroy cpuset A and cpuset B.

4. Use the cpusetCreate(3x) function to create the new cpuset for the prime job.

5. Run the prime job in the new cpuset.

6. Destroy the new cpuset when the prime job has completed running.

7. Recreate cpuset A and B exactly as before.

8. Restart the suspended jobs.

9. Use the cpusetAttachPID function to reattach each job to its respective cpuset.

Figure 4-2. Using the cpusetAttachPID and cpusetDetachPID Functions

Using the cpusetMove and cpusetMoveMigrate Functions

Figure 4-3, shows an example of using the cpusetMove(3x) and cpusetMoveMigrate(3x) functions.

Figure 4-3. Moving Processes From One Cpuset to Another

The cpusetMoveMigrate function is used to directly move a specific process and its associated memory--identified by its process ID (PID), process group ID (PGID), job ID (JID), session ID (SID), or array services handle (ASH) --to a specified cpuset. The cpusetMove function is used to temporarily move a process, identified by its PID, PGID, JID, or ASH--out of a specified cpuset to another cpuset or the global cpuset. In this case, the memory is not migrated (moved). Recall that global cpuset is a term used to describe all the CPUs that are not in a cpuset. Unlike the cpusetMoveMigrate function, the cpusetMove function does not move the memory associated with a process. One example of using these functions is shown in Figure 4-3. To move a process into a global cpuset from a cpuset you plan to destroy, use the cpusetMove function and specify the destination as NULL. You can then use the cpusetMoveMigrate function to move the process from the global cpuset into a newly created cpuset.

The man command provides online help on all resource management commands. To view a man page online, type man commandname. For printed versions of the cpuset library man pages, see “Application Programming Interface for the Cpuset System” in Appendix A in Appendix A.