Chapter 4. Cpuset System
Prev |  | Next

## Chapter 4. Cpuset System

A cpuset is a named set of CPUs, which may be defined to be restricted or open. A restricted cpuset allows only processes that are members of the cpuset to run on the set of CPUs. An open cpuset allows any process to run on its CPUs, but a process that is a member of the cpuset can run only on the CPUs belonging to the cpuset. A cpuset is defined by a cpuset configuration file and a name.

The Cpuset System is primarily a workload manager tool permitting a system administrator to restrict the number of processors that a process or set of processes may use. Cpusets may optionally restrict both kernel and user memory.

When the memory restriction feature is enabled, a set of nodes, each containing a set of CPUs, is computed from the list of CPUs supplied and memory allocations can be limited to the CPUs assigned to the nodes. Allocation limits can be restricted to the available physical memory or overflow can be swapped to the swap file.

A system administrator can use cpusets to create a division of CPUs within a larger system. Such a divided system allows a set of processes to be contained to specific CPUs, reducing the amount of interaction and contention those processes have with other work on the system. In the case of a restricted cpuset, the processes that are attached to that cpuset will not be affected by other work on the system; only those processes attached to the cpuset can be scheduled to run on the CPUs assigned to the cpuset. An open cpuset can be used to restrict processes to a set of CPUs so that the effect these processes have on the rest of the system is minimized.

A system administrator might want to restrict normal system usage of a large system to part of the machine and use the rest of the system for special purposes. The `boot_cpuset(4)` tool provides a method to restrict all normal start-up processes (including `init`, `inetd`, and so on) to some portion of the machine and allow specific users to use the other portion of the machine for their special purpose applications. The kernel maintains strict processor and memory separation between the two system portions. An administrator, for example, might choose to divide a system into two halves, with one half supporting normal system usage and the other half dedicated to a particular application. The advantage this mechanism has over physical reconfiguration is that the configuration may be changed with a simple reboot and does not need to be aligned on a hardware module boundary.

Kernel system threads and interrupt threads can be confined to the boot cpuset by using the XThread Control Interface (XTCI). This keeps the system and interrupt threads from competing with applications outside of the boot cpuset for resources. For more information on XTCI, see the `realtime(5)` man page and the REACT Real-Time Programmer's Guide. For more information on the boot cpuset, see “Boot Cpuset”.

The `cpuset` `-q cpuset_name -p` command allows you to see the properties of particular cpuset, such as the number of processes and CPUs associated with a specified cpuset. For more information on cpuset properties, see “Obtaining the Properties Associated with a Cpuset” and the `cpuset(1)` man page.

Static cpusets are defined by an administrator after a system had been started. Users can attach processes to these existing cpusets. The cpusets continue to exist after jobs are finished executing.

Dynamic cpusets are created by a workload manager when required by a job. The workload manager attaches a job to a newly created cpuset and destroys the cpuset when the job has finished executing.

Cpusets can be used in conjunction with a batch processing system, like the Load Sharing Facility (LSF) or Portable Batch System (PBS), for data center resource management to improve the performance of large applications. Using cpusets with applications such as LSF or PBS enables your SGI Origin system to run more efficiently, reduces interference between jobs, and can substantially improve the consistency and predictability of system run times.

The Cpuset library routines, `cpusetMove (3x)` and `cpusetMoveMigrate(3x)`, can be used to move processes between cpusets and optionally migrate their memory. They allow you to move specific processes, or groups of processes, between existing cpusets, and out of a named cpuset into the pool of CPUs not assigned to any specific named cpuset. This pool of unused CPUs is called the global cpuset.

Using this functionality, you can easily destroy existing cpusets to free resources to run a prime job and then easily reconstitute cpusets to continue prior jobs. Because memory used by a process can be migrated to the node associated with the new cpuset, memory locality is improved. For more information on the `cpusetMove(3x)` and `cpusetMoveMigrate(3x)` routines, see “Using the cpusetMove and cpusetMoveMigrate Functions” and “Application Programming Interface for the Cpuset System” in Appendix A.

For more information on dividing a system, see Chapter 4, “Configuring the IRIX Operating System” in the IRIX Admin: System Configuration and Operation manual.

The cpuset library provides interfaces that allow a programmer to create and destroy cpusets, retrieve information about existing cpusets, obtain the properties associated with a cpuset, and to attach a process and all of its children to a cpuset.

This chapter contains the following sections:

- “Using Cpusets”

- “Restrictions on CPUs within Cpusets”

- “Cpuset System Tutorial”

- “Boot Cpuset”

- “Cpuset Command and Configuration File”

- “Installing the Cpuset System”

- “Obtaining the Properties Associated with a Cpuset”

- “Cpuset System and Trusted IRIX”

- “Using the Cpuset Library”

- “Cpuset System Man Pages”

## Using Cpusets

This section describes the basic steps for using cpusets and the `cpuset(1)` command. For a detailed example, see “Cpuset System Tutorial”.

To install the Cpuset System software, see “Installing the Cpuset System”.

To use cpusets, perform the following steps:

1. Create a cpuset configuration file and give it a name. For the format of this file, see “Cpuset Configuration File”. For restrictions that apply to CPUs belonging to cpusets, see “Restrictions on CPUs within Cpusets”.

2. Create the cpuset with the configuration file specified by the `-f` parameter and the name specified by the `-q` parameter.

The `cpuset (1)` command is used to create and destroy cpusets, to retrieve information about existing cpusets, and to attach a process and all of its children to a cpuset. The syntax of the `cpuset` command is as follows:

    cpuset [-q cpuset_name[,cpuset_name_dest] [-A command]|[-c -f filename]|
    [-d]|[-i]|[-l][-m]|[-M]|[-Q]|[-p]]|[-T]| -C | -Q | -h

The `cpuset` command accepts the following options:

`-q` cpuset_name [`-A` command] | Runs the specified command on the cpuset identified by the `-q` parameter. If the user does not have access permissions or the cpuset does not exist, an error is returned.
`-q` cpuset_name [`-c` `-f` filename] | Creates a cpuset with the configuration file specified by the `-f` parameter and the name specified by the `-q` parameter. The operation fails if the cpuset name already exists, a CPU specified in the cpuset configuration file is already a member of a cpuset, or the user does not have the requisite permissions.
`-q` cpuset_name `-d` | Destroys the specified cpuset. A cpuset can only be destroyed if there are no processes currently attached to it.
`-q` cpuset_name `-l` | Lists all the processes in the cpuset.
`-q` cpuset_name `-m` | Moves all the attached processes out of the cpuset.
`-q` cpuset_name `-d` | Destroys the specified cpuset. A cpuset can only be destroyed if there are no processes currently attached to it.
`-q` cpuset_name `-Q` | Prints a list of the CPUs that belong to the cpuset.
`-q` cpuset_name `-p` | Prints out the permissions, ACLs, MAC labels, flags, number of processes, and the CPUs associated with the specified cpuset.
`-q` cpuset_name, cpuset_name_dest `-M` `suboption` | The `-M` option moves a process or a group of processes and their associated memory from cpuset_name to cpuset_name_dest. The valid suboptions are `PID`, `ASH`, `JID`, `SID`, and `PGID` indicate the ID type to be moved. The `-M` option also requires the `-i` option.
`-q` cpuset_name, cpuset_name_dest `-T` `suboption` | The `-T` option moves a proces or a group of processes but not their memory from `cpuset_name` to `cpuset_name_dest`. The valid suboptions `PID`, `ASH`, `JID`, `SID`, and `PGID` indicate the ID type to be moved. The `-T` option also requires the `-i` option.
`-q` cpuset_name, cpuset_name_dest [`-M` | `-T`] `suboption` `-i` id | The `-i` option tells the command what ID needs to be moved.
`-C` | Prints the name of the cpuset to which the process is currently attached.
`-Q` | Lists the names of all the cpusets currently defined.
`-h` | Print the command's usage message.

3. Execute the `cpuset` command to run a command on the cpuset you created as follows:

       cpuset -q cpuset_name -A command

For more information on using cpusets, see the `cpuset(1)` man page, “Restrictions on CPUs within Cpusets”, and “Cpuset System Tutorial”.

## Restrictions on CPUs within Cpusets

The following restrictions apply to CPUs belonging to cpusets:

- A CPU can belong to only one cpuset.

- CPU 0 cannot belong to an `EXCLUSIVE` cpuset.

- A CPU cannot be both restricted and isolated (see `mpadmin(1)` and `sysmp(2)`) and also be a member of a cpuset.

- Only the superuser can create or destroy cpusets.

- The `runon(1)` command cannot run a command on a CPU that is part of a cpuset unless the user has write or group write permission to access the configuration file of the cpuset.

For a description of `cpuset` command arguments and additional information, see the `cpuset(1)`, `cpuset(4)`, and `cpuset(5)` man pages.

## Cpuset System Tutorial

This section gives a detailed example of how to divide a system using cpusets. It contains a simple procedure to follow to divide the example system into cpusets with references to additional explanatory information.

Figure 4-1, shows a block diagram of a system with 16 processors and three cpusets. This section provides examples of configuration files and the commands used to create a boot cpuset containing half of the system's CPUs for normal system usage, and two cpusets named `Green` and `Blue`, respectively, for specified purposes. The `Green` cpuset specifies a closed cpuset restricted to a specific application to be executed by members of group `artists`. The `Blue` cpuset specifies a second closed cpuset restricted to a specific application to be executed by members of group `writers`.

Figure 4-1. Dividing a System Using Cpusets

Perform the following steps to divide a system with 16 processors into three cpusets, as shown in Figure 4-1:

1. Create a file named `boot_cpuset.config` to create a boot cpuset and divide half of a 16 CPU system dedicated to normal system usage. The boot cpuset contains all standard processes on the system such as daemons, interactive or background processing, scripts, and so on. The contents of this file are as follows:

       # boot
       MEMORY_LOCAL
       MEMORY_MANDATORY

       CPU 0
       CPU 1
       CPU 2
       CPU 3
       CPU 4
       CPU 5
       CPU 6
       CPU 7

   | Note: For this release, you can only designate one CPU on a single line in the `boot_cpuset.config` file. For more information on the `boot_cpuset.config` file, see “Boot Cpuset”.

   For an explanation of the `MEMORY_LOCAL` and `MEMORY_MANDATORY` flags, see “Cpuset Configuration File”.

2. Use the `chkconfig (1M)` command with the `-f` option to create an `/etc/config/boot_cpuset` file that contains the following:

       chkconfig boot_cpuset on

   For more information on the `/etc/config/boot_cpuset` file, see “Boot Cpuset”.

   When the system is rebooted, the boot cpuset will be created.

3. Create a dedicated cpuset called `Green` and assign a specific application, in this case, MovieMaker to run on it. Perform the following steps to accomplish this:

   1. Create a cpuset configuration file called `cpuset_1` with the following contents:

          # the cpuset configuration file called cpuset_1 that shows
          # a cpuset dedicated to a specific application
          EXCLUSIVE
          MEMORY_LOCAL
          MEMORY_MANDATORY

          CPU 8
          CPU 9
          CPU 10
          CPU 11

      | Note: You can designate more than one CPU or a range of CPUs on a single line in the cpuset configuration file. In this example, you could designate CPUs 8 through 11 on a single line as follows: `CPU 8-11`. For more information on the cpuset configuration file, see “Cpuset Configuration File”.

      For an explanation of the `EXCLUSIVE`, `MEMORY_LOCAL`, and `MEMORY_MANDATORY` flags, see “Cpuset Configuration File”.

   2. Use the `chmod(1)` command to set the file permissions on the `cpuset_1` configuration file so that only members of group `artists` can execute the application `moviemaker` on the `Green` cpuset.

   3. Use the `cpuset(1)` command to create the `Green` cpuset with the configuration file `cpuset_1` specified by the `-f` parameter and the name `Green` specified by the `-q` parameter.

          cpuset -q Green -f cpuset_1

   4. Execute the `cpuset` command as follows to run MovieMaker on a dedicated cpuset:

          cpuset -q Green -A moviemaker

      For more information on the `cpuset(1)` command, see “cpuset Command”.

      The `moviemaker` job threads will run only on CPUs in this cpuset. MovieMaker jobs will use memory from system nodes containing the CPUs in the cpuset. Jobs running on other cpusets will not use memory from these nodes. You could use the `cpuset` command to run additional applications on the same cpuset using the syntax shown in this example.

4. Create a third cpuset file called `Blue` and specify an application that will run only on this cpuset. Perform the following steps to accomplish this:

   1. Create a cpuset configuration file called `cpuset_2` with the following contents:

          # the cpuset configuration file called cpuset_2 that shows
          # a cpuset dedicated to a specific application
          EXCLUSIVE
          MEMORY_LOCAL
          MEMORY_MANDATORY

          CPU 12
          CPU 13
          CPU 14
          CPU 15

   2. Use the `chmod(1)` command to set the file permissions on the `cpuset_2` configuration file so that only members of group `writers` can execute the application `bookmaker` on the `Blue` cpuset.

   3. Use the `cpuset(1)` command to create the `Blue` cpuset with the configuration file `cpuset_2` specified by the `-f` parameter and the name specified by the `-q` parameter.

          cpuset -q Blue -f cpuset_2

   4. Execute the `cpuset (1)` command as follows to run `bookmaker` on CPUs in the `Green` cpuset.

          cpuset -q Blue -A bookmaker

      The `bookmaker` job threads will run only on this cpuset. BookMaker jobs will use memory from system nodes containing the CPUs in the cpuset. Jobs running on other cpusets will not use memory from these nodes.

## Boot Cpuset

The `boot_cpuset.so(4)` library provides a method for containing the `init(1M)` process and all of its descendents within a cpuset. Because all standard processes are descendents of the `init` process, this means that all standard processes on the system such as daemons, interactive or background processing, scripts, and so on, are confined to this cpuset. This cpuset is named `boot` .

Kernel system threads and interrupt threads can be confined to the boot cpuset by using the XThread Control Interface (XTCI), which is documented in the `realtime(5)` man page and the REACT Real-Time Programmer's Guide.

| Note: The `boot_cpuset.so` library is provided only on SGI 2000, SGI Origin 300, and SGI Orgin 3000 series of systems, that is, systems that are based on ccNUMA or NUMAflex architecture. The SGI Origin 3000 series of servers uses the NUMAflex interconnect fabric and modular components, or "bricks," to isolate the CPU and memory, I/O, and storage into separate bricks. A CPU brick, called a C-brick, contains four CPUs and up to 8 Gbytes of local memory. The SGI 2000 series of servers uses the earlier ccNUMA interconnect fabric. The smallest building block of the scalable ccNUMA architecture is the node board, consisting of two CPUs with associated cache and memory. The description of cpusets in this manual applies to both the NUMAflex and ccNUMA architectures.

The `boot_cpuset.so` library is located in the `/lib32` directory and its behavior is controlled by the following files:

- `/etc/config/boot_cpuset`

- `/etc/config/boot_cpuset.config`

Use `chkconfig(1M)` command to create the `/etc/config/boot_cpuset` file as follows:

    chkconfig -f boot_cpuset on

You can use the `chkconfig(1M)` command to configure the `boot_cpuset.so(4)` library on or off. If the library is configured on by `init` during system startup, the `boot_cpuset.so` library is loaded and executed and the cpuset is created. If the library is configured off, the library will exit and `init` will resume normal processing.

The `/etc/config/boot_cpuset.config` file is the configuration file specifying the cpuset. It follows the same conventions as the `cpuset(4)` configuration file.

The following example shows a `boot_cpuset.config` file that would divide half of an eight CPU system for normal system usage:

    # the boot_cpuset
    MEMORY_LOCAL
    MEMORY_MANDATORY

    CPU 0
    CPU 1
    CPU 2
    CPU 3

| Note: CPU 0 cannot belong to an `EXCLUSIVE` cpuset. For restrictions that apply to CPUs belonging to cpusets, see “Restrictions on CPUs within Cpusets”.

The second configuration file shows a cpuset that could be dedicated to a specific application:

    # the cpuset dedicated to a specific application
    EXCLUSIVE
    MEMORY_LOCAL
    MEMORY_MANDATORY

    CPU 4
    CPU 5
    CPU 6
    CPU 7

For more information, see “Cpuset Command and Configuration File” and the `cpuset(4)` man page.

## Cpuset Command and Configuration File

This section describes the `cpuset(1)` command and the cpuset configuration file.

### `cpuset` Command

The `cpuset (1)` command is used to define and manage a set of CPUs called a cpuset. A cpuset is a named set of CPUs, which may be defined as restricted or open. The `cpuset` command creates and destroys cpusets, retrieves information about existing cpusets, and attaches a process to a cpuset. Attachment to a cpuset is inherited across the `fork(2)` system call. Consequently, all processes that are children of an attached process will also be attached to the same cpuset.

| Note: The `cpuset` command does not require the use of the Miser batch processing system.

A restricted cpuset allows only processes that are attached to the cpuset to run on the set of CPUs. An open cpuset allows any process to run on its CPUs, but a process that is attached to the cpuset can run only on the CPUs belonging to the cpuset.

For the SGI 2000, SGI Origin 300, and SGI Origin 3000 series of systems-- systems that are based on ccNUMA architecture--the administrator can restrict memory allocation to the nodes that contain the CPUs defined in the cpuset. For more information, see the `MEMORY_MANDATORY` flag description that follows and the `cpuset(4)` man page.

### Cpuset Configuration File

A cpuset is defined by a cpuset configuration file and a name. See the `cpuset(4)` man page for a definition of the file format. The cpuset configuration file is used to list the CPUs that are members of the cpuset. It also contains any additional arguments required to define the cpuset. A cpuset name is between 3 and 8 characters long; names of 2 or fewer characters are reserved. You can designate one or more CPUs or a range of CPUs as part of a cpuset on a single line in the cpuset configuration file. CPUs in a cpuset do not have to be specified in a particular order. Each cpuset on your system must have a separate cpuset configuration file.

| Note: In a cluster environment, the cpuset configuration file should reside on the root file system. If the cpuset configuration file resides on a file system other than the root file system and you attempt to unmount the file system, the vnode for the cpuset remains active and the `unmount` command fails. For more information, see the `mount(1M)` man page.

The file permissions of the configuration file define access to the cpuset. When permissions need to be checked, the current permissions of the file are used. It is therefore possible to change access to a particular cpuset without having to tear it down and recreate it, simply by changing the access permission. Read access allows a user to retrieve information about a cpuset, while execute permission allows a user to attach a process to the cpuset.

By convention, CPU numbering on SGI systems ranges between zero and the number of processors on the system minus one. The `mpadmin` `-n` command reports which processors are physically configured on a system. You can also use the `hinv -vm` command to show the hardware configuration of your system. For more information on the CPU naming convention and system hardware configuration, see Chapter 4, “Configuring the IRIX Operating System”, in the IRIX Admin: System Configuration and Operation manual and the `mpadmin(1)` and `hinv(1)` man pages.

The following is a sample configuration file that describes an exclusive cpuset containing three CPUs:

    # cpuset configuration file
    EXCLUSIVE
    MEMORY_LOCAL
    MEMORY_EXCLUSIVE

    CPU 1
    CPU 5
    CPU 10

This specification will create a cpuset containing three CPUs. When the `EXCLUSIVE` flag is set, it restricts those CPUs to running threads that have been explicitly assigned to the cpuset. When the `MEMORY_LOCAL` flag is set, the jobs running on the cpuset will use memory from the nodes containing the CPUs in the cpuset. When the `MEMORY_EXCLUSIVE` flag is set, jobs running on other cpusets or on the global cpuset will normally not use memory from these nodes.

When the `MEMORY_MANDATORY` flag is set, the jobs running on the cpuset can only use memory from nodes containing the CPUs in this cpuset. The `MEMORY_LOCAL` flag is only an advisory but the `MEMORY_MANDATORY` flag is enforced by the kernel.

| Note: On a system with both Miser and cpuset configured, conflicts may occur between a CPU that a Miser queue is using and a CPU assigned to a cpuset. Miser does not have access to CPUs that belong to a cpuset configured with the `EXCLUSIVE` flag set. Avoid running Miser and cpusets on the same system.

The following is a sample configuration file that describes an exclusive cpuset containing seven CPUs:

    # cpuset configuration file
    EXCLUSIVE
    MEMORY_LOCAL
    MEMORY_EXCLUSIVE

    CPU 16
    CPU 17-19, 21
    CPU 27
    CPU 25

Commands are newline terminated; characters following the comment delimiter, #, are ignored; case matters; and tokens are separated by whitespace, which is ignored.

The valid tokens are as follows:

Valid tokens | Description
`EXCLUSIVE` | Defines the CPUs in the cpuset to be restricted. It can occur anywhere in the file. Anything else on the line is ignored.
`MEMORY_LOCAL` | Threads assigned to the cpuset will attempt to assign memory only from nodes within the cpuset. Assignment of memory from outside the cpuset will occur only if no free memory is available from within the cpuset. No restrictions are made on memory assignment to threads running outside the cpuset.
`MEMORY_EXCLUSIVE` | Threads not assigned to the cpuset will not use memory from within the cpuset unless no memory outside the cpuset is available. When a cpuset is created and memory is occupied by threads that are already running on the cpuset nodes, no attempt is made to explicitly move this memory. If page migration is enabled, the pages will be migrated when the system detects the most references to the pages that are nonlocal.
`MEMORY_KERNEL_AVOID` | The kernel avoids allocating memory from nodes contained in this cpuset. If kernel memory requests cannot be satisfied from outside this cpuset, this option is ignored and allocations occur from within the cpuset. Currently, this option prevents only the system buffer cache from being placed on the specified nodes.
| Caution: Most sites running cpusets should not use this option. The use of this option can degrade system performance because kernel memory allocations become concentrated on the remaining system nodes. This option is effective only for certain workload patterns and can cause severe performance penalties in other situations. Do not use this option unless it is indicated by SGI support staff.
This option was introduced in the IRIX 6.5.7 release.
`MEMORY_MANDATORY` | The kernel will limit all memory allocations to nodes that are contained in this cpuset. If memory requests cannot be satisfied, the allocating process will sleep until memory is available. The process will be killed if no more memory can be allocated.
`POLICY_PAGE` | Requires the `MEMORY_MANDATORY` token. This is the default policy if no policy is specified. This policy will cause the kernel to move user pages to the swap file (see `swap(1M)`) to free physical memory on the nodes contained in this cpuset. If swap space is exhausted, the process will be killed.
`POLICY_KILL` | Requires the `MEMORY_MANDATORY` token. The kernel will attempt to free as much space as possible from kernel heaps, but will not page user pages to the swap file. If all physical memory on the nodes contained in this cpuset are exhausted, the process will be killed.
`CPU` | Specifies that a CPU will be part of the cpuset.

## Installing the Cpuset System

Although the Cpuset System is functionally separate from the Miser batch processing system, the current Cpuset System was developed in conjunction with the software development of Miser. The Cpuset System software is contained within the Miser subsystem software. To install the Cpuset System software, see “Enabling or Disabling Miser” in Chapter 3.

## Obtaining the Properties Associated with a Cpuset

The `cpuset` `-q cpuset_name -p` command allows you to see the various properties associated with a particular cpuset as follows:

- Permissions on the configuration file that define access to the cpuset

- Access control lists (ACLs)

- Mandatory access control (MAC) labels

- Flags such as `MEMORY_EXCLUSIVE`

  For more information on flags associated with a cpuset, see “Cpuset Configuration File”, and the `cpuset(4)` man page.

- Number of processes

- CPUs

The `cpusetGetProperties(3x)` function in the cpuset library is used retrieve various properties of the specified cpuset. The `cpusetFreeProperties(3x)` function is used to release memory used by the `cpuset_Properties_t` structure. For more information, see “Retrieval Functions” in Appendix A, and “Clean-up Functions” in Appendix A, and the `cpusetGetProperties (3x)` and `cpusetFreeProperties(3x)` man pages.

## Cpuset System and Trusted IRIX

This section describes how to run cpusets in a Trusted IRIX environment.

The file permissions of the configuration file define access to the cpuset. When permissions need to be checked, the current permissions of the file are used.

Read access allows a user to retrieve information about a cpuset and execute permission allows the user to attach a process to the cpuset.

Cpusets on IRIX require two user classes: root and user. The root class creates, destroys, moves a process, and adds a process to the cpuset. The user class is governed by the file permissions of the configuration file for the given cpuset.

Given a configuration file with the following characteristics:

    Permissions     Owner   Group   Size    Filename
    ---------------------------------------------------------

     -rwxr-----      root    cpuset  512     cpuset.test

Group read permission allows a user belonging to the group cpuset to list all cpusets in the cpuset defined by the `cpuset.test` file and get a listing of all processes in this cpuset. In order for the user to add processes to the cpuset governed by the `cpuset.test` file, you would need to change the permissions as follows:

    Permissions     Owner   Group   Size    Filename
    ---------------------------------------------------------

    -rwxr-x---      root    cpuset  512     cpuset.test

In a Trusted IRIX environment, permissions are governed by the `/etc/capability` file. See the `capability (4)` and `capabilities(4)` man pages for more information on the capability mechanism that provides fine grained control over the privileges of a process. Each user in the `capability` file has a set of minimum and maximum permissions. Consequently, root does not have any special abilities except to be able to use the `suattr(1M)` call so that it may assume any capabilities and permissions. Capabilities and permissions are also narrowed by the use of mandatory access control (MAC) labels and access control lists (ACLs).

In Trusted IRIX, to allow a user belonging to the group `cpuset` to list all cpusets in the cpuset defined by the `cpuset.test` file and get a listing of all processes in this cpuset, you must perform the following:

- Assign the user with a MAC label of userlow.

- Make the following entry in the `/etc/capability` file: `cpuuser1:all=:all=`

You cannot assign a user all capabilities with effective, inherited, and permissive rights (`+eip`) added. If you add `+eip`, the user will gain more privileges than allowed by the Cpuset System.

A Trusted IRIX user with a `cpuuser1:all=:all=` entry in the `/etc/capability` file has the same permissions as the user class in IRIX.

The root class in Trusted IRIX must have the `CAP_SCHED_MGT+eip` capability to create and destroy cpusets and to move process out of the cpuset.

In Trusted IRIX, you can use ACLs to control group permissions. With ACLs, you can easily select which users in the group can add a process to the cpuset. You can use ACLs to control a user's access to a cpuset without that user belonging to the group owner of the configuration file.

## Using the Cpuset Library

The cpuset library provides interfaces that allow a programmer to create and destroy cpusets, retrieve information about existing cpusets, obtain the properties associated with an existing cpuset, and to attach a process and all of its children to a cpuset.

For information on using the Cpuset Library, see “Application Programming Interface for the Cpuset System” in Appendix A.

This section describes the following topics:

- “Using the cpusetAttachPID and cpusetDetachPID Functions”

- “Using the cpusetMove and cpusetMoveMigrate Functions”

### Using the `cpusetAttachPID` and `cpusetDetachPID` Functions

The `cpusetAttachPID(3x)` function in the cpuset library allows a specific process, identified by its PID value, to be attached to a cpuset. The new `cpusetDetachPID` function allows a specific process, identified by its PID value, to be detached from a cpuset. The ability to attach and detach specific processes to or from a cpuset is controlled by the permissions of the cpuset configuration file and the ownership of the processes involved. For more information on the cpuset configuration file, see “Cpuset Configuration File”.

The `cpusetAttachPID(3x)` and `cpusetDetachPID(3x)` functions should not be used with the `MEMORY_MANDATORY` flag set to avoid memory latency problems. Because a cpuset will use memory only from the original compute nodes, use the `cpusetAttachPID` and `cpusetDetachPID` functions as follows:

Figure 4-2, shows several jobs running in two cpusets each containing four CPUs. A prime job requires a new cpuset using all eight CPUs. To create the new cpuset, perform the following steps:

1. Use the `cpusetDetachPID` function to move all jobs out of cpuset A and cpuset B.

2. Suspend the jobs running on cpuset A and B.

3. Use the `cpusetDestroy (3x)` function to destroy cpuset A and cpuset B.

4. Use the `cpusetCreate (3x)` function to create the new cpuset for the prime job.

5. Run the prime job in the new cpuset.

6. Destroy the new cpuset when the prime job has completed running.

7. Recreate cpuset A and B exactly as before.

8. Restart the suspended jobs.

9. Use the `cpusetAttachPID` function to reattach each job to its respective cpuset.

Figure 4-2. Using the `cpusetAttachPID` and `cpusetDetachPID` Functions

### Using the `cpusetMove` and `cpusetMoveMigrate` Functions

Figure 4-3, shows an example of using the `cpusetMove(3x)` and `cpusetMoveMigrate (3x)` functions.

Figure 4-3. Moving Processes From One Cpuset to Another

The `cpusetMoveMigrate` function is used to directly move a specific process and its associated memory--identified by its process ID (PID), process group ID (PGID), job ID (JID), session ID (SID), or array services handle (ASH) --to a specified cpuset. The `cpusetMove` function is used to temporarily move a process, identified by its PID, PGID, JID, or ASH--out of a specified cpuset to another cpuset or the global cpuset. In this case, the memory is not migrated (moved). Recall that global cpuset is a term used to describe all the CPUs that are not in a cpuset. Unlike the `cpusetMoveMigrate` function, the `cpusetMove` function does not move the memory associated with a process. One example of using these functions is shown in Figure 4-3. To move a process into a global cpuset from a cpuset you plan to destroy, use the `cpusetMove` function and specify the destination as NULL. You can then use the `cpusetMoveMigrate` function to move the process from the global cpuset into a newly created cpuset.

## Cpuset System Man Pages

The `man` command provides online help on all resource management commands. To view a man page online, type `man commandname`. For printed versions of the cpuset library man pages, see “Application Programming Interface for the Cpuset System” in Appendix A in Appendix A.

### User-Level Man Pages

The following user-level man pages are provided with Cpuset System software:

User-level man page | Description
`cpuset(1)` | Defines and manages a set of CPUs

### Cpuset Library Man Pages

The following cpuset Lcibrary man pages are provided with Cpuset System software:

Cpuset library man page | Description
`cpusetAllocQueueDef(3x)` | Allocates a `cpuset_QueueDef_t` structure
`cpusetAttach(3x)` | Attaches the current process to a cpuset
`cpusetAttachPID(3x)` | Attaches a specific process to a cpuset
`cpusetCreate(3x)` | Creates a cpuset
`cpusetDestroy(3x)` | Destroys a cpuset
`cpusetDetachAll(3x)` | Detaches all threads from a cpuset
`cpusetDetachPID(3x)` | Detaches a specific process from a cpuset
`cpusetFreeCPUList(3x)` | Releases memory used by a `cpuset_CPUList_t` structure
`cpusetFreeNameList(3x)` | Releases memory used by a `cpuset_NameList_t` structure
`cpusetFreePIDList(3x)` | Releases memory used by a `cpuset_PIDList_t` structure
`cpusetFreeProperties(3x)` | Releases memory used by a `cpuset_Properties_t` structure
`cpusetFreeQueueDef(3x)` | Releases memory used by a `cpuset_QueueDef_t` structure
`cpusetGetCPUCount(3x)` | Obtains the number of CPUs configured on the system
`cpusetGetCPUList(3x)` | Gets the list of all CPUs assigned to a cpuset
`cpusetGetName(3x)` | Gets the name of the cpuset to which a process is attached
`cpusetGetNameList(3x)` | Gets a list of names for all defined cpusets
`cpusetGetPIDList(3x)` | Gets a list of all PIDs attached to a cpuset
`cpusetGetProperties(3x)` | Retrieves various properties associated with a cpuset
`cpusetMove(3x)` | Temporarily moves a process, identified by its PID, PGID, JID, SID, or ASH, out of specified cpuset
`cpusetMoveMigrate(3x)` | Moves a specific process, identified by its PID, PGID, JID, SID, or ASH, and its associated memory, from one cpuset to another

### File Format Man Pages

The following file format description man pages are provided with Cpuset System software:

File Format man page | Description
`cpuset(4)` | Cpuset configuration files

### Miscellaneous Man Pages

The following miscellaneous man pages are provided with Cpuset System software:

Miscellaneous man page | Description
`cpuset(5)` | Overview of the Cpuset System

Prev | Table of Contents | Next
Chapter 3. Miser Batch Processing System |  | Chapter 5. Comprehensive System Accounting