Chapter 1. Using Checkpoint and Restart
Prev |  | Next

## Chapter 1. Using Checkpoint and Restart

This chapter introduces the IRIX Checkpoint and Restart (IRIX CPR) facility. It describes how to checkpoint and restart a process, and how to set IRIX CPR control options.

This chapter contains the following sections:

- “What Is IRIX Checkpoint and Restart”

- “Definition of Terms”

- “Checkpointing Processes”

- “Restarting Processes”

- “Querying Checkpoint Status”

- “Deleting Statefiles”

- “Graphical Interface–cview”

- “Checkpoint and Restart Attributes”

## What Is IRIX Checkpoint and Restart

IRIX Checkpoint and Restart (CPR) is a facility for saving a running process or set of processes and, at some later time, restarting the saved process or processes from the point already reached, without starting all over again. The checkpoint image is saved in a set of disk files, and restarted by reading the saved state from these files to resume execution.

The `cpr` command provides a command-line interface for checkpointing, restarting checkpointed processes, checking the status of checkpoint and restart operations, and deleting files that contain images of checkpointed processes.

Checkpointing is useful for halting and continuing resource-intensive programs that take a long time to run. IRIX CPR can help when you need to:

- Improve a system's load balancing and scheduling

- Run complex simulation or modeling applications

- Replace hardware for high-availability or fail-safe applications

Processes can continue to run after checkpoint and can be checkpointed multiple times.

The IRIX 6.5.20 release adds support for Trusted IRIX attributes to Checkpoint and Restart (CPR) processes. For more information, see “Using CPR in Trusted IRIX” in Chapter 3.

## Definition of Terms

A statefile is a directory containing information about a process or set of processes (including the names of open files and system objects). Statefiles contain all available information about a running process, to enable restart. The new process(es) should behave just as if the old process(es) had continued. Statefiles are stored as files inside a directory and are protected by normal IRIX security mechanisms.

A checkpoint owner is the owner of all checkpointed processes and the resulting statefiles. Only the checkpoint owner or superuser is permitted to perform a checkpoint. If targeted processes have multiple owners, only the superuser is permitted to checkpoint them. Only the checkpoint owner or superuser can restart checkpointed process(es) from a statefile. If the superuser performed a checkpoint, only the superuser can restart it.

A process group is a set of processes that constitute a logical job—they share the same process group ID. For example, modern UNIX shells arrange pipelined programs into a process group, so they all can be suspended and managed with the shell's job control facilities. You can determine the process group ID using the `-j` option of the `ps` command; for more information see the `ps`(1) man page. Programmers can change the process group ID using the `setpgid()` system call; for more information see the `setpgid`(2) man page.

A process session is a set of processes started from the same physical or logical terminal. Such processes share the same session ID. You can determine the process group ID and the session ID (SID) of any process by using the `-j` option to the `ps` command; for more information see the `ps`(1) man page. Programmers can change the session ID using the `setsid()` system call; for more information, see the `setsid`(2) man page.

An IRIX array session is a set of conceptually related processes running on different nodes in an array. Support is provided by the array services daemon, which knows about array configuration and provides functions for describing and administering the processes of a single job. The principal use of array services is to run jobs that are large enough to span two or more machines.

A process hierarchy is the set of all child processes with a common parent. The process hierarchy is identified by giving the process ID of the parent process. A process session is one example of a process hierarchy, but by no means the only example.

A share group is a group of processes created from a common ancestor by `sproc()` system calls; for more information see the `sproc`(2) man page. The `sproc()` call is like `fork()`, except that after `sproc()`, the new child process can share the virtual address space of the parent process. The parent and child each have their own program counter value and stack pointer, but text and data space are visible to both processes. This provides a mechanism for building parallel programs.

An IRIX job is a group of related processes all descended from a point of entry process and identified by a unique job ID. A job can contain multiple process groups, sessions, or array sessions, and all processes in one of these subgroups are always contained within one job. For more information see the `job_limits`(5) man page.

## Verifying CPR Installation

To verify that CPR runs on your system, check that the `eoe.sw.cpr` subsystem is installed:

    $ versions eoe.sw.cpr
    I = Installed, R = Removed
       Name                 Date      Description
    I  eoe                  09/28/96  IRIX Execution Environment, 6.3
    I  eoe.sw               09/14/96  IRIX Execution Environment Software
    I  eoe.sw.cpr           09/14/96  Checkpoint and Restart

If no CPR subsystem is installed, see “Installing CPR” in Chapter 2 for instructions on installing CPR.

## Checkpointing Processes

To checkpoint a set of processes (one process or more), use the `-c` option of the `cpr` command, providing a statefile name, and specifying a process ID with the `-p` option. For example, to checkpoint process 1111 into statefile `ckptSep7`, enter the following:

    $ cpr -c ckptSep7 -p 1111

To checkpoint all processes in a process group, enter the process group ID (for example, 123) followed by the : GID modifier:

    $ cpr -c statefile -p 123:GID

To checkpoint all processes in a process session, enter the process session ID (for example, 345) followed by the : SID modifier:

    $ cpr -c statefile -p 345:SID

To checkpoint all processes in an IRIX array session, enter the array session ID (for example, 0x8000abcd00001111) followed by the : ASH modifier:

    $ cpr -c statefile -p 0x8000abcd00001111:ASH

To checkpoint all processes in a process hierarchy, enter the parent process ID (for example, 567) followed by the : HID modifier:

    $ cpr -c statefile -p 567:HID

To checkpoint all processes in an `sproc()` share group, enter the share group ID (for example, 789) followed by the : SGP modifier:

    $ cpr -c statefile -p 789:SGP

To checkpoint all processes in an IRIX job, enter the job ID (for example, 0x8000abcd00001234) followed by the :JID modifier:

    $ cpr -c statefile -p 0x8000abcd00001234:JID

It is possible to combine process designators using the comma separator, as in the following example. All processes are recorded in the same statefile.

    $ cpr -c ckptSep8 -p 1113,1225,1397:HID

The `-w` option specifies that `cpr` use the attribute file located in the current working directory (versus `$HOME/.cpr`).

$ `cpr -c -w ckptDec13 -p 1113`

### Naming the Checkpoint Image

You can place the statefile anywhere, provided you have write permission for the target directory, and provided there is enough disk space to store the checkpoint images. You might want to include the date as part of the statefile name, or you might want to number statefiles consecutively. The `-f` option of the `cpr` command forces an overwrite of an existing statefile.

### Job Control Shells

The C shell (`csh`), Korn shell (`ksh` or, after IRIX 6.3, `sh`), Tops C shell (`tcsh`), and GNU shell (`bash`) all support job control. The Bourne shell (`bsh`, formerly `sh`) does not. Jobs can be suspended with `Ctrl+Z`, backgrounded with the `bg` built-in command, or foregrounded with `fg`. All job control shells provide the `jobs` built-in command with an `-l` option to list process ID numbers and a `-p` option to show the process group ID of a job.

## Restarting Processes

To restart a set of processes (one process or more), use the `-r` option of the `cpr` command, providing just the statefile name. For example, to restart the set of processes checkpointed in `ckptSep7`, enter the following:

    $ cpr -jm -r ckptSep7

Use the `-j` option if you want to perform interactive job control after restart. Otherwise, the process group restored belongs to `init`, effectively disabling job control.

Use the `-m` option if you want to migrate the checkpointed memory to the location in the system topology where the restart operation is executing.

    $ cpr -c -w ckptDec13 -p 1113

Use `-w` option if you want to use the attribute file located in the current working directory (versus `$HOME/.cpr`).

You may restart more than one statefile with the same `cpr` command. If a restart involves more than one process, all restarts must succeed before any process is allowed to run; otherwise all restarts fail. Restart failure can occur for any of the following reasons:

unavailable PID
| The original process ID is not available (already in use), and the option to allow ANY process ID was not in effect.
component unavailable
| Application binaries or libraries are no longer available on the system, and neither the REPLACE nor SUBSTITUTE option was in effect.
security and data integrity
| The user lacks proper permission to restart the statefile, or the restart will destroy or replace data without proper authorization. Only the checkpoint owner and the superuser may restart a set of processes.
resource limitation
| System resources such as disk space, memory (swap space), or number of processes allowed, ran out during restart.
file contents change
| If the CONTENTS action was used for FILE policies in the user's `cpr` attribute file, the restart could fail if file contents have changed between checkpoint and the restart. For more information, see “FILE Policy”.
other fatal failure
| Some important part of a process restart failed for unknown reasons.

### Persistence of Statefiles

The statefile remains unchanged after restart; `cpr` does not delete it automatically. To free disk space, use the `-D` option of `cpr`; for more information, see the section “Deleting Statefiles”.

### Job Control Option

If a checkpoint is issued against an interactive process or a group of processes rooted at an interactive process, it can be restarted interactively using the with `-j` option of the `cpr` command. This option makes processes interactive and job-controllable. The restarted processes run in the foreground, even the original ones ran in the background. Users may issue job control signals to background the process if desired. An interactive job is defined as a process with a controlling terminal; for more information see the `termio`(7) man page. Only one controlling terminal is restored even if the original process had multiple controlling terminals.

### Memory Migration Option

The `-m` option of the `cpr` command migrates process memory so it is restored to the location in the system topology where the restart operation is executing, for example, within a specific cpuset, within the global cpuset, and so on. Without this option, the default restart behavior on NUMA systems is to restore process memory back to where it was at the time of the checkpoint. See the `migration`(3) man page for scenarios that may prevent pages from migrating properly. This option has no effect on non-NUMA systems.

## Querying Checkpoint Status

To obtain information about checkpoint status, employ the `-i` option of the `cpr` command, providing the statefile name. You may query more than one statefile at a time. For example, to get information about the set of processes checkpointed in `ckptSep7`, either before or after restart, enter the following command:

    $ cpr -i ckptSep7

This displays information about the statefile revision number, process names, credential information for the processes, the current working directory, open file information, the time when the checkpoint was done, and so forth.

## Deleting Statefiles

To delete a statefile and its associated open files and system objects, use the `-D` option of the `cpr` command, providing a statefile name. You may delete more than one statefile at a time. For example, to delete the file `ckptSep7`, enter the following command:

    $ cpr -D ckptSep7

Only the checkpoint owner and the superuser may delete a statefile directory. Once a checkpoint statefile has been deleted, restart is no longer possible.

## Graphical Interface–`cview`

The `cview` command brings up a graphical interface for CPR and provides access to some features of the `cpr` command. As of the IRIX 6.5.16 release, new features are no longer being added to the `cview` command or interface. The `cview` command will be removed in the next major release of the IRIX operating system. The checkpoint control panel, shown in Figure 1-1, displays a list of processes that may be checkpointed.

Figure 1-1. Checkpoint Control Panel (`cview`)

Checkpoint options may be set in step II, and are explained in the section “Checkpoint and Restart Attributes”. Click the right tab at the bottom to switch panels.

The restart control panel, shown in Figure 1-2, displays a list of statefiles that may be restarted. The buttons near the bottom query checkpoints and delete statefiles.

Figure 1-2. Restart Control Panel (`cview`)

## Checkpoint and Restart Attributes

The `cpr` command reads an attribute file at start-up time to set checkpoint configuration and control restart behavior. Typical defaults are given in the `/etc/cpr_proto` sample file. You can control CPR behavior by creating a similar `.cpr` attribute file in your home directory (if $HOME is not set, `cpr` consults the password entry). The CPR attribute file consists of one or more CKPT attribute definitions, each in the following format:

    CKPT IDtype IDvalue {
        policy:  instance:  action
        ...
    }

Possible values for IDtype are similar to process ID modifiers for the `-c` option of `cpr`, and are shown in Table 1-1. IDvalue specifies the process ID or process set ID.

Table 1-1. IDtype Modifier Options

IDtype | Process Type Designation
--- | ---
PID | UNIX process ID or POSIX thread ID.
GID | UNIX process group ID; see `setpgrp`.
SID | UNIX process session ID; see `setsid`(2).
ASH | IRIX array session ID; see `array_sessions`(5).
JID | IRIX job ID : see `job_limits`(5).
HID | Process hierarchy (tree) rooted at the given process ID.
SGP | IRIX `sproc()` shared group; see `sproc`(2).
* | A wild card for anything.

The policy lines inside the CKPT block specify default actions for CPR to take. Possible values for policy are shown in Table 1-2.

Table 1-2. Policy Names and Actions

Policy Name | Domain of Action
--- | ---
FILE | Policies for handling open files.
WILL | Actions on the original process after checkpoint.
CDIR | Policy on the original working directory; see `chdir`(2).
RDIR | Policy on the original root directory; see `chroot`(2).
FORK | Policy on original process ID.

### FILE Policy

The FILE policy can take an optional instance field. This field specifies files that have a unique disposition, other than the default action. For example, in one case you want to replace a file, but in another case you want to append to a file. The instance field is enclosed in double quotes and may contain wildcards. For example, `/tmp/*` identifies all files in the `/tmp` directory, and `/*` identifies all files in the system.

The following action keywords are available for the FILE policy:

MERGE |  | No explicit file save at checkpoint. Upon restart, reopen the file and seek to the previous offset. This is the default file disposition. It may be used for files that are not modified after checkpoint, or for files where it is acceptable to overwrite changes made between checkpoint and restart time, particularly past the saved offset point. If programs seek before writing, changes preceding the offset point could be overwritten as well.
IGNORE |  | No explicit file save at checkpoint. Upon restart, reopen the file as it was originally opened, at offset zero (even if originally opened for append). If the file was originally opened for writing, as with the `fopen()` “w” or “a” flag, this action has the effect of overwriting the entire file.
APPEND |  | No explicit file save at checkpoint. Upon restart, append to the end of the file. This disposition is good for log files.
REPLACE |  | Explicitly save the file at checkpoint. Upon restart, replace the original file with the saved one. Any changes made to the original file between checkpoint and restart time are overwritten by the saved file.
SUBSTITUTE |  | Explicitly save the file at checkpoint. Upon restart, reopen the saved file as an anonymous substitution for the original file. This is similar to the REPLACE mode except that the original file remains untouched, unless specifically altered by the program.
CONTENTS |  | Calculate checksum (currently MD5) on the file at checkpoint. Upon restart, detect if the file has been modified between begin-of-file and file-size-at-checkpoint; if the file has been modified in this area, the process is refused restart, otherwise, seek to the previous offset and continue.

### WILL Policy

The following action keywords are available for the WILL policy:

EXIT |  | The original process exits after checkpoint. This is the default action.
KILL |  | Same as above. Has the same effect as the `cpr` `-k` option.
CONT |  | The original process continues to run after checkpoint. Has the same effect as the `cpr` `-g` option.

### CDIR or RDIR Policy

The following action keywords are available for the CDIR and RDIR policies:

REPLACE |  | Set the current working directory (CDIR) or the root directory (RDIR) to those of the original process. This is the default action.
IGNORE |  | Ignore the current working directory (CDIR) or the root directory (RDIR) of the original process, and restart processes based on the current working directory or the root directory of the new process environment.

### FORK Policy

The FORK policy can take an optional instance field, either PID or JID. If no instance is specified, the specified action is applied to all instances. The following action keywords are available for the FORK policy:

ORIGINAL |  | Do a special `fork()` to recover the original process ID. This is the default action.
ANY |  | This says it is acceptable for the application to have any process ID as its underlying process if the original process ID is already taken by another running process. In other words, the application itself, internally and in its relationship to other processes, is PID-unaware. If a set of processes is PID-unaware, the ANY action can be specified to avoid PID collisions.

There is no attribute equivalent to the `cpr` `-u` option for operating system upgrade.

### Example Attribute File

The `$HOME/cpr` file specifies a user's CPR default attributes. Here is an example of a custom `.cpr` attribute file:

    CKPT PID 1111 {
        FILE:   "/tmp/*": 	 REPLACE
        WILL: 	 	 	 	 	 	 	 	 	 	 	 	 	 CONT
        FORK: 	 	 	PID: 				 	 	 	 	 	 	ANY
    }

This saves and restores all `/tmp` files, allows the process to continue after checkpoint, and permits process ID substitution if needed.

Prev | Table of Contents | Next
About This Guide |  | Chapter 2. Administering Checkpoint and Restart