Chapter 7. Array Services
Prev |  | Next

## Chapter 7. Array Services

Array Services includes administrator commands, libraries, daemons, and kernel extensions that support the execution of programs across an array.

A central concept in Array Services is the array session handle (ASH), a number that is used to logically group related processes that may be distributed across multiple systems. The ASH creates a global process namespace across the Array, facilitating accounting and administration

Array Services also provides an array configuration database, listing the nodes comprising an array. Array inventory inquiry functions provide a centralized, canonical view of the configuration of each node. Other array utilities let the administrator query and manipulate distributed array applications.

The Array Services package comprises the following primary components:

array daemon |  | Allocates ASH values and maintain information about node configuration and the relation of process IDs to ASHs. Array daemons reside on each node and work in cooperation.

array configuration database |  | Describes the array configuration used by array daemons and user programs. One copy at each node.
`ainfo` command |  | Lets the user or administrator query the Array configuration database and information about ASH values and processes.
`array` command |  | Executes a specified command on one or more nodes. Commands are predefined by the administrator in the configuration database.
`arshell` command |  | Starts a command remotely on a different node using the current ASH value.
`aview` command |  | Displays a multiwindow, graphical display of each node's status.
`libarray` library |  | Library of functions that allow user programs to call on the services of array daemons and the array configuration database.

The use of the `ainfo`, `array`, `arshell`, and `aview` commands is covered in “Using an Array”. The use of the `libarray` library is covered in“Array Services Library” .

## Using an Array

An Array system is an aggregation of nodes, which are servers bound together with a high-speed network and Array Services 3.5 software. Array users have the advantage of greater performance and additional services. Array users access the system with familiar commands for job control, login and password management, and remote execution.

Array Services 3.5 augments conventional facilities with additional services for array users and for array administrators. The extensions include support for global session management, array configuration management, batch processing, message passing, system administration, and performance visualization.

This section introduces the extensions for Array use, with pointers to more detailed information. The main topics are as follows:

- “Using an Array System”, summarizes what a user needs to know and the main facilities a user has available.

- “Managing Local Processes”, reviews the conventional tools for listing and controlling processes within one node.

- “Using Array Services Commands”, describes the common concepts, options, and environment variables used by the Array Services commands.

- “Interrogating the Array”, summarizes how to use Array Services commands to learn about the Array and its workload, with examples.

- “Summary of Common Command Options”

- “Managing Distributed Processes”, summarizes how to use Array Services commands to list and control processes in multiple nodes.

### Using an Array System

The array system allows you to run distributed sessions on multiple nodes of an array. You can access the Array from either:

- A workstation

- An X terminal

- An ASCII terminal

In each case, you log in to one node of the Array in the way you would log in to any remote UNIX host. From a workstation or an X terminal you can of course open more than one terminal window and log into more than one node.

#### Finding Basic Usage Information

In order to use an Array, you need the following items of information:

- The name of the Array.

  You use this arrayname in Array Services commands.

- The login name and password you will use on the Array.

  You use these when logging in to the Array to use it.

- The hostnames of the array nodes.

  Typically these names follow a simple pattern, often arrayname1, arrayname2, and so on.

- Any special resource-distribution or accounting rules that may apply to you or your group under a job scheduling system.

You can learn the hostnames of the array nodes if you know the array name, using the `ainfo` command as follows:

    ainfo -a arrayname machines

#### Logging In to an Array

Each node in an Array has an associated hostname and IP network address. Typically, you use an Array by logging in to one node directly, or by logging in remotely from another host (such as the Array console or a networked workstation). For example, from a workstation on the same network, this command would log you in to the node named `hydra6` as follows:

    rlogin hydra6

For details of the `rlogin` command, see the `rlogin(1)` man page.

The system administrators of your array may choose to disallow direct node logins in order to schedule array resources. If your site is configured to disallow direct node logins, your administrators will be able to tell you how you are expected to submit work to the array-perhaps through remote execution software or batch queueing facilities.

#### Invoking a Program

Once you have access to an array, you can invoke programs of several classes:

- Ordinary (sequential) applications

- Parallel shared-memory applications within a node

- Parallel message-passing applications within a node

- Parallel message-passing applications distributed over multiple nodes (and possibly other servers on the same network running Array Services 3.5

If you are allowed to do so, you can invoke programs explicitly from a logged-in shell command line; or you may use remote execution or a batch queueing system.

Programs that are X Windows clients must be started from an X server, either an X Terminal or a workstation running X Windows.

Some application classes may require input in the form of command line options, environment variables, or support files upon execution. For example:

- X client applications need the `DISPLAY` environment variable set to specify the X server (workstation or X-terminal) where their windows will display.

- A multithreaded program may require environment variables to be set describing the number of threads.

  For example, C and Fortran programs that use parallel processing directives test the `MP_SET_NUMTHREADS` variable.

- Message Passing Interface (MPI) and Parallel Virtual Machine (PVM) message-passing programs may require support files to describe how many tasks to invoke on specified nodes.

Some information sources on program invocation are listed in Table 7-1.

Table 7-1. Information Sources for Invoking a Program

Topic | Man Page |  |
Remote login | `rlogin(1)` |  |
Setting environment variables | `environ(5)`, `env(1)` |  |

## Managing Local Processes

Each UNIX process has a process identifier (PID), a number that identifies that process within the node where it runs. It is important to realize that a PID is local to the node; so it is possible to have processes in different nodes using the same PID numbers.

Within a node, processes can be logically grouped in process groups. A process group is composed of a parent process together with all the processes that it creates. Each process group has a process group identifier (PGID). Like a PID, a PGID is defined locally to that node, and there is no guarantee of uniqueness across the Array.

### Monitoring Local Processes and System Usage

You query the status of processes using the system command `ps`. To generate a full list of all processes on a local system, use a command such as the following:

    ps -elfj

You can monitor the activity of processes using the command `top` (an ASCII display in a terminal window).

### Scheduling and Killing Local Processes

You can start a process at a reduced priority, so that it interferes less with other processes, using the `nice` command. If you use the csh shell, specify `/usr/bin/nice` to avoid the built-in shell command `nice`. To start a whole shell at low priority, use a command like the one that follows:

    /bin/nice /bin/sh

You can schedule commands to run at specific times using the `at` command. You can kill or stop processes using the `kill` command. To destroy the process with PID 13032, use a command such as the following:

    kill -KILL 13032

### Summary of Local Process Management Commands

Table 7-2, summarizes information about local process management.

Table 7-2. Information Sources: Local Process Management

Topic | Man Page
--- | ---
Process ID and process group | ``intro(2)``
Listing and monitoring processes | `ps(1)`, `top(1)`
Running programs at low priority | `nice(1)`, `batch(1)`
Running programs at a scheduled time | `at(1)`
Terminating a process | `kill(1)`

## Using Array Services Commands

When an application starts processes on more than one node, the PID and PGID are no longer adequate to manage the application. The commands of Array Services 3.5 give you the ability to view the entire array, and to control the processes of multinode programs.

| Note: You can use Array Services commands from any workstation connected to an array system. You don't have to be logged in to an array node.

The following commands are common to Array Services operations as shown in Table 7-3.

Table 7-3. Common Array Services Commands

Topic | Man Page
--- | ---
Array Services Overview | `array_services(5)`
`ainfo` command | `ainfo(1)`
`array` command | Use `array(1)`; configuration: `arrayd.conf(4)`
`arshell` command | `arshell(1)`
`aview` command | `aview(1)`
`newsess` command | `newsess (1)`

### About Array Sessions

Array Services is composed of a daemon-a background process that is started at boot time in every node-and a set of commands such as `ainfo(1)`. The commands call on the daemon process in each node to get the information they need.

One concept that is basic to Array Services is the array session, which is a term for all the processes of one application, wherever they may execute. Normally, your login shell, with the programs you start from it, constitutes an array session. A batch job is an array session; and you can create a new shell with a new array session identity.

Each session is identified by an array session handle (ASH), a number that identifies any process that is part of that session. You use the ASH to query and to control all the processes of a program, even when they are running in different nodes.

### About Names of Arrays and Nodes

Each node is server, and as such has a hostname. The hostname of a node is returned by the `hostname(1)` command executed in that node as follows:

    % hostname
    tokyo

The command is simple and documented in the `hostname(1)` man page. The more complicated issues of hostname syntax, and of how hostnames are resolved to hardware addresses are covered in `hostname(5)`.

An Array system as a whole has a name too. In most installations there is only a single Array, and you never need to specify which Array you mean. However, it is possible to have multiple Arrays available on a network, and you can direct Array Services commands to a specific Array.

### About Authentication Keys

It is possible for the Array administrator to establish an authentication code, which is a 64-bit number, for all or some of the nodes in an array (see "Configuring Authentication Codes" on page 58). When this is done, each use of an Array Services command must specify the appropriate authentication key, as a command option, for the nodes it uses. Your system administrator will tell you if this is necessary.

## Summary of Common Command Options

The following Array Services commands have a consistent set of command options: `ainfo(1)`, `array(1)`, `arshell(1)`, `aview(1)`, and `newsess(1)`. Table 7-4 is a summary of these options. Not all options are valid with all commands; and each command has unique options besides those shown. The default values of some options are set by environment variables listed in the next topic.

Table 7-4. Array Services Command Option Summary

Option | Used In | Description
--- | --- | ---
`-a` array | `ainfo`, `array`, `aview` | Specify a particular Array when more than one is accessible.
`-D` | `ainfo`, `array`, `arshell`, `aview` | Send commands to other nodes directly, rather than through `array` daemon.
`-F` | `ainfo`, `array`, `arshell, aview` | Forward commands to other nodes through the `array` daemon.
`-Kl` number | `ainfo`, `array`, `aview` | Authentication key (a 64-bit number) for the local node.
`-Kr` number | `ainfo`,`array`, `aview` | Authentication key (a 64-bit number) for the remote node.
`-l` (letter ell) | `ainfo`,`array` | Execute in context of the destination node, not necessarily the current node.
`-l````` port | `ainfo`, `array`, `arshell`,`aview` | Nonstandard port number of array daemon.
-`s` hostname | `ainfo`, `array`, `aview` | Specify a destination node.

### Specifying a Single Node

The `-l` and `-s` options work together. The `-l` (letter ell for “local”) option restricts the scope of a command to the node where the command is executed. By default, that is the node where the command is entered. When `-l` is not used, the scope of a query command is all nodes of the array. The `-s` (server, or node name) option directs the command to be executed on a specified node of the array. These options work together in query commands as follows:

- To interrogate all nodes as seen by the local node, use neither option.

- To interrogate only the local node, use only `-l`.

- To interrogate all nodes as seen by a specified node, use only `-s`.

- To interrogate only a particular node, use both `-s` and `-l`.

### Common Environment Variables

The Array Services commands depend on environment variables to define default values for the less-common command options. These variables are summarized in Table 7-5.

Table 7-5. Array Services Environment Variables

Variable Name | Use | Default When Undefined
--- | --- | ---
`ARRAYD_FORWARD` | When defined with a string starting with the letter y, all commands default to forwarding through the array daemon (option `-F`). | Commands default to direct communication (option `-D`).
`ARRAYD_PORT` | The port (socket) number monitored by the array daemon on the destination node. | The standard number of 5434, or the number given with option `-p`.
`ARRAYD_LOCALKEY` | Authentication key for the local node (option `-Kl`). | No authentication unless `-Kl` option is used.
`ARRAYD_REMOTEKEY` | Authentication key for the destination node (option `-Kr`). | No authentication unless `-Kr` option is used.
`ARRAYD` | The destination node, when not specified by the `-s` option. | The local node, or the node given with `-s`.

## Interrogating the Array

Any user of an Array system can use Array Services commands to check the hardware components and the software workload of the Array. The commands needed are `ainfo`, `array`, and `aview`.

### Learning Array Names

If your network includes more than one Array system, you can use `ainfo arrays` at one array node to list all the Array names that are configured, as in the following example.

    homegrown% ainfo arrays
    Arrays known to array services daemon
    ARRAY DevArray
        IDENT 0x3381
    ARRAY BigDevArray
        IDENT 0x7456
    ARRAY test
        IDENT 0x655e

Array names are configured into the array database by the administrator. Different Arrays might know different sets of other Array names.

### Learning Node Names

You can use `ainfo machines` to learn the names and some features of all nodes in the current Array, as in the following example.

    homegrown 175% ainfo -b machines
    machine homegrown homegrown 5434 192.48.165.36 0
    machine disarray disarray 5434 192.48.165.62 0
    machine datarray datarray 5434 192.48.165.64 0
    machine tokyo tokyo 5434 150.166.39.39 0

In this example, the `-b` option of `ainfo` is used to get a concise display.

### Learning Node Features

You can use `ainfo nodeinfo` to request detailed information about one or all nodes in the array. To get information about the local node, use `ainfo -l nodeinfo`. However, to get information about only a particular other node, for example node `tokyo`, use `-l` and `-s`, as in the following example. (The example has been edited for brevity.)

    homegrown 181% ainfo -s tokyo -l nodeinfo
    Node information for server on machine "tokyo"
    MACHINE tokyo
        VERSION  1.2
        8 PROCESSOR BOARDS
            BOARD: TYPE 15   SPEED 190
                CPU:   TYPE 9   REVISION 2.4
                FPU:   TYPE 9   REVISION 0.0
    ...
        16 IP INTERFACES  HOSTNAME tokyo   HOSTID 0xc01a5035
            DEVICE  et0    NETWORK    150.166.39.0    ADDRESS   150.166.39.39  UP
            DEVICE atm0    NETWORK 255.255.255.255    ADDRESS         0.0.0.0  UP
            DEVICE atm1    NETWORK 255.255.255.255    ADDRESS         0.0.0.0  UP
    ...
        0 GRAPHICS INTERFACES
        MEMORY
            512 MB MAIN MEMORY
            INTERLEAVE 4

If the `-l` option is omitted, the destination node will return information about every node that it knows.

### Learning User Names and Workload

The system commands `who(1)`, `top(1)`, and `uptime(1)` are commonly used to get information about users and workload on one server. The `array(1)` command offers Array-wide equivalents to these commands.

#### Learning User Names

To get the names of all users logged in to the whole array, use `array who`. To learn the names of users logged in to a particular node, for example `tokyo`, use `-l` and `-s`, as in the following example. (The example has been edited for brevity and security.)

     homegrown 180% array -s tokyo -l who
    joecd    tokyo        frummage.eng.sgi -tcsh
    joecd    tokyo        frummage.eng.sgi -tcsh
    benf     tokyo        einstein.ued.sgi. /bin/tcsh
    yohn     tokyo        rayleigh.eng.sg vi +153 fs/procfs/prd
    ...

#### Learning Workload

Two variants of the `array` command return workload information. The array-wide equivalent of `uptime` is `array uptime`, as follows:

    homegrown 181% array uptime
       homegrown:  up 1 day,  7:40,  26 users,  load average: 7.21, 6.35, 4.72
        disarray:  up  2:53,  0 user,  load average: 0.00, 0.00, 0.00
        datarray:  up  5:34,  1 user,  load average: 0.00, 0.00, 0.00
           tokyo:  up 7 days,  9:11,  17 users,  load average: 0.15, 0.31, 0.29
    homegrown 182% array -l -s tokyo uptime
           tokyo:  up 7 days,  9:11,  17 users,  load average: 0.12, 0.30, 0.28

The command `array top` lists the processes that are currently using the most CPU time, with their ASH values, as in the following example.

    homegrown 183% array top
            ASH        Host           PID User       %CPU Command
    ----------------------------------------------------------------
    0x1111ffff00000000 homegrown        5 root       1.20 vfs_sync
    0x1111ffff000001e9 homegrown     1327 guest      1.19 atop
    0x1111ffff000001e9 tokyo        19816 guest      0.73 atop
    0x1111ffff000001e9 disarray      1106 guest      0.47 atop
    0x1111ffff000001e9 datarray      1423 guest      0.42 atop
    0x1111ffff000000c0 homegrown    29683 kchang     0.37 ld
    0x1111ffff0000001e homegrown     1324 root       0.17 arrayd
    0x1111ffff00000000 homegrown      229 root       0.14 routed
    0x1111ffff00000000 homegrown       19 root       0.09 pdflush
    0x1111ffff000001e9 disarray      1105 guest      0.02 atopm

The `-l` and `-s` options can be used to select data about a single node, as usual.

### Browsing With ArrayView

The ArrayView window shows the status of an array. You can start it with the command `aview` and it displays a window similar to the one shown in Figure 7-1. The top window shows one line per node. There is a window for each node, headed by the node name and its hardware configuration. Each window contains a snapshot of the busiest processes in that node.

Figure 7-1. Typical Display from ArrayView

## Managing Distributed Processes

Using commands from Array Services 3.5, you can create and manage processes that are distributed across multiple nodes of the Array system.

### About Array Session Handles (ASH)

In an Array system you can start a program with processes that are in more than one node. In order to name such collections of processes, Array Services 3.5 software assigns each process to an array session handle (ASH).

An ASH is a number that is unique across the entire array (unlike a PID or PGID). An ASH is the same for every process that is part of a single array session--no matter which node the process runs in. You display and use ASH values with Array Services commands. Each time you log in to an Array node, your shell is given an ASH, which is used by all the processes you start from that shell.

The command `ainfo ash` returns the ASH of the current process on the local node, which is simply the ASH of the `ainfo` command itself.

    homegrown 178% ainfo ash
    Array session handle of process 10068: 0x1111ffff000002c1
    homegrown 179% ainfo ash
    Array session handle of process 10069: 0x1111ffff000002c1

In the preceding example, each instance of the `ainfo` command was a new process: first PID 10068, then PID 10069. However, the ASH is the same in both cases. This illustrates a very important rule: every process inherits its parent's ASH. In this case, each instance of `array` was forked by the command shell, and the ASH value shown is that of the shell, inherited by the child process.

You can create a new global ASH with the command `ainfo newash`, as follows:

    homegrown 175% ainfo newash
    Allocating new global ASH
    0x11110000308b2f7c

This feature has little use at present. There is no existing command that can change its ASH, so you cannot assign the new ASH to another command. It is possible to write a program that takes an ASH from a command-line option and uses the Array Services function `setash()` to change to that ASH (however such a program must be privileged). No such program is distributed with Array Services 3.5 (but see “Managing Array Service Handles”).

### Listing Processes and ASH Values

The command `array ps` returns a summary of all processes running on all nodes in an array. The display shows the ASH, the node, the PID, the associated username, the accumulated CPU time, and the command string.

To list all the processes on a particular node, use the `-l` and `-s` options. To list processes associated with a particular ASH, or a particular username, pipe the returned values through `grep`, as in the following example. (The display has been edited to save space.)

    homegrown 182% array -l -s tokyo ps | fgrep wombat
    0x261cffff0000054c        tokyo 19007   wombat    0:00 -csh
    0x261cffff0000054a        tokyo 17940   wombat    0:00 csh -c (setenv...
    0x261cffff0000054c        tokyo 18941   wombat    0:00 csh -c (setenv...
    0x261cffff0000054a        tokyo 17957   wombat    0:44 xem -geometry 84x42
    0x261cffff0000054a        tokyo 17938   wombat    0:00 rshd
    0x261cffff0000054a        tokyo 18022   wombat    0:00 /bin/csh -i
    0x261cffff0000054a        tokyo 17980   wombat    0:03 /usr/gnu/lib/ema...
    0x261cffff0000054c        tokyo 18928   wombat    0:00 rshd

### Controlling Processes

The `arshell` command lets you start an arbitrary program on a single other node. The `array` command gives you the ability to suspend, resume, or kill all processes associated with a specified ASH.

#### Using arshell

The `arshell` command is an Array Services extension of the familiar `rsh` command; it executes a single system command on a specified Array node. The difference from `rsh` is that the remote shell executes under the same ASH as the invoking shell (this is not true of simple `rsh`). The following example demonstrates the difference.

    homegrown 179% ainfo ash
    Array session handle of process 8506: 0x1111ffff00000425
    homegrown 180% rsh guest@tokyo ainfo ash
    Array session handle of process 13113: 0x261cffff0000145e
    homegrown 181% arshell guest@tokyo ainfo ash
    Array session handle of process 13119: 0x1111ffff00000425

You can use `arshell` to start a collection of unrelated programs in multiple nodes under a single ASH; then you can use the commands described under “Managing Session Processes” to stop, resume, or kill them.

Both MPI and PVM use `arshell` to start up distributed processes.

| Tip: The shell is a process under its own ASH. If you use the `array` command to stop or kill all processes started from a shell, you will stop or kill the shell also. In order to create a group of programs under a single ASH that can be killed safely, proceed as follows: Create a nested shell with a new ASH using `newsess`. Note the ASH value. Within the new shell, start one or more programs using `arshell`. Exit the nested shell. Now you are back to the original shell. You know the ASH of all programs started from the nested shell. You can safely kill all jobs that have that ASH because the current shell is not affected.

#### About the Distributed Example

The programs launched with `arshell` are not coordinated (they could of course be written to communicate with each other, for example using sockets), and you must start each program individually.

The `array` command is designed to permit the simultaneous launch of programs on all nodes with a single command. However, `array` can only launch programs that have been configured into it, in the Array Services configuration file. (The creation and management of this file is discussed under “About Array Configuration”.)

In order to demonstrate process management in a simple way from the command line, the following command was inserted into the configuration file `/usr/lib/array/arrayd.conf`:

    #
    # Local commands
    #
    command spin                    # Do nothing on multiple machines
            invoke /usr/lib/array/spin
            user    %USER
            group   %GROUP
            options nowait

The invoked command, `/usr/lib/array/spin`, is a shell script that does nothing in a loop, as follows:

    #!/bin/sh
    # Go into a tight loop
    #
    interrupted() {
            echo "spin has been interrupted - goodbye"
            exit 0
    }
    trap interrupted 1 2
    while [ ! -f /tmp/spin.stop ]; do
            sleep 5
    done
    echo "spin has been stopped - goodbye"
    exit 1

With this preparation, the command `array spin` starts a process executing that script on every processor in the array. Alternatively, `array -l -s nodename spin` would start a process on one specific node.

#### Managing Session Processes

The following command sequence creates and then kills a `spin` process in every node. The first step creates a new session with its own ASH. This is so that later, `array kill` can be used without killing the interactive shell.

    homegrown 175% ainfo ash
    Array session handle of process 8912: 0x1111ffff0000032d
    homegrown 176% newsess
    homegrown 175% ainfo ash
    Array session handle of process 8941: 0x11110000308b2fa6

In the new session with ASH 0x11110000308b2fa6, the command array `spin` starts the `/usr/lib/array/spin` script on every node. In this test array, there were only two nodes on this day, `homegrown` and `tokyo`.

    homegrown 176% array spin

After exiting back to the original shell, the command `array ps` is used to search for all processes that have the ASH 0x11110000308b2fa6.

    homegrown 177% exit
    homegrown 178% homegrown 177%
    homegrown 177% ainfo ash
    Array session handle of process 9257: 0x1111ffff0000032d
    homegrown 179% array ps | fgrep 0x11110000308b2fa6
    0x11110000308b2fa6  homegrown  9033   guest   0:00 /bin/sh /usr/lib/array/spin
    0x11110000308b2fa6  homegrown  9618  guest  0:00 sleep 5
    0x11110000308b2fa6        tokyo 26021  guest   0:00 /bin/sh /usr/lib/array/spin
    0x11110000308b2fa6      tokyo 26072  guest  0:00 sleep 5
    0x1111ffff0000032d  homegrown  9642  guest  0:00 fgrep 0x11110000308b2fa6

There are two processes related to the `spin` script on each node. The next command kills them all.

    homegrown 180% array kill 0x11110000308b2fa6
    homegrown 181% array ps | fgrep 0x11110000308b2fa6
    0x1111ffff0000032d  homegrown 10030  guest  0:00 fgrep 0x11110000308b2fa6

The command `array suspend 0x11110000308b2fa6` would suspend the processes instead (however, it is hard to demonstrate that a `sleep` command has been suspended).

#### About Job Container IDs

Array systems that are running IRIX 6.5.7f (or later) and Array Services version 3.4 (or later) have the capability to forward job IDs (JIDs) from the initiating host. All of the processes running in the ASH across one or more nodes in an array also belong to the same job. For a complete description of the job container and it usage, see the `job_limits(5)` man page or IRIX Admin: Resource Administration.

When processes are running on the initiating host, they belong to the same job as the initiating process and operate under the limits established for that job. On remote nodes, a new job is created using the same JID as the initiating process. Job limits for a job on remote nodes use the `systune` defaults and are set using the `systune(1M)` command on the initiating host.

## About Array Configuration

The system administrator has to initialize the Array configuration database, a file that is used by the Array Services daemon in executing almost every `ainfo` and `array` command. For details about array configuration, see the man pages cited in Table 7-6.

Table 7-6. Information Sources: Array Configuration

Topic | Man Page
--- | ---
Array Services overview | `array_services(5)`
Array Services user commands | `ainfo(1)` , `array(1)`
Array Services daemon overview | `arrayd(1m)`
Configuration file format | `arrayd.conf(4)` , `/usr/lib/array/arrayd.conf.template`
Configuration file validator | `ascheck(1)`
Array Services simple configurator | `arrayconfig(1m)`

### About the Uses of the Configuration File

The configuration files are read by the Array Services daemon when it starts. Normally it is started in each node during the system startup. (You can also run the daemon from a command line in order to check the syntax of the configuration files.)

The configuration files contain data needed by `ainfo` and `array`:

- The names of Array systems, including the current Array but also any other Arrays on which a user could run an Array Services command (reported by `ainfo`).

- The names and types of the nodes in each named Array, especially the hostnames that would be used in an Array Services command (reported by `ainfo`).

- The authentication keys, if any, that must be used with Array Services commands (required as `-Kl` and `-Kr` command options, see “Summary of Common Command Options”).

- The commands that are valid with the `array` command.

### About Configuration File Format and Contents

A configuration file is a readable text file. The file contains entries of the following four types, which are detailed in later topics.

Array definition |  | Describes this array and other known arrays, including array names and the node names and types.
Command definition |  | Specifies the usage and operation of a command that can be invoked through the `array` command.
Authentication |  | Specifies authentication numbers that must be used to access the Array.
Local option |  | Options that modify the operation of the other entries or `arrayd`.

Blank lines, white space, and comment lines beginning with “#” can be used freely for readability. Entries can be in any order in any of the files read by `arrayd`.

Besides punctuation, entries are formed with a keyword-based syntax. Keyword recognition is not case-sensitive; however keywords are shown in uppercase in this text and in the man page. The entries are primarily formed from keywords, numbers, and quoted strings, as detailed in the man page `arrayd.conf(4)` .

### Loading Configuration Data

The Array Services daemon, `arrayd`, can take one or more filenames as arguments. It reads them all, and treats them like logical continuations (in effect, it concatenates them). If no filenames are specified, it reads `/usr/lib/array/arrayd.conf` and `/usr/lib/array/arrayd.auth`. A different set of files, and any other `arrayd` command-line options, can be written into the file `/etc/config/arrayd.options`, which is read by the startup script that launches `arrayd` at boot time.

Since configuration data can be stored in two or more files, you can combine different strategies, for example:

- One file can have different access permissions than another. Typically, `/usr/lib/array/arrayd.conf` is world-readable and contains the available `array` commands, while `/usr/lib/array/arrayd.auth` is readable only by root and contains authentication codes.

- One node can have different configuration data than another. For example, certain commands might be defined only in certain nodes; or only the nodes used for interactive logins might know the names of all other nodes.

- You can use NFS-mounted configuration files. You could put a small configuration file on each machine to define the Array and authentication keys, but you could have a larger file defining `array` commands that is NFS-mounted from one node.

After you modify the configuration files, you can make `arrayd` reload them by killing the daemon and restarting it in each machine. The script `/etc/init.d/array` supports this operation:

To kill daemon, execute this command:

    /etc/init.d/array stop

To kill and restart the daemon in one operation; peform the following command:

    /etc/init.d/array restart

| Note: On Linux systems, the script path name is `/etc/rc.d/init.d/array`.

The Array Services daemon in any node knows only the information in the configuration files available in that node. This can be an advantage, in that you can limit the use of particular nodes; but it does require that you take pains to keep common information synchronized. (An automated way to do this is summarized under “Designing New Array Commands”.)

### About Substitution Syntax

The man page `arrayd.conf(4)` details the syntax rules for forming entries in the configuration files. An important feature of this syntax is the use of several kinds of text substitution, by which variable text is substituted into entries when they are executed.

Most of the supported substitutions are used in command entries. These substitutions are performed dynamically, each time the `array` command invokes a subcommand. At that time, substitutions insert values that are unique to the invocation of that subcommand. For example, the value `%USER` inserts the user ID of the user who is invoking the `array` command. Such a substitution has no meaning except during execution of a command.

Substitutions in other configuration entries are performed only once, at the time the configuration file is read by `arrayd`. Only environment variable substitution makes sense in these entries. The environment variable values that are substituted are the values inherited by `arrayd` from the script that invokes it, which is `/etc/init.d/array`.

### Testing Configuration Changes

The configuration files contain many sections and options (detailed in the section that follow this one). The Array Services command `ascheck` performs a basic sanity check of all configuration files in the array.

After making a change, you can test an individual configuration file for correct syntax by executing `arrayd` as a command with the `-c` and `-f` options. For example, suppose you have just added a new command definition to `/usr/lib/array/arrayd.local`. You can check its syntax with the following command:

    arrayd -c -f /usr/lib/array/arrayd.local

When testing new commands for correct operation, you need to see the warning and error messages produced by `arrayd` and processes that it may spawn. The `stderr` messages from a daemon are not normally visible. You can make them visible by the following procedure:

1. On one node, kill the daemon.

2. In one shell window on that node, start `arrayd` with the options `-n -v`. Instead of moving into the background, it remains attached to the shell terminal.

   | Note: Although `arrayd` becomes functional in this mode, it does not refer to `/etc/config/arrayd.options`, so you need to specify explicitly all command-line options, such as the names of nonstandard configuration files.

3. From another shell window on the same or other nodes, issue `ainfo` and `array` commands to test the new configuration data. Diagnostic output appears in the `arrayd` shell window.

4. Terminate `arrayd` and restart it as a daemon (without `-n`).

During steps 1, 2, and 4, the test node may fail to respond to `ainfo` and `array` commands, so users should be warned that the Array is in test mode.

## Configuring Arrays and Machines

Each ARRAY entry gives the name and composition of an Array system that users can access. At least one ARRAY must be defined at every node, the array in use.

| Note: ARRAY is a keyword.

### Specifying Arrayname and Machine Names

A simple example of an ARRAY definition is a follows:

    array simple
            machine congo
            machine niger
            machine nile

The arrayname `simple` is the value the user must specify in the `-a` option (see “Summary of Common Command Options”). One arrayname should be specified in a DESTINATION ARRAY local option as the default array (reported by `ainfo dflt`). Local options are listed under “Configuring Local Options”.

It is recommended that you have at least one array called `me` that just contains the localhost. The default `arrayd.conf` file has the `me` array defined as the default destination array.

The MACHINE subentries of ARRAY define the nodenames that the user can specify with the `-s` option. These names are also reported by the command `ainfo machines`.

### Specifying IP Addresses and Ports

The simple MACHINE subentries shown in the example are based on the assumption that the hostname is the same as the machine's name to Domain Name Services (DNS). If a machine's IP address cannot be obtained from the given hostname, you must provide a HOSTNAME subentry to specify either a completely qualified domain name or an IP address, as follows:

    array simple
            machine congo
                hostname congo.engr.hitech.com
                port 8820
            machine niger
                hostname niger.engr.hitech.com
            machine nile
                hostname "198.206.32.85"

The preceding example also shows how the PORT subentry can be used to specify that `arrayd` in a particular machine uses a different socket number than the default 5434.

### Specifying Additional Attributes

Under both ARRAY and MACHINE you can insert attributes, which are named string values. These attributes are not used by Array Services, but they are displayed by `ainfo` and can be returned to programs using the Array Services library (see “Array Services Library”). Some examples of attributes would be as follows:

    array simple
           array_attribute config_date="04/03/96"
           machine a_node
           machine_attribute aka="congo"
           hostname congo.engr.hitech.com

| Tip: You can write code that fetches any arrayname, machine name, or attribute string from any node in the array. See “Database Interrogation”.

## Configuring Authentication Codes

In Array Services 3.5 only one type of authentication is provided: a simple numeric key that can be required with any Array Services command. You can specify a single authentication code number for each node. The user must specify the code with any command entered at that node, or addressed to that node using the `-s` option (see “Summary of Common Command Options”).

The `arshell` command is like `rsh` in that it runs a command on another machine under the userid of the invoking user. Use of authentication codes makes Array Services somewhat more secure than `rsh`.

## Configuring Array Commands

The user can invoke arbitrary system commands on single nodes using the `arshell` command (see “Using arshell”). The user can also launch MPI and PVM programs that automatically distribute over multiple nodes. However, the only way to launch coordinated system programs on all nodes at once is to use the `array` command. This command does not accept any system command; it only permits execution of commands that the administrator has configured into the Array Services database.

You can define any set of commands that your users need. You have complete control over how any single Array node executes a command (the definition can be different in different nodes). A command can simply invoke a standard system command, or, since you can define a command as invoking a script, you can make a command arbitrarily complex.

### Operation of Array Commands

When a user invokes the `array` command, the subcommand and its arguments are processed by the destination node specified by `-s`. Unless the `-l` option was given, that daemon also distributes the subcommand and its arguments to all other array nodes that it knows about (the destination node might be configured with only a subset of nodes). At each node, `arrayd` searches the configuration database for a COMMAND entry with the same name as the array subcommand.

In the following example, the subcommand `uptime` is processed by `arrayd` in node `tokyo`:

    array -s tokyo uptime

When `arrayd` finds the subcommand valid, it distributes it to every node that is configured in the default array at node `tokyo`.

The COMMAND entry for `uptime` is distributed in this form (you can read it in the file `/usr/lib/array/arrayd.conf`).

    command uptime          # Display uptime/load of all nodes in array
            invoke /usr/lib/array/auptime %LOCAL

The INVOKE subentry tells `arrayd` how to execute this command. In this case, it executes a shell script `/usr/lib/array/auptime` , passing it one argument, the name of the local node. This command is executed at every node, with %LOCAL replaced by that node's name.

### Summary of Command Definition Syntax

Look at the basic set of commands distributed with Array Services 3.5 (`/usr/lib/array/arrayd.conf` ). Each COMMAND entry is defined using the subentries shown in Table 7-7. (These are described in great detail in the man page `arrayd.conf(4)`.)

Table 7-7. Subentries of a COMMAND Definition

Keyword | Meaning of Following Values
--- | ---
COMMAND | The name of the command as the user gives it to `array`.
INVOKE | A system command to be executed on every node. The argument values can be literals, or arguments given by the user, or other substitution values.
MERGE | A system command to be executed only on the distributing node, to gather the streams of output from all nodes and combine them into a single stream.
USER | The user ID under which the INVOKE and MERGE commands run. Usually given as USER %USER, so as to run as the user who invoked `array`.
GROUP | The group name under which the INVOKE and MERGE commands run. Usually given as GROUP %GROUP, so as to run in the group of the user who invoked `array` (see the `groups(1)` man page).
PROJECT | The project under which the INVOKE and MERGE commands run. Usually given as PROJECT %PROJECT, so as to run in the project of the user who invoked `array` (see the `projects(5)` man page).
OPTIONS | A variety of options to modify this command; see Table 7-9.

The system commands called by INVOKE and MERGE must be specified as full pathnames, because `arrayd` has no defined execution path. As with a shell script, these system commands are often composed from a few literal values and many substitution strings. The substitutions that are supported (which are documented in detail in the `arrayd.conf(4)` man page) are summarized in Table 7-8.

Table 7-8. Substitutions Used in a COMMAND Definition

Substitution | Replacement Value
--- | ---
%1..%9; %ARG(n); %ALLARGS; %OPTARG(n) | Argument tokens from the user's subcommand. %OPTARG does not produce an error message if the specified argument is omitted.
%USER, %GROUP, %PROJECT | The effective user ID, effective group ID, and project of the user who invoked `array`.
%REALUSER, %REALGROUP | The real user ID and real group ID of the user who invoked `array`.
%ASH | The ASH under which the INVOKE or MERGE command is to run.
%PID(ash) | List of PID values for a specified ASH. %PID(%ASH) is a common use.
%ARRAY | The array name, either default or as given in the `-a` option.
%LOCAL | The hostname of the executing node.
%ORIGIN | The full domain name of the node where the `array` command ran and the output is to be viewed.
%OUTFILE | List of names of temporary files, each containing the output from one node's INVOKE command (valid only in the MERGE subentry).

The OPTIONS subentry permits a number of important modifications of the command execution; these are summarized in Table 7-9.

Table 7-9. Options of the COMMAND Definition

Keyword | Effect on Command
--- | ---
LOCAL | Do not distribute to other nodes (effectively forces the `-l` option).
NEWSESSION | Execute the INVOKE command under a newly created ASH. %ASH in the INVOKE line is the new ASH. The MERGE command runs under the original ASH, and %ASH substitutes as the old ASH in that line.
SETRUID | Set both the real and effective user ID from the USER subentry (normally USER only sets the effective UID).
SETRGID | Set both the real and effective group ID from the GROUP subentry (normally GROUP sets only the effective GID).
QUIET | Discard the output of INVOKE, unless a MERGE subentry is given. If a MERGE subentry is given, pass INVOKE output to MERGE as usual and discard the MERGE output.
NOWAIT | Discard the output and return as soon as the processes are invoked; do not wait for completion (a MERGE subentry is ineffective).

### Configuring Local Options

The LOCAL entry specifies options to `arrayd` itself. The most important options are summarized in Table 7-10.

Table 7-10. Subentries of the LOCAL Entry

Subentry | Purpose
--- | ---
DIR | Pathname for the `arrayd` working directory, which is the initial, current working directory of INVOKE and MERGE commands. The default is `/usr/lib/array`.
DESTINATION ARRAY | Name of the default array, used when the user omits the `-a` option. When only one ARRAY entry is given, it is the default destination.
USER, GROUP, PROJECT | Default values for COMMAND execution when USER, GROUP, or PROJECT are omitted from the COMMAND definition.
HOSTNAME | Value returned in this node by %LOCAL. Default is the hostname.
PORT | Socket to be used by `arrayd`.

If you do not supply LOCAL USER, GROUP, and PROJECT values, the default values for USER and GROUP are “guest.”

The HOSTNAME entry is needed whenever the `hostname` command does not return a node name as specified in the ARRAY MACHINE entry. In order to supply a LOCAL HOSTNAME entry unique to each node, each node needs an individualized copy of at least one configuration file.

### Designing New Array Commands

A basic set of commands is distributed in the file `/usr/lib/array/arrayd.conf.template` . You should examine this file carefully before defining commands of your own. You can define new commands which then become available to the users of the Array system.

Typically, a new command will be defined with an INVOKE subentry that names a script written in sh, csh, or Perl syntax. You use the substitution values to set up arguments to the script. You use the USER, GROUP, PROJECT, and OPTIONS subentries to establish the execution conditions of the script. For one example of a command definition using a simple script, see “About the Distributed Example”.

Within the invoked script, you can write any amount of logic to verify and validate the arguments and to execute any sequence of commands. For an example of a script in Perl, see `/usr/lib/array/aps`, which is invoked by the `array ps` command.

| Note: Perl is a particularly interesting choice for `array` commands, since Perl has native support for socket I/O. In principle at least, you could build a distributed application in Perl in which multiple instances are launched by `array` and coordinate and exchange data using sockets. Performance would not rival the highly tuned MPI and PVM libraries, but development would be simpler.

The administrator has need for distributed applications as well, since the configuration files are distributed over the Array. Here is an example of a distributed command to reinitialize the Array Services database on all nodes at once. The script to be executed at each node, called `/usr/lib/array/arrayd-reinit` would read as follows:

    #!/bin/sh
    # Script to reinitialize arrayd with a new configuration file
    # Usage:  arrayd-reinit <hostname:new-config-file>
    sleep 10      # Let old arrayd finish distributing
    rcp $1 /usr/lib/array/
    /etc/init.d/array restart
    exit 0

The script uses `rcp` to copy a specified file (presumably a configuration file such as `arrayd.conf`) into `/usr/lib/array` (this will fail if %USER is not privileged). Then the script restarts `arrayd` (see `/etc/init.d/array`) to reread configuration files.

The command definition would be as follows:

    command reinit
       invoke /usr/lib/array/arrayd-reinit %ORIGIN:%1
       user   %USER
       group  %GROUP
       options nowait   # Exit before restart occurs!

The INVOKE subentry calls the restart script shown above. The NOWAIT option prevents the daemon's waiting for the script to finish, since the script will kill the daemon.

## Array Services Library

Array Services consists of a configuration database, a daemon (`arrayd`) that runs in each node to provide services, and several user-level commands. The facilities of Array Services are also available to developers through the Array Services library, a set of functions through which you can interrogate the configuration database and call on the services of `arrayd`.

The commands of Array Services are covered in “Using Array Services Commands”. The administration of Array Services is described in “About Array Configuration”, and the sections that follow it. These sections provide useful background information for understanding the Array Services library.

The programming interface to Array Services is declared in the header file `/usr/include/arraysvcs.h`. The object code is located in `/usr/lib/libarray.so`, included in a program by specifying `-larray` during compilation. The library is distributed in o32, n32, and 64-bit versions on IRIX and IA-64 versions on Linux (not all need to be installed).

The library functions can be grouped into these categories:

- Functions to connect to Array Services daemons in the local or other nodes, and to get and set `arrayd` options.

- Functions to interrogate the Array Services configuration database, listing arrays, nodes, and attributes of arrays and nodes.

- Functions to allocate Array Session Handles (ASHs), to query active ASHs, and to change the relationship between PIDs and ASHs.

- A function to execute a command for the `array` command (see “Operation of Array Commands”).

- A function to execute any arbitrary user command on an array node.

These functions are examined in following sections.

### Data Structures

The Array Services functions work with a number of data structures that are declared in `arraysvcs.h`. In general, each data structure is allocated by one particular function, which returns a pointer to the structure as the function's result. Your code uses the returned structure, possibly passing it as an argument to other functions.

When your code is finished with a structure, it is expected to call a specific function that frees that type of structure. If your code does not free each structure, a memory leak results.

The data structures and their contents are summarized in Table 7-11.

Table 7-11. Array Services Data Structures

Structure | Contents | Freed By Function
--- | --- | ---
asarray_t | Name and attributes of an Array. | `asfreearray()`
asarraylist_t | List of asarray_t structures. | `asfreearraylist()`
asashlist_t | List of ASH values. | `asfreeashlist()`
ascmdrslt_t | Describes output of executing an `array` command on one node, including temporary files and socket numbers. | Freed as part of a list
ascmdrsltlist_t | List of command results, one ascmdrslt_t per node where an `array` command was executed. | `asfreecmdrsltlist()`
asmachine_t | Configuration data about one node: machine name and attributes. | Freed as part of a list
asmachinelist_t | List of asmachine_t structures, one per machine in the queried array | `asfreemachinelist()`
aspidlist_t | List of PID values. | `asfreepidlist()`

### Error Message Conventions

The functions of the Array Services library have a complicated convention for error return codes. The man pages related to this convention are listed in Table 7-12.

Table 7-12. Error Message Functions

Function | Operation
--- | ---
`aserrorcode(3X)` | Discusses the error code conventions and some macro functions used to extract subfields from an error code.
`asmakeerror(3X)` | Constructs an error code value from its component parts.
`asstrerror(3X)` | Returns a descriptive string for a given error code value.
`asperror(3X)` | Prints a descriptive string, with a specified heading string, on stderr.

In general, each function sets a value in the global aserrorcode structure, which has type aserror_t (not necessarily an int). An error code is a structured value with these parts:

- aserrno is a general error number similar to those declared in `sys/errno.h`.

- aserrwhy documents the cause of the error.

- aserrwhat documents the component that detected the error.

- aserrextra may give additional information.

Macro functions to extract these subfields from the global aserrorcode structure are provided.

### Connecting to Array Services Daemons

The functions listed in Table 7-13 are used to open a connection between the node where your program runs and an instance of `arrayd` in the same or another node.

Table 7-13. Functions for Connections to Array Services Daemons

Function | Operation
--- | ---
`asopenserver(3X)` | Establishes a logical connection to `arrayd` in a specified node, returning a token that represents that connection for use in other functions.
`ascloseserver(3X)` | Closes an `arrayd` connection created by `asopenserver()`.
`asgetserveropt(3X)` | Returns the local options currently in use by an instance of `arrayd`.
`asdfltserveropt(3X)` | Returns the default options in effect at an instance of `arrayd`.
`assetserveropt(3X)` | Sets new options for an instance of `arrayd`.

The key function is `asopenserver()`. It takes a node name as a character string (as a user would give it in the `-s` option; see “Summary of Common Command Options”), and optionally a socket number to override the default `arrayd` socket number. This function opens a socket connection to the specified instance of `arrayd`. The returned token (type asserver_t) stands for that connection and is passed to other functions.

The functions for getting and setting server options can change the configured options shown in Table 7-14. To set these options is the programmatic equivalent of passing command line options in an Array Services command (see “About Array Configuration”, and “Using Array Services Commands”).

Table 7-14. Server Options That Functions Can Query or Change

Constant | Changeable? | Meaning
--- | --- | ---
`AS_SO_TIMEOUT` | yes | Timeout interval for any request to this server
`AS_SO_CTIMEOUT` | yes | Timeout interval for connecting to this server
`AS_SO_FORWARD` | yes | Whether or not Array Services requests should be forwarded through the local `arrayd` or sent directly (using the `-F` option)
`AS_SO_LOCALKEY` | yes | The local authentication key (the `-Kl` command option)
`AS_SO_REMOTEKEY` | yes | The remote authentication key (`-Kr` command option)
`AS_SO_PORTNUM` | no | In default options only, the default socket number
`AS_SO_HOSTNAME` | no | The hostname for this connection

### Database Interrogation

The functions summarized in Table 7-15 are used to interrogate the configuration database used by `arrayd` in a specified node (see “About Array Configuration”).

Table 7-15. Functions for Interrogating the Configuration

Function | Operation
--- | ---
`asgetdfltarray(3X)` | Returns the array name and all attribute strings for the default array known to a specified server in an asarray_t structure
`aslistarrays(3X)` | Returns the names of all arrays, with their attribute strings, from a specified server as an asarraylist_t structure
`aslistmachines(3X)` | Returns the names of all machines, with their attribute strings, from a specified server as an asmachinelist_t structure
`asgetattr(3X)` | Searches for a particular attribute name in a list of attribute strings and return its value

Using these functions you can extract any array name, node name, or attribute that is known to an `arrayd` instance you have opened.

### Managing Array Service Handles

The functions summarized in Table 7-16 are used to create and interrogate ASH values.

Table 7-16. Functions for Managing Array Service Handles

Function | Operation
--- | ---
`asallocash(3X)` | Allocates a new ASH value. The value is only created, it is not applied to any process.
`aspidsinash(3X)` | Returns a list of PID values associated with an ASH at a specified server, as an aspidlist_t structure.
`asashofpid(3X)` | Returns the ASH associated with a specified PID.
`setash(2)` | Changes the ASH of the calling process.

The `asallocash()` function is like the command `ainfo newash` (see “About Array Session Handles (ASH)”). Only a program with root privilege can use the `setash()` system function to change the ASH of the current process. Unprivileged processes can create new ASH values but cannot change their ASH.

The functions summarized in Table 7-17 are used to enumerate the active ASH values at a specified node. In each case, the list of ASH values is returned in an asashlist_t structure.

Table 7-17. Functions for ASH Interrogation

Function | Operation
--- | ---
`aslistashs(3X)` | Returns active ASH values from one node or all nodes of a specified Array via a specified server
`aslistashs_array(3X)` | Returns active ASH values from an Array by name
`aslistashs_server(3X)` | Returns active ASH values known to a specified server node
`aslistashs_local(3X)` | Returns active ASH values in the local node
`asashisglobal(3X)` | Tests to see if an ASH is global

### Executing an Array Command

The `ascommand()` function is the programmatic equivalent of the `array` command (see “Operation of Array Commands” and the `array(1)` man page). This command has many options and can be used to execute commands in three distinct modes.

The command to be executed must be prepared in an `ascmdreq_t` structure, which contains the following fields:

    typedef struct ascmdreq {
       char *array;            /* Name of target array */
       int flags;              /* Option flags */
       int numargs;            /* Number of arguments */
       char **args;            /* Cmd arguments (ala argv) */
       int ioflags;            /* I/O flags for interactive commands */
       char rsrvd[100];        /* reserved for expansion: init to 0's */
    } ascmdreq_t;

Your program must prepare this structure in order to execute a command. The option flags allow for the same controls as the command line options of `array`.

The result of the command is returned as an ascmdrsltlist_t structure, which is a vector of ascmdrslt_t structures, one for each node at which the command was executed. Each ascmdrslt_t contains the following fields:

    typedef struct ascmdrslt {
        char     *machine;  /* Name of responding machine */
        ash_t    ash;       /* ASH of running command */
        int      flags;     /* Result flags */
        aserror_t error;    /* Error code for this command */
        int      status;    /* Exit status */
        char     *outfile;  /* Name of output file */
        int      ioflags;   /* I/O connections (see ascmdreq_t) */
        int      stdinfd;   /* File descriptor for command's stdin */
        int      stdoutfd;  /* File descriptor for command's stdout */
        int      stderrfd;  /* File descriptor for command's stderr */
        int      signalfd;  /* File descriptor for sending signals */
    } ascmdrslt_t;

The fields `machine`, `ash`, `flags`, `error`, and `status` reflect the result of the command execution in that machine. The other fields depend on the mode of execution.

#### Normal Batch Execution

To execute a command in the normal way, waiting for it to complete and collecting its output, you do not set either `ASCMDREQ_NOWAIT` or `ASCMDREQ_INTERACTIVE` in the command option flags.

Control returns from `ascommand()` when the command is complete on all nodes. If the `ASCMDREQ_OUTPUT` flag was specified, and if the command definition does not specify a MERGE subentry (see “Summary of Command Definition Syntax”), the `outfile` result field contains the name of a temporary file containing one node's output stream.

When the command is implemented with a MERGE subentry, there is only one output file no matter how many nodes are invoked. In this case, the returned list contains only one `ascmdrslt_t` structure. It contains the `ASCMDRSLT_MERGED` and `ASCMDREQ_OUTPUT` flags, and the `outfile` result field contains the name of a temporary file containing the merged output.

#### Immediate Execution

When a command has no useful output and should execute concurrently with the calling program, you specify the `ASCMDREQ_NOWAIT` option. In this case, output cannot be collected because no program will be waiting to use it. Control returns as soon as the command has been distributed. The result structures do not reflect the command's result but only the result of trying to start it.

#### Interactive Execution

You can start a command in such a way that your program has direct interaction with the input and output streams of the command process in every node. When you do this, your program can supply input and inspect output in near real time.

To establish interactive execution, specify `ASCMDREQ_INTERACTIVE` in the command option flag. Also set one or more of the following flags in the `ioflags` field:

`ASCMDIO_STDIN` |  | Requests a socket attached to the command's stdin.
`ASCMDIO_STDOUT` |  | Requests a socket attached to the command's stdout.
`ASCMDIO_STDERR` |  | Requests a socket attached to the command's stderr.
`ASCMDIO_SIGNAL` |  | Requests a socket that can be used to deliver signals.

As with `ASCMDREQ_NOWAIT`, control returns as soon as the command has been distributed. Each result structure contains file descriptors for the requested sockets for the command process in that node.

Your program writes data into the stdinfd file descriptor of one node in order to send data to the stdin stream in that node. Your program reads data from the stdoutfd file descriptor to read one node's output stream.

You will typically use either the `select()` or the `poll()` system function to learn when one of the sockets is ready for use. You may choose to start one or more subprocesses using `fork()` to handle I/O to the sockets of each node (see the `select(2)` , `poll(2)`, and `sproc(2)` man pages). (You may also use `sproc()` to make subprocesses, but keep in mind that the `libarray` is not thread-safe, so it should only be used from one process in a share group.)

### Executing a User Command

The `asrcmd()` function allows a program to initiate any user command string on a specified node. This provides a powerful facility for remote execution that does not require root privilege, as the standard `rcmd()` function does (compare the `asrcmd(3)` and `rcmd(3)` man pages).

The `asrcmd()` function takes arguments specifying:

- The array node to use, as returned by `asopenserver()` (see “Connecting to Array Services Daemons”).

- The user name to use on the remote node.

- The command line to be executed.

The returned value (as with `rcmd()`) is a socket that represents the standard input and output streams of the executing command. Optionally, a separate socket for the standard error stream can be obtained.

Prev | Table of Contents | Next
Chapter 6. IRIX Memory Usage |  | Appendix A. Programming Guide for Resource Management