Chapter 7. Array Services

Array Services includes administrator commands, libraries, daemons, and kernel extensions that support the execution of programs across an array.

A central concept in Array Services is the array session handle (ASH), a number that is used to logically group related processes that may be distributed across multiple systems. The ASH creates a global process namespace across the Array, facilitating accounting and administration

Array Services also provides an array configuration database, listing the nodes comprising an array. Array inventory inquiry functions provide a centralized, canonical view of the configuration of each node. Other array utilities let the administrator query and manipulate distributed array applications.

The Array Services package comprises the following primary components:

array daemon 

Allocates ASH values and maintain information about node configuration and the relation of process IDs to ASHs. Array daemons reside on each node and work in cooperation.

array configuration database 

Describes the array configuration used by array daemons and user programs. One copy at each node.

ainfo command 

Lets the user or administrator query the Array configuration database and information about ASH values and processes.

array command 

Executes a specified command on one or more nodes. Commands are predefined by the administrator in the configuration database.

arshell command 

Starts a command remotely on a different node using the current ASH value.

aview command 

Displays a multiwindow, graphical display of each node's status.

libarray library 

Library of functions that allow user programs to call on the services of array daemons and the array configuration database.

The use of the ainfo, array, arshell, and aview commands is covered in “Using an Array”. The use of the libarray library is covered in“Array Services Library” .

Using an Array

An Array system is an aggregation of nodes, which are servers bound together with a high-speed network and Array Services 3.5 software. Array users have the advantage of greater performance and additional services. Array users access the system with familiar commands for job control, login and password management, and remote execution.

Array Services 3.5 augments conventional facilities with additional services for array users and for array administrators. The extensions include support for global session management, array configuration management, batch processing, message passing, system administration, and performance visualization.

This section introduces the extensions for Array use, with pointers to more detailed information. The main topics are as follows:

Using an Array System

The array system allows you to run distributed sessions on multiple nodes of an array. You can access the Array from either:

  • A workstation

  • An X terminal

  • An ASCII terminal

In each case, you log in to one node of the Array in the way you would log in to any remote UNIX host. From a workstation or an X terminal you can of course open more than one terminal window and log into more than one node.

Finding Basic Usage Information

In order to use an Array, you need the following items of information:

  • The name of the Array.

    You use this arrayname in Array Services commands.

  • The login name and password you will use on the Array.

    You use these when logging in to the Array to use it.

  • The hostnames of the array nodes.

    Typically these names follow a simple pattern, often arrayname1, arrayname2, and so on.

  • Any special resource-distribution or accounting rules that may apply to you or your group under a job scheduling system.

You can learn the hostnames of the array nodes if you know the array name, using the ainfo command as follows:

ainfo -a arrayname machines

Logging In to an Array

Each node in an Array has an associated hostname and IP network address. Typically, you use an Array by logging in to one node directly, or by logging in remotely from another host (such as the Array console or a networked workstation). For example, from a workstation on the same network, this command would log you in to the node named hydra6 as follows:

rlogin hydra6

For details of the rlogin command, see the rlogin(1) man page.

The system administrators of your array may choose to disallow direct node logins in order to schedule array resources. If your site is configured to disallow direct node logins, your administrators will be able to tell you how you are expected to submit work to the array-perhaps through remote execution software or batch queueing facilities.

Invoking a Program

Once you have access to an array, you can invoke programs of several classes:

  • Ordinary (sequential) applications

  • Parallel shared-memory applications within a node

  • Parallel message-passing applications within a node

  • Parallel message-passing applications distributed over multiple nodes (and possibly other servers on the same network running Array Services 3.5

If you are allowed to do so, you can invoke programs explicitly from a logged-in shell command line; or you may use remote execution or a batch queueing system.

Programs that are X Windows clients must be started from an X server, either an X Terminal or a workstation running X Windows.

Some application classes may require input in the form of command line options, environment variables, or support files upon execution. For example:

  • X client applications need the DISPLAY environment variable set to specify the X server (workstation or X-terminal) where their windows will display.

  • A multithreaded program may require environment variables to be set describing the number of threads.

    For example, C and Fortran programs that use parallel processing directives test the MP_SET_NUMTHREADS variable.

  • Message Passing Interface (MPI) and Parallel Virtual Machine (PVM) message-passing programs may require support files to describe how many tasks to invoke on specified nodes.

Some information sources on program invocation are listed in Table 7-1.

Table 7-1. Information Sources for Invoking a Program

Topic

Man Page

 

 

Remote login

rlogin (1)

 

 

Setting environment variables

environ (5), env(1)

 

 

Managing Local Processes

Each UNIX process has a process identifier (PID), a number that identifies that process within the node where it runs. It is important to realize that a PID is local to the node; so it is possible to have processes in different nodes using the same PID numbers.

Within a node, processes can be logically grouped in process groups. A process group is composed of a parent process together with all the processes that it creates. Each process group has a process group identifier (PGID). Like a PID, a PGID is defined locally to that node, and there is no guarantee of uniqueness across the Array.

Monitoring Local Processes and System Usage

You query the status of processes using the system command ps. To generate a full list of all processes on a local system, use a command such as the following:

ps -elfj

You can monitor the activity of processes using the command top (an ASCII display in a terminal window).

Scheduling and Killing Local Processes

You can start a process at a reduced priority, so that it interferes less with other processes, using the nice command. If you use the csh shell, specify /usr/bin/nice to avoid the built-in shell command nice. To start a whole shell at low priority, use a command like the one that follows:

/bin/nice /bin/sh

You can schedule commands to run at specific times using the at command. You can kill or stop processes using the kill command. To destroy the process with PID 13032, use a command such as the following:

kill -KILL 13032

Summary of Local Process Management Commands

Table 7-2, summarizes information about local process management.

Table 7-2. Information Sources: Local Process Management

Topic

Man Page

Process ID and process group

intro(2)

Listing and monitoring processes

ps(1), top(1)

Running programs at low priority

nice(1), batch(1)

Running programs at a scheduled time

at(1)

Terminating a process

kill(1)


Using Array Services Commands

When an application starts processes on more than one node, the PID and PGID are no longer adequate to manage the application. The commands of Array Services 3.5 give you the ability to view the entire array, and to control the processes of multinode programs.


Note: You can use Array Services commands from any workstation connected to an array system. You don't have to be logged in to an array node.


The following commands are common to Array Services operations as shown in Table 7-3.

Table 7-3. Common Array Services Commands

Topic

Man Page

Array Services Overview

array_services (5)

ainfo command

ainfo(1)

array command

Use array (1); configuration: arrayd.conf(4)

arshell command

arshell (1)

aview command

aview(1)

newsess command

newsess (1)


About Array Sessions

Array Services is composed of a daemon-a background process that is started at boot time in every node-and a set of commands such as ainfo(1). The commands call on the daemon process in each node to get the information they need.

One concept that is basic to Array Services is the array session, which is a term for all the processes of one application, wherever they may execute. Normally, your login shell, with the programs you start from it, constitutes an array session. A batch job is an array session; and you can create a new shell with a new array session identity.

Each session is identified by an array session handle (ASH), a number that identifies any process that is part of that session. You use the ASH to query and to control all the processes of a program, even when they are running in different nodes.

About Names of Arrays and Nodes

Each node is server, and as such has a hostname. The hostname of a node is returned by the hostname(1) command executed in that node as follows:

% hostname
tokyo

The command is simple and documented in the hostname(1) man page. The more complicated issues of hostname syntax, and of how hostnames are resolved to hardware addresses are covered in hostname(5).

An Array system as a whole has a name too. In most installations there is only a single Array, and you never need to specify which Array you mean. However, it is possible to have multiple Arrays available on a network, and you can direct Array Services commands to a specific Array.

About Authentication Keys

It is possible for the Array administrator to establish an authentication code, which is a 64-bit number, for all or some of the nodes in an array (see "Configuring Authentication Codes" on page 58). When this is done, each use of an Array Services command must specify the appropriate authentication key, as a command option, for the nodes it uses. Your system administrator will tell you if this is necessary.

Summary of Common Command Options

The following Array Services commands have a consistent set of command options: ainfo(1), array(1), arshell(1), aview(1), and newsess(1). Table 7-4 is a summary of these options. Not all options are valid with all commands; and each command has unique options besides those shown. The default values of some options are set by environment variables listed in the next topic.

Table 7-4. Array Services Command Option Summary

Option

Used In

Description

-a array

ainfo, array, aview

Specify a particular Array when more than one is accessible.

-D

ainfo, array, arshell, aview

Send commands to other nodes directly, rather than through array daemon.

-F

ainfo, array, arshell, aview

Forward commands to other nodes through the array daemon.

-Kl number

ainfo, array, aview

Authentication key (a 64-bit number) for the local node.

-Kr number

ainfo, array, aview

Authentication key (a 64-bit number) for the remote node.

-l (letter ell)

ainfo, array

Execute in context of the destination node, not necessarily the current node.

-l port

ainfo, array, arshell, aview

Nonstandard port number of array daemon.

-s hostname

ainfo, array, aview

Specify a destination node.


Specifying a Single Node

The -l and -s options work together. The -l (letter ell for “local”) option restricts the scope of a command to the node where the command is executed. By default, that is the node where the command is entered. When -l is not used, the scope of a query command is all nodes of the array. The -s (server, or node name) option directs the command to be executed on a specified node of the array. These options work together in query commands as follows:

  • To interrogate all nodes as seen by the local node, use neither option.

  • To interrogate only the local node, use only -l.

  • To interrogate all nodes as seen by a specified node, use only -s.

  • To interrogate only a particular node, use both -s and -l.

Common Environment Variables

The Array Services commands depend on environment variables to define default values for the less-common command options. These variables are summarized in Table 7-5.

Table 7-5. Array Services Environment Variables

Variable Name

Use

Default When Undefined

ARRAYD_FORWARD

When defined with a string starting with the letter y, all commands default to forwarding through the array daemon (option -F).

Commands default to direct communication (option -D).

ARRAYD_PORT

The port (socket) number monitored by the array daemon on the destination node.

The standard number of 5434, or the number given with option -p.

ARRAYD_LOCALKEY

Authentication key for the local node (option -Kl).

No authentication unless -Kl option is used.

ARRAYD_REMOTEKEY

Authentication key for the destination node (option -Kr).

No authentication unless -Kr option is used.

ARRAYD

The destination node, when not specified by the -s option.

The local node, or the node given with -s.


Interrogating the Array

Any user of an Array system can use Array Services commands to check the hardware components and the software workload of the Array. The commands needed are ainfo, array, and aview.

Learning Array Names

If your network includes more than one Array system, you can use ainfo arrays at one array node to list all the Array names that are configured, as in the following example.

homegrown% ainfo arrays
Arrays known to array services daemon
ARRAY DevArray
    IDENT 0x3381
ARRAY BigDevArray
    IDENT 0x7456
ARRAY test
    IDENT 0x655e

Array names are configured into the array database by the administrator. Different Arrays might know different sets of other Array names.

Learning Node Names

You can use ainfo machines to learn the names and some features of all nodes in the current Array, as in the following example.

homegrown 175% ainfo -b machines 
machine homegrown homegrown 5434 192.48.165.36 0
machine disarray disarray 5434 192.48.165.62 0
machine datarray datarray 5434 192.48.165.64 0
machine tokyo tokyo 5434 150.166.39.39 0

In this example, the -b option of ainfo is used to get a concise display.

Learning Node Features

You can use ainfo nodeinfo to request detailed information about one or all nodes in the array. To get information about the local node, use ainfo -l nodeinfo. However, to get information about only a particular other node, for example node tokyo, use -l and -s, as in the following example. (The example has been edited for brevity.)

homegrown 181% ainfo -s tokyo -l nodeinfo
Node information for server on machine "tokyo"
MACHINE tokyo
    VERSION  1.2
    8 PROCESSOR BOARDS
        BOARD: TYPE 15   SPEED 190
            CPU:   TYPE 9   REVISION 2.4
            FPU:   TYPE 9   REVISION 0.0
...
    16 IP INTERFACES  HOSTNAME tokyo   HOSTID 0xc01a5035
        DEVICE  et0    NETWORK    150.166.39.0    ADDRESS   150.166.39.39  UP
        DEVICE atm0    NETWORK 255.255.255.255    ADDRESS         0.0.0.0  UP
        DEVICE atm1    NETWORK 255.255.255.255    ADDRESS         0.0.0.0  UP
...
    0 GRAPHICS INTERFACES
    MEMORY
        512 MB MAIN MEMORY
        INTERLEAVE 4

If the -l option is omitted, the destination node will return information about every node that it knows.

Learning User Names and Workload

The system commands who(1), top(1), and uptime(1) are commonly used to get information about users and workload on one server. The array(1) command offers Array-wide equivalents to these commands.

Learning User Names

To get the names of all users logged in to the whole array, use array who. To learn the names of users logged in to a particular node, for example tokyo, use -l and -s, as in the following example. (The example has been edited for brevity and security.)

 homegrown 180% array -s tokyo -l who 
joecd    tokyo        frummage.eng.sgi -tcsh 
joecd    tokyo        frummage.eng.sgi -tcsh 
benf     tokyo        einstein.ued.sgi. /bin/tcsh 
yohn     tokyo        rayleigh.eng.sg vi +153 fs/procfs/prd
...

Learning Workload

Two variants of the array command return workload information. The array-wide equivalent of uptime is array uptime, as follows:

homegrown 181% array uptime 
   homegrown:  up 1 day,  7:40,  26 users,  load average: 7.21, 6.35, 4.72
    disarray:  up  2:53,  0 user,  load average: 0.00, 0.00, 0.00
    datarray:  up  5:34,  1 user,  load average: 0.00, 0.00, 0.00
       tokyo:  up 7 days,  9:11,  17 users,  load average: 0.15, 0.31, 0.29
homegrown 182% array -l -s tokyo uptime 
       tokyo:  up 7 days,  9:11,  17 users,  load average: 0.12, 0.30, 0.28

The command array top lists the processes that are currently using the most CPU time, with their ASH values, as in the following example.

homegrown 183% array top 
        ASH        Host           PID User       %CPU Command
----------------------------------------------------------------
0x1111ffff00000000 homegrown        5 root       1.20 vfs_sync
0x1111ffff000001e9 homegrown     1327 guest      1.19 atop
0x1111ffff000001e9 tokyo        19816 guest      0.73 atop
0x1111ffff000001e9 disarray      1106 guest      0.47 atop
0x1111ffff000001e9 datarray      1423 guest      0.42 atop
0x1111ffff00000000 homegrown       20 root       0.41 ShareII
0x1111ffff000000c0 homegrown    29683 kchang     0.37 ld
0x1111ffff0000001e homegrown     1324 root       0.17 arrayd
0x1111ffff00000000 homegrown      229 root       0.14 routed
0x1111ffff00000000 homegrown       19 root       0.09 pdflush
0x1111ffff000001e9 disarray      1105 guest      0.02 atopm

The -l and -s options can be used to select data about a single node, as usual.

Browsing With ArrayView

The ArrayView window shows the status of an array. You can start it with the command aview and it displays a window similar to the one shown in Figure 7-1. The top window shows one line per node. There is a window for each node, headed by the node name and its hardware configuration. Each window contains a snapshot of the busiest processes in that node.

Figure 7-1. Typical Display from ArrayView

Typical Display from ArrayView

Managing Distributed Processes

Using commands from Array Services 3.5, you can create and manage processes that are distributed across multiple nodes of the Array system.

About Array Session Handles (ASH)

In an Array system you can start a program with processes that are in more than one node. In order to name such collections of processes, Array Services 3.5 software assigns each process to an array session handle (ASH).

An ASH is a number that is unique across the entire array (unlike a PID or PGID). An ASH is the same for every process that is part of a single array session--no matter which node the process runs in. You display and use ASH values with Array Services commands. Each time you log in to an Array node, your shell is given an ASH, which is used by all the processes you start from that shell.

The command ainfo ash returns the ASH of the current process on the local node, which is simply the ASH of the ainfo command itself.

homegrown 178% ainfo ash 
Array session handle of process 10068: 0x1111ffff000002c1
homegrown 179% ainfo ash 
Array session handle of process 10069: 0x1111ffff000002c1

In the preceding example, each instance of the ainfo command was a new process: first PID 10068, then PID 10069. However, the ASH is the same in both cases. This illustrates a very important rule: every process inherits its parent's ASH. In this case, each instance of array was forked by the command shell, and the ASH value shown is that of the shell, inherited by the child process.

You can create a new global ASH with the command ainfo newash, as follows:

homegrown 175% ainfo newash 
Allocating new global ASH
0x11110000308b2f7c

This feature has little use at present. There is no existing command that can change its ASH, so you cannot assign the new ASH to another command. It is possible to write a program that takes an ASH from a command-line option and uses the Array Services function setash() to change to that ASH (however such a program must be privileged). No such program is distributed with Array Services 3.5 (but see “Managing Array Service Handles”).

Listing Processes and ASH Values

The command array ps returns a summary of all processes running on all nodes in an array. The display shows the ASH, the node, the PID, the associated username, the accumulated CPU time, and the command string.

To list all the processes on a particular node, use the -l and -s options. To list processes associated with a particular ASH, or a particular username, pipe the returned values through grep, as in the following example. (The display has been edited to save space.)

homegrown 182% array -l -s tokyo ps | fgrep wombat 
0x261cffff0000054c        tokyo 19007   wombat    0:00 -csh
0x261cffff0000054a        tokyo 17940   wombat    0:00 csh -c (setenv...
0x261cffff0000054c        tokyo 18941   wombat    0:00 csh -c (setenv...
0x261cffff0000054a        tokyo 17957   wombat    0:44 xem -geometry 84x42
0x261cffff0000054a        tokyo 17938   wombat    0:00 rshd
0x261cffff0000054a        tokyo 18022   wombat    0:00 /bin/csh -i
0x261cffff0000054a        tokyo 17980   wombat    0:03 /usr/gnu/lib/ema...
0x261cffff0000054c        tokyo 18928   wombat    0:00 rshd

Controlling Processes

The arshell command lets you start an arbitrary program on a single other node. The array command gives you the ability to suspend, resume, or kill all processes associated with a specified ASH.

Using arshell

The arshell command is an Array Services extension of the familiar rsh command; it executes a single system command on a specified Array node. The difference from rsh is that the remote shell executes under the same ASH as the invoking shell (this is not true of simple rsh). The following example demonstrates the difference.

homegrown 179% ainfo ash
Array session handle of process 8506: 0x1111ffff00000425
homegrown 180% rsh guest@tokyo ainfo ash
Array session handle of process 13113: 0x261cffff0000145e
homegrown 181% arshell guest@tokyo ainfo ash
Array session handle of process 13119: 0x1111ffff00000425

You can use arshell to start a collection of unrelated programs in multiple nodes under a single ASH; then you can use the commands described under “Managing Session Processes” to stop, resume, or kill them.

Both MPI and PVM use arshell to start up distributed processes.


Tip: The shell is a process under its own ASH. If you use the array command to stop or kill all processes started from a shell, you will stop or kill the shell also. In order to create a group of programs under a single ASH that can be killed safely, proceed as follows:
  1. Create a nested shell with a new ASH using newsess. Note the ASH value.

  2. Within the new shell, start one or more programs using arshell.

  3. Exit the nested shell.

Now you are back to the original shell. You know the ASH of all programs started from the nested shell. You can safely kill all jobs that have that ASH because the current shell is not affected.


About the Distributed Example

The programs launched with arshell are not coordinated (they could of course be written to communicate with each other, for example using sockets), and you must start each program individually.

The array command is designed to permit the simultaneous launch of programs on all nodes with a single command. However, array can only launch programs that have been configured into it, in the Array Services configuration file. (The creation and management of this file is discussed under “About Array Configuration”.)

In order to demonstrate process management in a simple way from the command line, the following command was inserted into the configuration file /usr/lib/array/arrayd.conf:

#
# Local commands
#
command spin                    # Do nothing on multiple machines
        invoke /usr/lib/array/spin
        user    %USER
        group   %GROUP
        options nowait

The invoked command, /usr/lib/array/spin, is a shell script that does nothing in a loop, as follows:

#!/bin/sh
# Go into a tight loop
#
interrupted() {
        echo "spin has been interrupted - goodbye"
        exit 0
}
trap interrupted 1 2
while [ ! -f /tmp/spin.stop ]; do
        sleep 5
done
echo "spin has been stopped - goodbye"
exit 1

With this preparation, the command array spin starts a process executing that script on every processor in the array. Alternatively, array -l -s nodename spin would start a process on one specific node.

Managing Session Processes

The following command sequence creates and then kills a spin process in every node. The first step creates a new session with its own ASH. This is so that later, array kill can be used without killing the interactive shell.

homegrown 175% ainfo ash
Array session handle of process 8912: 0x1111ffff0000032d
homegrown 176% newsess
homegrown 175% ainfo ash
Array session handle of process 8941: 0x11110000308b2fa6

In the new session with ASH 0x11110000308b2fa6, the command array spin starts the /usr/lib/array/spin script on every node. In this test array, there were only two nodes on this day, homegrown and tokyo.

homegrown 176% array spin

After exiting back to the original shell, the command array ps is used to search for all processes that have the ASH 0x11110000308b2fa6.

homegrown 177% exit
homegrown 178% homegrown 177% 
homegrown 177% ainfo ash
Array session handle of process 9257: 0x1111ffff0000032d
homegrown 179% array ps | fgrep 0x11110000308b2fa6
0x11110000308b2fa6  homegrown  9033   guest   0:00 /bin/sh /usr/lib/array/spin
0x11110000308b2fa6  homegrown  9618  guest  0:00 sleep 5
0x11110000308b2fa6        tokyo 26021  guest   0:00 /bin/sh /usr/lib/array/spin
0x11110000308b2fa6      tokyo 26072  guest  0:00 sleep 5
0x1111ffff0000032d  homegrown  9642  guest  0:00 fgrep 0x11110000308b2fa6

There are two processes related to the spin script on each node. The next command kills them all.

homegrown 180% array kill 0x11110000308b2fa6
homegrown 181% array ps | fgrep 0x11110000308b2fa6
0x1111ffff0000032d  homegrown 10030  guest  0:00 fgrep 0x11110000308b2fa6

The command array suspend 0x11110000308b2fa6 would suspend the processes instead (however, it is hard to demonstrate that a sleep command has been suspended).

About Job Container IDs

Array systems that are running IRIX 6.5.7f (or later) and Array Services version 3.4 (or later) have the capability to forward job IDs (JIDs) from the initiating host. All of the processes running in the ASH across one or more nodes in an array also belong to the same job. For a complete description of the job container and it usage, see the job_limits(5) man page or IRIX Admin: Resource Administration.

When processes are running on the initiating host, they belong to the same job as the initiating process and operate under the limits established for that job. On remote nodes, a new job is created using the same JID as the initiating process. Job limits for a job on remote nodes use the systune defaults and are set using the systune(1M) command on the initiating host.

About Array Configuration

The system administrator has to initialize the Array configuration database, a file that is used by the Array Services daemon in executing almost every ainfo and array command. For details about array configuration, see the man pages cited in Table 7-6.

Table 7-6. Information Sources: Array Configuration

Topic

Man Page

Array Services overview

array_services (5)  

Array Services user commands

ainfo(1) , array(1)  

Array Services daemon overview

arrayd (1m)  

Configuration file format

arrayd.conf (4) , /usr/lib/array/arrayd.conf.template  

Configuration file validator

ascheck (1)  

Array Services simple configurator

arrayconfig (1m)


About the Uses of the Configuration File

The configuration files are read by the Array Services daemon when it starts. Normally it is started in each node during the system startup. (You can also run the daemon from a command line in order to check the syntax of the configuration files.)

The configuration files contain data needed by ainfo and array:

  • The names of Array systems, including the current Array but also any other Arrays on which a user could run an Array Services command (reported by ainfo).

  • The names and types of the nodes in each named Array, especially the hostnames that would be used in an Array Services command (reported by ainfo).

  • The authentication keys, if any, that must be used with Array Services commands (required as -Kl and -Kr command options, see “Summary of Common Command Options”).

  • The commands that are valid with the array command.

About Configuration File Format and Contents

A configuration file is a readable text file. The file contains entries of the following four types, which are detailed in later topics.

Array definition 

Describes this array and other known arrays, including array names and the node names and types.

Command definition 

Specifies the usage and operation of a command that can be invoked through the array command.

Authentication 

Specifies authentication numbers that must be used to access the Array.

Local option 

Options that modify the operation of the other entries or arrayd.

Blank lines, white space, and comment lines beginning with “#” can be used freely for readability. Entries can be in any order in any of the files read by arrayd.

Besides punctuation, entries are formed with a keyword-based syntax. Keyword recognition is not case-sensitive; however keywords are shown in uppercase in this text and in the man page. The entries are primarily formed from keywords, numbers, and quoted strings, as detailed in the man page arrayd.conf(4) .

Loading Configuration Data

The Array Services daemon, arrayd, can take one or more filenames as arguments. It reads them all, and treats them like logical continuations (in effect, it concatenates them). If no filenames are specified, it reads /usr/lib/array/arrayd.conf and /usr/lib/array/arrayd.auth. A different set of files, and any other arrayd command-line options, can be written into the file /etc/config/arrayd.options, which is read by the startup script that launches arrayd at boot time.

Since configuration data can be stored in two or more files, you can combine different strategies, for example:

  • One file can have different access permissions than another. Typically, /usr/lib/array/arrayd.conf is world-readable and contains the available array commands, while /usr/lib/array/arrayd.auth is readable only by root and contains authentication codes.

  • One node can have different configuration data than another. For example, certain commands might be defined only in certain nodes; or only the nodes used for interactive logins might know the names of all other nodes.

  • You can use NFS-mounted configuration files. You could put a small configuration file on each machine to define the Array and authentication keys, but you could have a larger file defining array commands that is NFS-mounted from one node.

After you modify the configuration files, you can make arrayd reload them by killing the daemon and restarting it in each machine. The script /etc/init.d/array supports this operation:

To kill daemon, execute this command:

/etc/init.d/array stop

To kill and restart the daemon in one operation; peform the following command:

/etc/init.d/array restart


Note: On Linux systems, the script path name is /etc/rc.d/init.d/array.


The Array Services daemon in any node knows only the information in the configuration files available in that node. This can be an advantage, in that you can limit the use of particular nodes; but it does require that you take pains to keep common information synchronized. (An automated way to do this is summarized under “Designing New Array Commands”.)

About Substitution Syntax

The man page arrayd.conf(4) details the syntax rules for forming entries in the configuration files. An important feature of this syntax is the use of several kinds of text substitution, by which variable text is substituted into entries when they are executed.

Most of the supported substitutions are used in command entries. These substitutions are performed dynamically, each time the array command invokes a subcommand. At that time, substitutions insert values that are unique to the invocation of that subcommand. For example, the value %USER inserts the user ID of the user who is invoking the array command. Such a substitution has no meaning except during execution of a command.

Substitutions in other configuration entries are performed only once, at the time the configuration file is read by arrayd. Only environment variable substitution makes sense in these entries. The environment variable values that are substituted are the values inherited by arrayd from the script that invokes it, which is /etc/init.d/array.

Testing Configuration Changes

The configuration files contain many sections and options (detailed in the section that follow this one). The Array Services command ascheck performs a basic sanity check of all configuration files in the array.

After making a change, you can test an individual configuration file for correct syntax by executing arrayd as a command with the -c and -f options. For example, suppose you have just added a new command definition to /usr/lib/array/arrayd.local. You can check its syntax with the following command:

arrayd -c -f /usr/lib/array/arrayd.local

When testing new commands for correct operation, you need to see the warning and error messages produced by arrayd and processes that it may spawn. The stderr messages from a daemon are not normally visible. You can make them visible by the following procedure:

  1. On one node, kill the daemon.

  2. In one shell window on that node, start arrayd with the options -n -v. Instead of moving into the background, it remains attached to the shell terminal.


    Note: Although arrayd becomes functional in this mode, it does not refer to /etc/config/arrayd.options, so you need to specify explicitly all command-line options, such as the names of nonstandard configuration files.


  3. From another shell window on the same or other nodes, issue ainfo and array commands to test the new configuration data. Diagnostic output appears in the arrayd shell window.

  4. Terminate arrayd and restart it as a daemon (without -n).

During steps 1, 2, and 4, the test node may fail to respond to ainfo and array commands, so users should be warned that the Array is in test mode.

Configuring Arrays and Machines

Each ARRAY entry gives the name and composition of an Array system that users can access. At least one ARRAY must be defined at every node, the array in use.


Note: ARRAY is a keyword.


Specifying Arrayname and Machine Names

A simple example of an ARRAY definition is a follows:

array simple
        machine congo
        machine niger
        machine nile

The arrayname simple is the value the user must specify in the -a option (see “Summary of Common Command Options”). One arrayname should be specified in a DESTINATION ARRAY local option as the default array (reported by ainfo dflt). Local options are listed under “Configuring Local Options”.

It is recommended that you have at least one array called me that just contains the localhost. The default arrayd.conf file has the me array defined as the default destination array.

The MACHINE subentries of ARRAY define the nodenames that the user can specify with the -s option. These names are also reported by the command ainfo machines.

Specifying IP Addresses and Ports

The simple MACHINE subentries shown in the example are based on the assumption that the hostname is the same as the machine's name to Domain Name Services (DNS). If a machine's IP address cannot be obtained from the given hostname, you must provide a HOSTNAME subentry to specify either a completely qualified domain name or an IP address, as follows:

array simple
        machine congo
            hostname congo.engr.hitech.com
            port 8820
        machine niger
            hostname niger.engr.hitech.com
        machine nile
            hostname "198.206.32.85"

The preceding example also shows how the PORT subentry can be used to specify that arrayd in a particular machine uses a different socket number than the default 5434.

Specifying Additional Attributes

Under both ARRAY and MACHINE you can insert attributes, which are named string values. These attributes are not used by Array Services, but they are displayed by ainfo and can be returned to programs using the Array Services library (see “Array Services Library”). Some examples of attributes would be as follows:

array simple
       array_attribute config_date="04/03/96"
       machine a_node
       machine_attribute aka="congo"
       hostname congo.engr.hitech.com


Tip: You can write code that fetches any arrayname, machine name, or attribute string from any node in the array. See “Database Interrogation”.


Configuring Authentication Codes

In Array Services 3.5 only one type of authentication is provided: a simple numeric key that can be required with any Array Services command. You can specify a single authentication code number for each node. The user must specify the code with any command entered at that node, or addressed to that node using the -s option (see “Summary of Common Command Options”).

The arshell command is like rsh in that it runs a command on another machine under the userid of the invoking user. Use of authentication codes makes Array Services somewhat more secure than rsh.

Configuring Array Commands

The user can invoke arbitrary system commands on single nodes using the arshell command (see “Using arshell”). The user can also launch MPI and PVM programs that automatically distribute over multiple nodes. However, the only way to launch coordinated system programs on all nodes at once is to use the array command. This command does not accept any system command; it only permits execution of commands that the administrator has configured into the Array Services database.

You can define any set of commands that your users need. You have complete control over how any single Array node executes a command (the definition can be different in different nodes). A command can simply invoke a standard system command, or, since you can define a command as invoking a script, you can make a command arbitrarily complex.

Operation of Array Commands

When a user invokes the array command, the subcommand and its arguments are processed by the destination node specified by -s. Unless the -l option was given, that daemon also distributes the subcommand and its arguments to all other array nodes that it knows about (the destination node might be configured with only a subset of nodes). At each node, arrayd searches the configuration database for a COMMAND entry with the same name as the array subcommand.

In the following example, the subcommand uptime is processed by arrayd in node tokyo:

array -s tokyo uptime

When arrayd finds the subcommand valid, it distributes it to every node that is configured in the default array at node tokyo.

The COMMAND entry for uptime is distributed in this form (you can read it in the file /usr/lib/array/arrayd.conf ).

command uptime          # Display uptime/load of all nodes in array
        invoke /usr/lib/array/auptime %LOCAL

The INVOKE subentry tells arrayd how to execute this command. In this case, it executes a shell script /usr/lib/array/auptime , passing it one argument, the name of the local node. This command is executed at every node, with %LOCAL replaced by that node's name.

Summary of Command Definition Syntax

Look at the basic set of commands distributed with Array Services 3.5 (/usr/lib/array/arrayd.conf ). Each COMMAND entry is defined using the subentries shown in Table 7-7. (These are described in great detail in the man page arrayd.conf(4).)

Table 7-7. Subentries of a COMMAND Definition

Keyword

Meaning of Following Values

COMMAND

The name of the command as the user gives it to array.

INVOKE

A system command to be executed on every node. The argument values can be literals, or arguments given by the user, or other substitution values.

MERGE

A system command to be executed only on the distributing node, to gather the streams of output from all nodes and combine them into a single stream.

USER

The user ID under which the INVOKE and MERGE commands run. Usually given as USER %USER, so as to run as the user who invoked array.

GROUP

The group name under which the INVOKE and MERGE commands run. Usually given as GROUP %GROUP, so as to run in the group of the user who invoked array (see the groups(1) man page).

PROJECT

The project under which the INVOKE and MERGE commands run. Usually given as PROJECT %PROJECT, so as to run in the project of the user who invoked array (see the projects(5) man page).

OPTIONS

A variety of options to modify this command; see Table 7-9.


The system commands called by INVOKE and MERGE must be specified as full pathnames, because arrayd has no defined execution path. As with a shell script, these system commands are often composed from a few literal values and many substitution strings. The substitutions that are supported (which are documented in detail in the arrayd.conf(4)  man page) are summarized in Table 7-8.

Table 7-8. Substitutions Used in a COMMAND Definition

Substitution

Replacement Value

%1..%9; %ARG(n); %ALLARGS; %OPTARG(n)

Argument tokens from the user's subcommand. %OPTARG does not produce an error message if the specified argument is omitted.

%USER, %GROUP, %PROJECT

The effective user ID, effective group ID, and project of the user who invoked array.

%REALUSER, %REALGROUP

The real user ID and real group ID of the user who invoked array.

%ASH

The ASH under which the INVOKE or MERGE command is to run.

%PID(ash)

List of PID values for a specified ASH. %PID(%ASH) is a common use.

%ARRAY

The array name, either default or as given in the -a option.

%LOCAL

The hostname of the executing node.

%ORIGIN

The full domain name of the node where the array command ran and the output is to be viewed.

%OUTFILE

List of names of temporary files, each containing the output from one node's INVOKE command (valid only in the MERGE subentry).

The OPTIONS subentry permits a number of important modifications of the command execution; these are summarized in Table 7-9.

Table 7-9. Options of the COMMAND Definition

Keyword

Effect on Command

LOCAL

Do not distribute to other nodes (effectively forces the -l option).

NEWSESSION

Execute the INVOKE command under a newly created ASH. %ASH in the INVOKE line is the new ASH. The MERGE command runs under the original ASH, and %ASH substitutes as the old ASH in that line.

SETRUID

Set both the real and effective user ID from the USER subentry (normally USER only sets the effective UID).

SETRGID

Set both the real and effective group ID from the GROUP subentry (normally GROUP sets only the effective GID).

QUIET

Discard the output of INVOKE, unless a MERGE subentry is given. If a MERGE subentry is given, pass INVOKE output to MERGE as usual and discard the MERGE output.

NOWAIT

Discard the output and return as soon as the processes are invoked; do not wait for completion (a MERGE subentry is ineffective).


Configuring Local Options

The LOCAL entry specifies options to arrayd itself. The most important options are summarized in Table 7-10.

Table 7-10. Subentries of the LOCAL Entry

Subentry

Purpose

DIR

Pathname for the arrayd working directory, which is the initial, current working directory of INVOKE and MERGE commands. The default is /usr/lib/array .

DESTINATION ARRAY

Name of the default array, used when the user omits the -a option. When only one ARRAY entry is given, it is the default destination.

USER, GROUP, PROJECT

Default values for COMMAND execution when USER, GROUP, or PROJECT are omitted from the COMMAND definition.

HOSTNAME

Value returned in this node by %LOCAL. Default is the hostname.

PORT

Socket to be used by arrayd.


If you do not supply LOCAL USER, GROUP, and PROJECT values, the default values for USER and GROUP are “guest.”

The HOSTNAME entry is needed whenever the hostname command does not return a node name as specified in the ARRAY MACHINE entry. In order to supply a LOCAL HOSTNAME entry unique to each node, each node needs an individualized copy of at least one configuration file.

Designing New Array Commands

A basic set of commands is distributed in the file /usr/lib/array/arrayd.conf.template . You should examine this file carefully before defining commands of your own. You can define new commands which then become available to the users of the Array system.

Typically, a new command will be defined with an INVOKE subentry that names a script written in sh, csh, or Perl syntax. You use the substitution values to set up arguments to the script. You use the USER, GROUP, PROJECT, and OPTIONS subentries to establish the execution conditions of the script. For one example of a command definition using a simple script, see “About the Distributed Example”.

Within the invoked script, you can write any amount of logic to verify and validate the arguments and to execute any sequence of commands. For an example of a script in Perl, see /usr/lib/array/aps, which is invoked by the array ps command.


Note: Perl is a particularly interesting choice for array commands, since Perl has native support for socket I/O. In principle at least, you could build a distributed application in Perl in which multiple instances are launched by array and coordinate and exchange data using sockets. Performance would not rival the highly tuned MPI and PVM libraries, but development would be simpler.

The administrator has need for distributed applications as well, since the configuration files are distributed over the Array. Here is an example of a distributed command to reinitialize the Array Services database on all nodes at once. The script to be executed at each node, called /usr/lib/array/arrayd-reinit would read as follows:

#!/bin/sh
# Script to reinitialize arrayd with a new configuration file
# Usage:  arrayd-reinit <hostname:new-config-file>
sleep 10      # Let old arrayd finish distributing
rcp $1 /usr/lib/array/
/etc/init.d/array restart
exit 0

The script uses rcp to copy a specified file (presumably a configuration file such as arrayd.conf) into /usr/lib/array (this will fail if %USER is not privileged). Then the script restarts arrayd (see /etc/init.d/array) to reread configuration files.

The command definition would be as follows:

command reinit
   invoke /usr/lib/array/arrayd-reinit %ORIGIN:%1
   user   %USER
   group  %GROUP
   options nowait   # Exit before restart occurs!

The INVOKE subentry calls the restart script shown above. The NOWAIT option prevents the daemon's waiting for the script to finish, since the script will kill the daemon.

Array Services Library

Array Services consists of a configuration database, a daemon ( arrayd) that runs in each node to provide services, and several user-level commands. The facilities of Array Services are also available to developers through the Array Services library, a set of functions through which you can interrogate the configuration database and call on the services of arrayd.

The commands of Array Services are covered in “Using Array Services Commands”. The administration of Array Services is described in “About Array Configuration”, and the sections that follow it. These sections provide useful background information for understanding the Array Services library.

The programming interface to Array Services is declared in the header file /usr/include/arraysvcs.h. The object code is located in /usr/lib/libarray.so, included in a program by specifying -larray during compilation. The library is distributed in o32, n32, and 64-bit versions on IRIX and IA-64 versions on Linux (not all need to be installed).

The library functions can be grouped into these categories:

  • Functions to connect to Array Services daemons in the local or other nodes, and to get and set arrayd options.

  • Functions to interrogate the Array Services configuration database, listing arrays, nodes, and attributes of arrays and nodes.

  • Functions to allocate Array Session Handles (ASHs), to query active ASHs, and to change the relationship between PIDs and ASHs.

  • A function to execute a command for the array command (see “Operation of Array Commands”).

  • A function to execute any arbitrary user command on an array node.

These functions are examined in following sections.

Data Structures

The Array Services functions work with a number of data structures that are declared in arraysvcs.h. In general, each data structure is allocated by one particular function, which returns a pointer to the structure as the function's result. Your code uses the returned structure, possibly passing it as an argument to other functions.

When your code is finished with a structure, it is expected to call a specific function that frees that type of structure. If your code does not free each structure, a memory leak results.

The data structures and their contents are summarized in Table 7-11.

Table 7-11. Array Services Data Structures

Structure

Contents

Freed By Function

asarray_t 

Name and attributes of an Array.

asfreearray() 

asarraylist_t 

List of asarray_t structures.

asfreearraylist() 

asashlist_t 

List of ASH values.

asfreeashlist() 

ascmdrslt_t 

Describes output of executing an array command on one node, including temporary files and socket numbers.

Freed as part of a list

ascmdrsltlist_t 

List of command results, one ascmdrslt_t per node where an array command was executed.

asfreecmdrsltlist() 

asmachine_t 

Configuration data about one node: machine name and attributes.

Freed as part of a list

asmachinelist_t 

List of asmachine_t structures, one per machine in the queried array

asfreemachinelist()

aspidlist_t 

List of PID values.

asfreepidlist() 


Error Message Conventions

The functions of the Array Services library have a complicated convention for error return codes. The man pages related to this convention are listed in Table 7-12.

Table 7-12. Error Message Functions

Function

Operation

aserrorcode (3X)  

Discusses the error code conventions and some macro functions used to extract subfields from an error code.

asmakeerror (3X)  

Constructs an error code value from its component parts.

asstrerror (3X)  

Returns a descriptive string for a given error code value.

asperror (3X)  

Prints a descriptive string, with a specified heading string, on stderr.

In general, each function sets a value in the global aserrorcode structure, which has type aserror_t (not necessarily an int). An error code is a structured value with these parts:

  • aserrno is a general error number similar to those declared in sys/errno.h.

  • aserrwhy documents the cause of the error.

  • aserrwhat documents the component that detected the error.

  • aserrextra may give additional information.

Macro functions to extract these subfields from the global aserrorcode structure are provided.

Connecting to Array Services Daemons

The functions listed in Table 7-13 are used to open a connection between the node where your program runs and an instance of arrayd in the same or another node.

Table 7-13. Functions for Connections to Array Services Daemons

Function

Operation

asopenserver (3X)  

Establishes a logical connection to arrayd in a specified node, returning a token that represents that connection for use in other functions.

ascloseserver (3X)  

Closes an arrayd connection created by asopenserver().

asgetserveropt (3X)  

Returns the local options currently in use by an instance of arrayd.

asdfltserveropt (3X)  

Returns the default options in effect at an instance of arrayd.

assetserveropt (3X)  

Sets new options for an instance of arrayd.

The key function is asopenserver(). It takes a node name as a character string (as a user would give it in the -s option; see “Summary of Common Command Options”), and optionally a socket number to override the default arrayd socket number. This function opens a socket connection to the specified instance of arrayd. The returned token (type asserver_t ) stands for that connection and is passed to other functions.

The functions for getting and setting server options can change the configured options shown in Table 7-14. To set these options is the programmatic equivalent of passing command line options in an Array Services command (see “About Array Configuration”, and “Using Array Services Commands”).

Table 7-14. Server Options That Functions Can Query or Change

Constant

Changeable?

Meaning

AS_SO_TIMEOUT

yes

Timeout interval for any request to this server

AS_SO_CTIMEOUT

yes

Timeout interval for connecting to this server

AS_SO_FORWARD

yes

Whether or not Array Services requests should be forwarded through the local arrayd or sent directly (using the -F option)

AS_SO_LOCALKEY

yes

The local authentication key (the -Kl command option)

AS_SO_REMOTEKEY

yes

The remote authentication key ( -Kr command option)

AS_SO_PORTNUM

no

In default options only, the default socket number

AS_SO_HOSTNAME

no

The hostname for this connection


Database Interrogation

The functions summarized in Table 7-15 are used to interrogate the configuration database used by arrayd in a specified node (see “About Array Configuration”).

Table 7-15. Functions for Interrogating the Configuration

Function

Operation

asgetdfltarray (3X)

Returns the array name and all attribute strings for the default array known to a specified server in an asarray_t structure

aslistarrays (3X)

Returns the names of all arrays, with their attribute strings, from a specified server as an asarraylist_t structure

aslistmachines (3X)

Returns the names of all machines, with their attribute strings, from a specified server as an asmachinelist_t structure

asgetattr (3X)

Searches for a particular attribute name in a list of attribute strings and return its value

Using these functions you can extract any array name, node name, or attribute that is known to an arrayd instance you have opened.

Managing Array Service Handles

The functions summarized in Table 7-16 are used to create and interrogate ASH values.

Table 7-16. Functions for Managing Array Service Handles

Function

Operation

asallocash (3X)

Allocates a new ASH value. The value is only created, it is not applied to any process.

aspidsinash (3X)

Returns a list of PID values associated with an ASH at a specified server, as an aspidlist_t structure.

asashofpid (3X)

Returns the ASH associated with a specified PID.

setash (2)

Changes the ASH of the calling process.

The asallocash() function is like the command ainfo newash (see “About Array Session Handles (ASH)”). Only a program with root privilege can use the setash() system function to change the ASH of the current process. Unprivileged processes can create new ASH values but cannot change their ASH.

The functions summarized in Table 7-17 are used to enumerate the active ASH values at a specified node. In each case, the list of ASH values is returned in an asashlist_t structure.

Table 7-17. Functions for ASH Interrogation

Function

Operation

aslistashs (3X)

Returns active ASH values from one node or all nodes of a specified Array via a specified server

aslistashs_array (3X)

Returns active ASH values from an Array by name

aslistashs_server (3X)

Returns active ASH values known to a specified server node

aslistashs_local (3X)

Returns active ASH values in the local node

asashisglobal (3X)

Tests to see if an ASH is global


Executing an Array Command

The ascommand() function is the programmatic equivalent of the array command (see “Operation of Array Commands” and the array(1) man page). This command has many options and can be used to execute commands in three distinct modes.

The command to be executed must be prepared in an ascmdreq_t structure, which contains the following fields:

typedef struct ascmdreq {
   char *array;            /* Name of target array */
   int flags;              /* Option flags */
   int numargs;            /* Number of arguments */
   char **args;            /* Cmd arguments (ala argv) */
   int ioflags;            /* I/O flags for interactive commands */
   char rsrvd[100];        /* reserved for expansion: init to 0's */
} ascmdreq_t;

Your program must prepare this structure in order to execute a command. The option flags allow for the same controls as the command line options of array.

The result of the command is returned as an ascmdrsltlist_t structure, which is a vector of ascmdrslt_t structures, one for each node at which the command was executed. Each ascmdrslt_t contains the following fields:

typedef struct ascmdrslt {
    char     *machine;  /* Name of responding machine */
    ash_t    ash;       /* ASH of running command */
    int      flags;     /* Result flags */
    aserror_t error;    /* Error code for this command */
    int      status;    /* Exit status */
    char     *outfile;  /* Name of output file */
    int      ioflags;   /* I/O connections (see ascmdreq_t) */
    int      stdinfd;   /* File descriptor for command's stdin */
    int      stdoutfd;  /* File descriptor for command's stdout */
    int      stderrfd;  /* File descriptor for command's stderr */
    int      signalfd;  /* File descriptor for sending signals */
} ascmdrslt_t;

The fields machine, ash, flags, error, and status reflect the result of the command execution in that machine. The other fields depend on the mode of execution.

Normal Batch Execution

To execute a command in the normal way, waiting for it to complete and collecting its output, you do not set either ASCMDREQ_NOWAIT or ASCMDREQ_INTERACTIVE in the command option flags.

Control returns from ascommand() when the command is complete on all nodes. If the ASCMDREQ_OUTPUT flag was specified, and if the command definition does not specify a MERGE subentry (see “Summary of Command Definition Syntax”), the outfile result field contains the name of a temporary file containing one node's output stream.

When the command is implemented with a MERGE subentry, there is only one output file no matter how many nodes are invoked. In this case, the returned list contains only one ascmdrslt_t structure. It contains the ASCMDRSLT_MERGED and ASCMDREQ_OUTPUT flags, and the outfile result field contains the name of a temporary file containing the merged output.

Immediate Execution

When a command has no useful output and should execute concurrently with the calling program, you specify the ASCMDREQ_NOWAIT option. In this case, output cannot be collected because no program will be waiting to use it. Control returns as soon as the command has been distributed. The result structures do not reflect the command's result but only the result of trying to start it.

Interactive Execution

You can start a command in such a way that your program has direct interaction with the input and output streams of the command process in every node. When you do this, your program can supply input and inspect output in near real time.

To establish interactive execution, specify ASCMDREQ_INTERACTIVE in the command option flag. Also set one or more of the following flags in the ioflags field:

ASCMDIO_STDIN 

Requests a socket attached to the command's stdin.

ASCMDIO_STDOUT 

Requests a socket attached to the command's stdout.

ASCMDIO_STDERR 

Requests a socket attached to the command's stderr.

ASCMDIO_SIGNAL 

Requests a socket that can be used to deliver signals.

As with ASCMDREQ_NOWAIT, control returns as soon as the command has been distributed. Each result structure contains file descriptors for the requested sockets for the command process in that node.

Your program writes data into the stdinfd file descriptor of one node in order to send data to the stdin stream in that node. Your program reads data from the stdoutfd file descriptor to read one node's output stream.

You will typically use either the select() or the poll() system function to learn when one of the sockets is ready for use. You may choose to start one or more subprocesses using fork() to handle I/O to the sockets of each node (see the select(2) , poll(2), and sproc(2) man pages). (You may also use sproc() to make subprocesses, but keep in mind that the libarray is not thread-safe, so it should only be used from one process in a share group.)

Executing a User Command

The asrcmd() function allows a program to initiate any user command string on a specified node. This provides a powerful facility for remote execution that does not require root privilege, as the standard rcmd() function does (compare the asrcmd(3) and rcmd(3) man pages).

The asrcmd() function takes arguments specifying:

The returned value (as with rcmd()) is a socket that represents the standard input and output streams of the executing command. Optionally, a separate socket for the standard error stream can be obtained.