In a cluster deployment, every node must be GRIO-enabled. As of CXFS 3.4, every supported platform is GRIO-enabled. For more information, see the CXFS Administration Guide for SGI InfiniteStorage and the CXFS MultiOS Client-Only Guide for SGI InfiniteStorage.

grio2-cmds-version.ia64.rpm

grio2-server-version.ia64.rpm

You must observe the following constraints when setting up GRIO filesystems in a cluster:

To set up a filesystem on a RAID device such that you achieve correct filesystem device alignment and maximize I/O performance, remember to do the following:

For more information, see the grio2(5) man page.

You can adjust the qualified bandwidth to reflect the specific trade-off between delivered QoS and utilization of the storage infrastructure for your situation.

The following affect the qualified bandwidth you will choose:

Typically, the first concern is that the required bandwidth can be delivered by the storage system. The second concern is the service time observed for individual I/Os.

Determining qualified bandwidth is an iterative process. There are several strategies you can use to determine and fine-tune the qualified bandwidth for a filesystem. For example:

Although the hardware configuration provides a basis for calculating an estimate, remember that the qualified bandwidth is also affected by the particular work-flow issues and the QoS requirements of individual applications. For example, an application that has large tunable buffers (such as a flipbook application that does aggressive RAM caching) can tolerate a greater variation in I/O service time than can a media broadcast system that must cue and play a sequence of very short clips. In the first example, the qualified bandwidth would be configured as a larger proportion of the sustained maximum. In the second example, the qualified bandwidth might be reduced to minimize the system utilization levels and improve I/O service times.

A high qualified bandwidth will generally achieve the greatest overall throughput but with the consequence that individual applications may intermittently experience longer service times for some I/Os. This variation in individual service times is referred to as jitter; as the storage system approaches saturation, service-time jitter will typically increase. A lower qualified bandwidth means that total throughput will be reduced, but because the I/O infrastructure is under less stress, individual requests will typically be processed with less variation in individual I/O service times. Figure 3-1 illustrates these basic ideas. The precise relationship between load on the storage system and variation in I/O service time is highly dependent on your storage hardware.

Figure 3-1. Tradeoff Between Throughput and Variation in I/O Service Time (Jitter)

Some storage devices (particularly those with real-time schedulers) can provide a fixed bound on I/O service time even at utilization levels close to their maximum. In this case, the qualified bandwidth can be set higher even where applications have tight QoS requirements. The user-adjustable qualified bandwidth provides the flexibility required for GRIO to work with both dedicated real-time devices as well as more common off-the-shelf storage systems.

---

Note: In all cases, you must verify the chosen qualified bandwidth by testing the storage system under a realistic workload.

---

You can use the grioqos(1M) tool to measure the delivered QoS performance. This tool extracts QoS performance for an active stream without disturbing the application or the kernel scheduler. GRIO maintains very detailed performance metrics for each active stream. Using the grioqos command while running a workload test lets you answer questions such the following for every active stream in the system:

For more information about GRIO tools and the mechanisms for accessing QoS data within the kernel, see Chapter 5, “Monitoring Quality of Service”, and the grioqos(1M) man page.

A managed volume can be one of the following:

For more information about CXFS, see “Cluster Volume Domain Configuration”, CXFS Administration Guide for SGI InfiniteStorage, and CXFS MultiOS Client-Only Guide for SGI InfiniteStorage.

Cluster Volume Domain Configuration

You must use the cxfs_admin(1M) or cmgr(1M) command to configure cluster volumes for GRIO.

A prompting mode is also available for cxfs_admin(1M) or cmgr. For more information, see the CXFS Administration Guide for SGI InfiniteStorage.

If you have installed the cxfs.sw.grio2_cell subsystem and turned on GRIO, the ggd2 daemon will automatically query the cluster configuration database for GRIO volume configuration information. ggd2 dynamically tracks updates to the cluster database.

cxfs_admin Configuration Examples

There are two GRIO attributes associated with filesystems:

• grio_managed, which specifies whether a filesystem is managed by GRIOv2 (true) or not ( false). The default is false. Setting grio_managed to false disables GRIO management for the specified filesystem, but it does not reset the grio_qual_bandwidth value. In this case, grio_qual_bandwidth is left unmodified in the cluster database and ignored.

• grio_qual_bandwidth, which specifies a filesystem's qualified bandwidth in bytes (B suffix), kilobytes (KB), megabytes (MB), or gigabytes (GB), where the units are multiples of 1024. The default is MB for 4000 or less, B for 4001 or greater. If the filesystem is GRIO-managed, you must specify a qualified bandwidth with this attribute. You can modify the qualified bandwidth for a mounted filesystem without taking it offline.

  ---

  Note: These are advanced-mode attributes. When configuring GRIO with cxfs_admin, you should use set mode=advanced .

  ---

  For example, any one of the following commands sets a filesystem's qualified bandwidth to 1.2 GB/s:

  cxfs_admin:cluster> modify filesystem grio_qual_bandwidth=1288500000 cxfs_admin:cluster> modify filesystem grio_qual_bandwidth=1258300KB cxfs_admin:cluster> modify filesystem grio_qual_bandwidth=1288.8MB cxfs_admin:cluster> modify filesystem grio_qual_bandwidth=1288.8MB

  
  

cmgr Configuration Examples

To mark a filesystem as GRIO-managed and set its qualified bandwidth, use the following commands:

# /usr/cluster/bin/cmgr Welcome to SGI Cluster Manager Command-Line Interface cmgr> modify cxfs_filesystem filesystem in cluster cluster cmgr> set grio_managed to true cmgr> set grio_qualified_bandwidth to qualified_bandwidth cmgr> done

The value for qualified_bandwidth is specified in bytes per second. For example, the following sets the qualified bandwidth to 200 MB/s (200*1024*1024):

cmgr> set grio_qualified_bandwidth to 209715200

To show the current status of a shared filesystem:

cmgr> show cxfs_filesystem filesystem in cluster cluster ... GRIO Managed Filesystem: true GRIO Managed Bandwidth: qualified_bandwidth ...

---

Note: In cmgr, you must unmount a filesystem before you can modify it.

---

The GRIO FLEXlm licensing regime controls a number of configuration parameters including the total number of active streams and the total aggregate qualified bandwidth of filesystems under management. Separate license types are provided for the local and cluster volume domains, and license constraints are enforced for each volume domain separately.

The ggd2 daemon checks the license at startup, whenever it detects a configuration change, or when it receives a SIGHUP signal.

The license enforcement policy for streams is straightforward. The license for a given volume domain specifies a maximum number of active streams. All reservation requests above this limit are denied.

In the case of bandwidth, a license specifies the maximum total aggregate qualified bandwidths for all volumes within the volume domain. The ggd2 daemon validates the configuration at startup and whenever the configuration is changed:

If the configuration of a volume domain is altered and becomes unlicensed, ggd2 enters a passive mode in which all further requests pertaining to that domain, with the exception of release requests, are denied. A message is sent to the system log and that volume domain will remain deactivated until the configuration returns to a licensed state, at which time another message will be logged indicating the domain is again active.

For more information, see the license.dat (5) man page.

The ggd2.options file contains the command line options for ggd2 when launched at startup.

The location of the file differs by operating system:

You can uncomment and edit lines as required. The arguments are as follows:

-d level	Sets the maximum debug level and logs the specified level of messages to both the system log and to an additional log file called /var/tmp/ggd2log/ PID. level is an integer in the range 0 through 4(the higher the level number, the more debug information that is output). The levels are as follows: 0 logs critical system resource errors 1 logs the above plus ggd2 -specific error and warning messages 2 logs the above plus important events or state changes 3 logs the above plus infrequent, less-important events 4 logs the above plus debug-level messages By default, ggd2 logs just level 0 critical system resource errors to the system log only.
-f	Runs ggd2 in the foreground. By default, ggd2 is started as a daemon.
-m bw	Specifies the minimum amount of bw bandwidth in KB/s (default) that ggd2 will allocate for non-guaranteed user and system I/O per GRIO-managed volume. All nodes issuing non-GRIO I/O will receive a fair share of this minimum bandwidth. A node will be allocated the bigger value specified by -m or -s. For example, -m2048 causes ggd2 to allocate a minimum of 2048 KB/s to each GRIO-managed volume. This bandwidth becomes permanently allocated to non-GRIO I/O and cannot be reserved for GRIO I/O. Use the suffix K or M to explicitly specify bandwidth in KB/s or MB/s. For example, -m3M causes ggd2 to allocate a minimum of 3 MB/s to each GRIO-managed volume.
-r percent	Reserves a percentage of each volume's available qualified bandwidth for GRIO I/O. Reservation requests are then serviced directly from this pool of cached free bandwidth without blocking. percent is the percentage of each volume's qualified bandwidth that ggd2 attempts to keep unallocated, expressed as an integer in the range 0 through 100. You should choose this value based on the following: Expected I/O utilization levels Importance of minimizing the stream creation latency Expected rate at which reservation requests will be made By default, ggd2 allows unreserved bandwidth to be allocated for servicing non-GRIO I/O. This maximizes the total throughput of the system. However, as ggd2 only makes adjustments to these allocations periodically, a new reservation may block until ggd2 can reclaim the requested bandwidth. Note: Using the -r option causes a proportion of the unreserved I/O capacity to remain unused and reduces the total throughput and efficiency of the system for non-GRIO I/O. You should only use this option if minimizing reservation latency is a priority. For example, given a volume with a qualified bandwidth of 200 MB/s, -r 20 will instruct ggd2 to try to keep up to 20% (40 MB/s) of any remaining unreserved bandwidth cached and available for servicing reservation requests directly. ggd2 adjusts this cache of free bandwidth every time the distributed bandwidth allocator (DBA) runs, which defaults to once every 2 seconds (see -u). With these settings, ggd2 will be able to grant an additional 40 MB/s every 2 seconds without blocking any reservation requests.
-s bw	Specifies the minimum amount of bw bandwidth in KB/s (default) that ggd2 will allocate for non-GRIO user and system I/O per node. A node will be allocated the bigger value specified by -s or -m. For example, -s2048 causes ggd2 to allocate a minimum of 2048 KB/s to each node accessing a GRIO-managed volume. This bandwidth becomes permanently allocated to non-GRIO I/O and cannot be reserved for GRIO I/O. Use the suffix K or M to explicitly specify bandwidth in KB/s or MB/s. For example, -s3M causes ggd2 to allocate a minimum of 3 MB/s to each node accessing a GRIO-managed volume.
-u ms	Specifies the distributed bandwidth allocator (DBA) update interval in milliseconds (ms), where ms is a value in the range 250 through 100000. The default is 2000 (2 seconds). For more information about DBA, see “Managing Bandwidth: Encapsulation and Distributed Bandwidth Allocator” in Chapter 2. Note: The rate at which the DBA runs affects the delay that an application or node that does not have a GRIO reservation might experience when it starts doing I/O. The longer the interval, the longer a node may have to wait (with its I/O paused) before ggd2 will increase its allocation.

For example:

# command line options for ggd2 when launched from /etc/init.d/grio2
# uncomment/edit lines as required
#

# Minimum non-GRIO bandwidth per node. Units are Mbytes/sec
# -s 1M

# debug level, in the range 0 to 4
# -d 1

# minimum reserve bandwidth to accommodate short-latency reservation
# demands, expressed as a percentage of the total qualified bandwidth
# -r 30

# Distributed Bandwidth Allocator (DBA) update interval
# value in milliseconds
# -u 2000

For changes to this file to take effect, do one of the following on the GRIO server, which will cause ggd2 to reread its options file:

To determine the active GRIO server, use the grioadmin -sv command.

---

Note: In the event of a GRIO server relocation or recovery, you must perform the above steps on each GRIO server-capable node in the cluster.

---

On IRIX, it is possible to have both the GRIOv1 and GRIOv2 subsystems installed. However, only one of these subsystems can be active. The subsystem that is turned on in chkconfig is started by default at boot time and remains in effect until the chkconfig setting is changed and the machine is rebooted.

Starting GRIOv2 when GRIOv1 is Active

Suppose you were running GRIOv1 and wanted to switch to GRIOv2. After performing the configuration tasks discussed in this guide, you would do the following:

1. Turn off GRIOv1 ( grio) and turn on GRIOv2 (grio2):

   # chkconfig grio off # chkconfig grio2 on

   
   

2. Reboot the system to allow the kernel to be reinitialized with the GRIOv2 scheduler.

You do not need to manually start GRIOv2 because the daemon is automatically started upon reboot when the chkconfig setting is on.

---

Note: If GRIOv1 is still enabled when you perform a GRIOv2 library call, the return will be ENOSYS. If you do not have either the GRIOv1 or GRIOv2 kernel initialized, the return will be EAGAIN, indicating that the subsystem has not yet initialized and the application should retry the request.

---