This appendix contains a summary of the new features for each version of this guide.
CXFS version 1 is the original implementation of CXFS.
The 007-4016-002 update contains additional troubleshooting information and instructions for unmounting and remounting filesystems with the command line interface. It was reorganized to make the tasks of installation and configuration clearer.
The 007-4016-003 update contains the following:
Metadata server recovery information
Administrative shutdown procedures
Additional troubleshooting information
Instructions for unmounting and remounting filesystems with the CLI
Reorganized installation and configuration information
The 007-4016-004 update contains the following:
Support for hierarchical storage management (HSM) through data management application programming interface (DMAPI), also know as X/Open data storage management specification (XDSM)
Changes to administrative shutdown, including two new cmgr subcommands to stop CXFS services on the local nodes: admin cxfs_stop and admin cxfs_stop
Quorum changes without panics
The 007-4016-005 update contains the following:
Coexecution of CXFS and IRIS FailSafe 2.1, including commands to convert nodes and clusters to apply to both utilities
Ability to use the cmgr command without extra prompting (-p), permitting the use of scripts
New tasks to revoke and allow membership of the local node
Ability to specify the tie-breaker node, which is used in the process of computing node membership for the cluster when exactly half the nodes in the cluster are up and can communicate with each other
Clarification that a single subnet should be used
The 007-4016-006 update contains the following:
Clarifications about CXFS shutdown and database shutdown
Additional information about CXFS daemons
Clarifications to the comparison of XFS and CXFS
The 007-4016-007 update contains the following:
Addition of the Origin 3000 partition ID to node configuration
Troubleshooting information for a two-node cluster when both nodes go down
Information about editing the /etc/hosts file to use an alternate interface for heartbeat and control.
Clarification about the use of hardware reset and tie-breaker nodes
Ability to unset the tie-breaker node
Use of fsr
CXFS version 2 includes client-only nodes on operating system platforms other than IRIX (multiOS cluster, or heterogeneous clients).
The 007-4016-008 update contains the following:
A cluster of at least three weighted nodes is recommended for a production environment (that is, one requiring relocation and recovery of the metadata server).
If you use a two-weighted-node cluster for production, you must do one of the following:
Use reset lines to avoid data corruption and ensure that only one node is running in error conditions (reset lines are recommended for all CXFS clusters and required for use with IRIS FailSafe).
Weight one node as 1 and the other as 0.
Set a tie-breaker node.
However, there are issues with a two-weighted-node cluster.
The new cluster_status command, which provides a curses interface to display status information gathered by the cad daemon (this information is also displayed by the cxdetail command).
Cluster nodes can run adjacent levels of the IRIX operating system (OS); for example, 6.5.11f and 6.5.12f (this applies as of 6.5.12f).
The ability to execute your own custom scripts around mounting operations.
Partition ID information.
Clarification about the following:
Hostname resolution rules; it is critical that you understand these rules and have files configured poperly before attempting to configure a cluster.
The difference between CXFS membership and fs2d membership.
Configuration of nodes supported in an IRIS FailSafe and CXFS coexecution environment.
Unwritten extent tracking (unwritten=1|0 ) with CXFS.
Including the CXFS mount point in the /etc/exports file.
Number of nodes supported: 16 CXFS nodes, and up to 8 IRIS FailSafe nodes with coexecution.
The flow of information in a coexecution cluster.
The 007-4016-009 update contains the following:
The structure of the CXFS filesystem configuration has changed. CXFS filesystems can now be defined, modified, managed and deleted independently of each other and of the cluster definition. (Previously, the CXFS filesystems were defined as attributes to the cluster definition.)
The new design improves the performance, flexibility and reliability of filesystem configuration. To accommodate clusters mixing nodes running 6.5.12 and 6.5.13, backwards compatibility is enforced by default in 6.5.13. The result is that the performance achievements are not visible; however, if you are 6.5.13 on all nodes in the cluster, you may wish to turn off backwards compatibility. Backwards compatibility will be turned off in the 6.5.14 release.
Information about locating the xfsdump inventory in a shared directory.
Information about the IRIS FailSafe CXFS resource type that can be used to failover applications that use CXFS filesystems.
The -p option is no longer required when defining filesystems with the cmgr command; the scripting capability is therefore provided.
For certain GUI tasks, the ability to select all nodes at once in addition to specifying nodes individually.
New /var/cluster/cmgr-scripts/rotatelogs script to save log files with day and month name as suffixes.
The setting to force an unmount of a filesystem using the umount -k option is turned off by default in the GUI. There is no default when using the cmgr command.
Clarification of the term CLI to mean the underlying set of commands that are used by the cmgr cluster manager tool and by the GUI.
Use of sgi_apache.sw.server.
Correction: real-time filesystems are not currently supported. Changes to reflect this have been made in text.
New and revised figures.
The 007-4016-011 update contains the following:
The graphical user interface (GUI) has been improved. The separate cluster view (the cxdetail command) and task manager (the cxtask command) have been streamlined into one window, the CXFS Manager. Both the cxtask and cxdetail commands are kept for historical purposes; this document refers to just cxtask for simplicity.
The new GUI provides the following features:
Access to tasks through the menu bar or by clicking the right mouse button within the tree view
Faster filesystem status and cluster status updates
Access to the salog (4) file, which shows every command run from the GUI
A Find textfield helps you find components within the displayed tree-view
Information about the use of xfs_repair and CXFS filesystems.
Caution: Do not use xfs_repair on a CXFS filesystem unless you are certain there is a problem.
Information about using cmgr (1M):
Invoking subcommands directly on the command line with the -c option
Using template scripts provided in the /var/cluster/cmgr-templates directory
Information about MAC labels in a mixed Trusted IRIX and IRIX cluster.
The structure of the CXFS filesystem configuration was changed with the release of IRIX 6.5.13. Backward compatibility with earlier versions is no longer maintained as of IRIX 6.5.14, because all nodes in the cluster must be running the same or adjacent releases.
If you are upgrading from 6.5.13f entirely to 6.5.14f, there is no further impact.
If you intend to run a mixture of 6.5.13f and 6.5.14f nodes, you must turn off backward compatibility.
If you are upgrading from 6.5.12f or earlier without first installing and running 6.5.13f, then you must perform a one-time manual conversion of your CXFS filesystem definitions.
The 007-4016-012 update contains the following:
 | Note: Relocation and recovery are deferred in this release. |
Support for clients of other operating systems such as Solaris and Windows NT as defined in the CXFS MultiOS Client-Only Guide for SGI InfiniteStorage. These clients will be released asynchronously from the IRIX release. This support will require a minimum of IRIX 6.5.15f plus appropriate patches. For more information, see your SGI support contact.
Default scripts are now provided in the /var/cluster/clconfd-scripts directory to permit NFS-exporting of CXFS filesystems listed in /etc/exports.
Reset lines are mandatory for two-node and two-weighted node clusters. Larger clusters should have an odd number of weighted nodes, or must have serial reset lines if only two of the nodes are weighted.
Simplification of Chapter 1. General information about the CXFS Manager GUI and cmgr have been moved to their respective reference chapters, coexecution details have been moved into a separate chapter, and the communication flow diagrams and daemon information have been moved into an appendix.
Information about the error messages that may cause administrators to use xfs_repair inappropriately.
Changes to the rotatelogs script syntax. The root crontab file now has an entry to run the rotatelogs script weekly. If you run the script twice in one day, it will append the current log file to the previous saved copy, rather than overwriting it.
A new figure describing some of the various combinations of node and cluster types in a coexecution cluster.
The 007-4016-013 update contains the following:
| Note: Relocation and recovery are fully implemented, but the number of associated problems prevents support of these features in CXFS. While data integrity is not compromised, cluster node panics or hangs are likely to occur. These features will be fully supported when these issues are resolved. |
Support for Solaris and Windows NT systems in a multiple operating system (multiOS) cluster, including the following:
Information about defining the operating system for a node. For existing clusters that are upgraded to IRIX 6.5.16f, existing nodes will be assigned an operating system type of IRIX.
Information about I/O fencing, which allows a problem node to be isolated from the storage area network (SAN) so that it cannot corrupt data in the shared CXFS filesystem. Solaris and Windows NT nodes require a Brocade switch in order to support I/O fencing for data integrity protection; therefore, the Brocade switch is a required piece of hardware in a cluster running multiple operating systems.
The new terms multiOS and CXFS client-only node.
Support for the L1 controller on SGI Origin 300, SGI Origin 3200C, SGI Onyx 300, and SGI Onyx 3200C systems.
Information about the CXFS GUI tasks to define and modify a filesystem, which have been split into two pages for ease of use.
New GUI icons.
The 007-4016-014 update contains the following:
A new appendix contains an example /etc/ipfilterd.conf file that can be used to provide IP filtering for the CXFS private network.
The build_cmgr_script command, which generates a cmgr script from the cluster database. The script can be used later to recreate the cluster database after performing a cdbreinit command.
A sample script to unexport and locally unmount an lofs filesystem.
Use of the new command name cxfsmgr. The cxfsmgr command has the same function as the cxtask and cxdetail commands, which are kept for historical purposes.
Clarifications to the following:
Starting the CXFS Manager graphical user interface
Masking and I/O fencing
Terminology such as cluster, node, and pool
Terminology used to describe the GUI
The 007-4016-015 update contains the following:
| Note: In this release, relocation is disabled by default and recovery
is supported only when using standby nodes.
A standby node is a metadata server-capable administration node that is configured as a potential metadata server for a given filesystem, but does not currently run any applications that will use that filesystem. To use recovery, you must not run any applications on any of the potential metadata servers for a given filesystem; after the active metadata server has been chosen by the system, you can then run applications that use the filesystem on the active metadata server and client-only nodes. Relocation and recovery are fully implemented, but the number of associated problems prevents full support of these features in the current release. Although data integrity is not compromised, cluster node panics or hangs are likely to occur. Relocation and recovery will be fully supported in a future release when these issues are resolved. |
IRIX nodes may now be CXFS client-only nodes, meaning that they run a minimal implementation of the CXFS and cluster services, and do not contain a copy of the CXFS cluster database. Client-only nodes are installed with the cxfs_client software product.
This change also introduces the term CXFS administration node, which is a node that is installed with the cluster_admin software product, allowing the node to perform cluster administration tasks and contain a copy of the cluster database. Nodes that you want to run as metadata servers must be installed as CXFS server-capable administration nodes; SGI recommends that all other nodes be installed as client-only nodes.
When you define a node, you no longer need to specify the node weight. This has been replaced by the Node Function field, allowing you to choose Server-capable Admin, Client Admin, or Client-Only. (For Solaris and Windows nodes, Client-Only is automatically selected for you.) Similar fields are provided for the cmgr command.
When upgrading to 6.5.18f, already existing IRIX nodes will by default be assigned as Server-capable Admin if they had a weight of 1.
This version also clarifies the terms used for membership: CXFS kernel membership and cluster database membership .
New system-tunable parameters:
cxfs_relocation_ok lets you enable or disable the relocation feature; relocation is disabled by default in this release, and SGI recommends that you do not enable it.
cxfsd_min and cxfsd_max let you specify the minimum and maximum number of cxfsd threads to run per CXFS filesystem.
New commands:
cxfs_info provides status information about the cluster, nodes, and filesystems and is run from a client-only node.
cxfsdump gathers CXFS configuration information.
A CXFS cluster is supported with as many as 32 nodes. As many as 16 of those nodes can be CXFS administration nodes and all other nodes can be client-only nodes. You can choose to define a node as a CXFS client administration node, however, SGI strongly recommends that only potential metadata servers be configured as CXFS server-capable administration nodes and that there be an odd number of server-capable nodes for quorum calculation purposes.
The graphical user interfaces for XVM and CXFS have been combined into one. This guide provides an overview of the XVM-specific tasks provided by the GUI; for details about these tasks, see the XVM Volume Manager Administrator's Guide.
The tasks to make, grow, mount/unmount a filesystem are now provided in the GUI.
Tips about using CXFS and Trusted IRIX.
Support for Microsoft Windows 2000 systems as client-only nodes. (This guide uses Windows to refer to both Microsoft Windows NT and Microsoft Windows 2000 nodes when the information applies equally to both. Information that applies to only one of these types of nodes is identified.)
The 007-4016-016 update contains the following:
The new rolling annual upgrade policy that permits you to upgrade from 6.5.n to the n+1 or n+4 release, as of 6.5.18f.
The time required to update and propagate the database across nodes in the cluster has been significantly decreased.
If you use I/O fencing and ipfilterd on a node, the ipfilterd configuration must allow communication between the node and the telnet(1) port on the switch.
The following nodes do not contain system controllers and therefore require I/O fencing for data integrity protection:
Silicon Graphics Fuel visual workstation
Silicon Graphics Octane system
Silicon Graphics Octane2 system
The CXFS Manager graphical user interface (GUI) has added a new icon to represent client-only nodes.
In preparation for future CXFS MultiOS client releases, the CXFS software now also allows you to specify the Linux, IBM AIX, and Hewlett-Packard HP-UX operating systems when defining a node. For support details, see the CXFS MultiOS Client-Only Guide for SGI InfiniteStorage and release notes.
This version clarifies the various methods to perform cluster database backups and restorations.
Application programmers should be aware that XFS recently relaxed the requirement that direct I/O be aligned along filesystem block boundaries. As of IRIX 6.5.19f, direct I/O will also be accepted using 512-byte alignment.
This change makes the use of direct I/O on a CXFS partition more consistent with that of other vendor's requirements and thus makes the use of CXFS more transparent. See the description of direct I/O requirements in the fcntl man page.
This version lists the system tunable parameters found in the /var/sysgen/mtune/cell file, some of which should not be modified.
The 007-4016-017 update contains the following:
Changes to the CXFS graphical user interface (GUI):
New login connection choices, including support for a remote shell connection, which connects to the server via a user-specified command shell, such as rsh or ssh.
The ability for the root user to grant other users permission to execute specific GUI tasks.
Use of Java2 for the CXFS GUI, which simplifies the Java installation and co-operation with third-party GUIs. This also enhances the ability to run the GUI through a web browser (via http:// server/CXFSManager/).
Information about using the right mouse button to access tasks appropriate to the selected GUI item.
Changes to the cxfsd_min and cxfsd_max defaults, and the cxfsd_max legal values.
More information about memberships, quorums, and tiebreakers.
A new figure describing standby mode.
More information about IRIX client-only issues:
Client-only node system files
Status in log files
cxfs_client error messages
CXFS version 3 adds support for CXFS metadata servers on SGI Altix systems running Linux 64-bit.
The 007-4016-018 update contains the following:
Support for SGI ProPack for Linux 64-bit metadata servers on SGI Altix 3000 family of servers and superclusters. A CXFS cluster can contain either Linux 64-bit for SGI ProPack 2.3 server-capable nodes on Altix systems or IRIX server-capable nodes; you cannot mix IRIX and Linux 64-bit server-capable nodes within one cluster.
CXFS does not support the relocation or recovery of DMAPI filesystems that are being served by Linux 64-bit metadata servers.
Coexecution with FailSafe is not supported on Linux 64-bit nodes.
Due to packaging enhancements, CXFS may now be installed on the M stream or the F stream.
The IRIX CXFS software will no longer be bundled in the IRIX overlay CDs but instead is on a separate CXFS IRIX Server and Client 3.0 for IRIX 6.5.22 CD. This changes the installation procedure.
Note: If you are upgrading from a previous IRIX release and have CXFS installed, you must upgrade both IRIX and CXFS. If you try to upgrade one without the other, conflicts will occur. Information about defining networks for CXFS kernel messaging (in addition to the network used for heartbeat/control). However, use of these networks is deferred.
Support for IRIX real-time filesystems.
Suggestions for configuring large clusters.
Information about using ping to verify general connectivity and CXFS heartbeat in a multicast environment.
The GUI has been changed to show a single display for the nodes in the cluster and nodes that are in the pool but not in the cluster. This new selection is View: Nodes and Cluster.
Information about information retaining system core files and the output from the cxfsdump utility when reporting problems.
Information about monitoring heartbeat timeouts for IRIX using Performance Co-Pilot or the icrash command.
The ability to define multiple CXFS filesystems at one time with the GUI.
The 007-4016-019 update contains the following:
Information about migrating from an IRIX cluster to a Linux 64-bit cluster
Support for a cluster of up to 64 nodes.
Information about the TP9300 RAID.
Information about the cxfs-config command.
Clarification that serial hardware reset lines or I/O fencing is required for all nodes in order to protect data integrity.
The ability to define a reset method for a given node to one of the following:
powerCycle to turn power off and on
reset to perform a serial reset
nmi to perform a nonmaskable interrupt
You can define this method using either the cmgr command or the GUI. You can manually perform a powercycle or an NMI with the cmgr command.
New appendixes summarizing operating system path differences and new features from previous releases