When you install the CXFS software, you must modify certain system files. The network configuration is critical. Each node in the cluster must be able to communicate with every other node in the cluster by both logical name and IP address without going through any other network routing; proper name resolution is key. SGI recommends static routing.
This section provides an overview of the steps that you should perform on your nodes prior to installing the CXFS software. It contains the following sections:
 | Caution: It is critical that you understand these rules before attempting to configure a CXFS cluster. |
Use the following hostname resolution rules and recommendations when defining a node:
The first node you define in the pool must be an administration node.
Hostnames cannot begin with an underscore (_) or include any white-space characters.
The private network IP addresses on a running node in the cluster cannot be changed while cluster services are active.
You must be able to communicate directly between every node in the cluster (including client-only nodes) using IP addresses and logical names, without routing.
A private network must be dedicated to be the heartbeat and control network. No other load is supported on this network.
The heartbeat and control network must be connected to all nodes, and all nodes must be configured to use the same subnet.
If you change hostname resolution settings in the /etc/nsswitch.conf file after you have defined the first administration node (which creates the cluster database), you must re-create the cluster database.
When configuring your network, remember the following:
You must be able to communicate between every node in the cluster directly using IP address and logical name, without routing.
Dedicate a private network to be your heartbeat and control network. No other load is supported on this network.
The heartbeat and control network must be connected to all nodes, and all nodes must be configured to use the same subnet for that network.
The following procedure provides an overview of the steps required to add a private network.
 | Note: A private network is required for use with CXFS. |
You may skip some steps, depending upon the starting conditions at your site.
Edit the /etc/hosts file so that it contains entries for every node in the cluster and their private interfaces as well.
The /etc/hosts file has the following format, where primary_hostname can be the simple hostname or the fully qualified domain name:
IP_address primary_hostname aliases
You should be consistent when using fully qualified domain names in the /etc/hosts file. If you use fully qualified domain names on a particular node, then all of the nodes in the cluster should use the fully qualified name of that node when defining the IP/hostname information for that node in their /etc/hosts file.
The decision to use fully qualified domain names is usually a matter of how the clients are going to resolve names for their client/server programs (such as NFS), how their default resolution is done, and so on.
Even if you are using the domain name service (DNS) or the network information service (NIS), you must add every IP address and hostname for the nodes to /etc/hosts on all nodes. For example:
190.0.2.1 server1-company.com server1 190.0.2.3 stocks 190.0.3.1 priv-server1 190.0.2.2 server2-company.com server2 190.0.2.4 bonds 190.0.3.2 priv-server2
You should then add all of these IP addresses to /etc/hosts on the other nodes in the cluster.
For more information, see the hosts and resolve.conf man pages.
Note: Exclusive use of NIS or DNS for IP address lookup for the nodes will reduce availability in situations where the NIS or DNS service becomes unreliable.
Edit the /etc/nsswitch.conf file so that local files are accessed before either NIS or DNS. That is, the hosts line in /etc/nsswitch.conf must list files first.
For example:
hosts: files nis dns
(The order of nis and dns is not significant to CXFS, but files must be first.)
Configure your private interface according to the instructions in the Network Configuration section of your Linux 64-bit distribution manual. To verify that the private interface is operational, use the ifconfig -a command. For example:
[root@linux64 root]# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:50:81:A4:75:6A inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13782788 errors:0 dropped:0 overruns:0 frame:0 TX packets:60846 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:826016878 (787.7 Mb) TX bytes:5745933 (5.4 Mb) Interrupt:19 Base address:0xb880 Memory:fe0fe000-fe0fe038 eth1 Link encap:Ethernet HWaddr 00:81:8A:10:5C:34 inet addr:10.0.0.10 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:19 Base address:0xef00 Memory:febfd000-febfd038 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:162 errors:0 dropped:0 overruns:0 frame:0 TX packets:162 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11692 (11.4 Kb) TX bytes:11692 (11.4 Kb)
This example shows that two Ethernet interfaces, eth0 and eth1, are present and running (as indicated by UP in the third line of each interface description).
If the second network does not appear, it may be that a network interface card must be installed in order to provide a second network, or it may be that the network is not yet initialized.
(Optional) Make the modifications required to use CXFS connectivity diagnostics. See “IRIX Modifications Required for CXFS Connectivity Diagnostics” in Chapter 6, and “Linux 64-bit Modifications Required for CXFS Connectivity Diagnostics ” in Chapter 7.
To configure IRIX network interfaces, do the following:
Ensure that name services are available. See step 1 in “Adding a Private Network”.
On one node, add that node's interfaces and their IP addresses to the /etc/config/netif.options file.
if1name=ec0 if1addr=$HOSTNAME
$HOSTNAME is an alias for an IP address that appears in /etc/hosts.
If there are additional interfaces, their interface names and IP addresses appear on lines like the following:
if2name= if2addr=
In the example, the control network name and IP address are as follows:
if3name=ec3 if3addr=priv-$HOSTNAME
The control network IP address in this example, priv-$HOSTNAME , is an alias for an IP address that appears in /etc/hosts .
If there are more than eight interfaces on the node, change the value of if_num in /etc/config/netif.options to the number of interfaces. For fewer than eight interfaces, the line is as follows:
if_num=8
Repeat steps 1 through 3 for the other nodes.
Edit the /etc/config/routed.options file on each IRIX node so that the routes are not advertised over the control network. See the routed(1M) man page for a list of options.
For example:
-q -h -Prdisc_interval=45
The options do the following:
For each private network on each node in the pool, verify access with the ping command. Enter the following, where nodeIPaddress is the IP address of the node:
ping nodeIPaddress |
For example:
[root@linux64 root]# ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1) from 128.162.240.141 : 56(84) bytes of data. 64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.310 ms 64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.122 ms 64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.127 ms |
Also execute a ping on the public networks. If ping fails, follow these steps:
Verify that the network interface was configured up using ifconfig. For example:
[root@linux64 root]# ifconfig eth1 eth1 Link encap:Ethernet HWaddr 00:81:8A:10:5C:34 inet addr:10.0.0.10 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:19 Base address:0xef00 Memory:febfd000-febfd038
In the third output line above, UP indicates that the interface was configured up.
Verify that the cables are correctly seated.
Repeat this procedure on each node.
If one IRIX administration node is configured to reset another IRIX administration node, you must turn off the getty process for the tty ports to which the reset serial cables are connected. You must do this on the IRIX administration node performing the reset (not the node receiving the reset). To do this, perform the following steps on each IRIX administration node; if you have a cluster with nodes running other operating systems, see the CXFS MultiOS Client-Only Guide for SGI InfiniteStorage.
Determine which port is used for the reset line. ttyd2 is the most commonly used port, except on Origin 300 and Origin 350 system, where ttyd4 is commonly used.
Find the line for the port by looking at the comments on the right for the port number from step 1.
Change the third field of this line to off. For example, for an Origin 3000:
t2:23:off:/sbin/getty -N ttyd2 co_9600 # port 2Save the file.
Enter the following commands to make the change take effect:
# killall getty # init q