Use the following tools to determine the initial I/O performance and to verify improvements in I/O performance after you try different optimization techniques:

The following types of optimization may improve I/O performance:

“Enhancing Performance” in Chapter 11, also provides further information about using FFIO to enhance I/O performance.

The following source program changes may affect the I/O performance of a Fortran program:

I/O optimization can often be accomplished by simply addressing I/O speed. The following UNICOS storage systems are available, ranked in order of speed:

Fast storage systems are expensive and have smaller capacities. You can specify a fast device through FFIO layers and use several FFIO layers to gain the maximum performance benefit from each storage medium. The remainder of this chapter discusses many of these FFIO optimizations. These easy optimizations are frequently those that yield the highest payoffs.

The ja(1) command is a job accounting command that can help you determine if optimizing your program will return any significant gain. For complete details about the ja command, see the ja man page.

To use ja(1), enter the following commands:

ja
a.out
ja -ct

These commands produce the following program execution summary that indicates the time spent in I/O:

Command  Started    Elapsed    User CPU   Sys CPU   I/O Wait I/O Wait
  Name      At      Seconds     Seconds   Seconds   Sec Lck  Sec Unlck
======== ======== =========== ========== ========== ======== ==========
a.out    17:15:56      4.5314     0.2599     0.2242   3.9499     0.1711

This output indicates that this program has a large amount of I/O wait time. The following section describes how to obtain a profile of the I/O activity in the program.

A significant part of this example program performs I/O; therefore, you can use procstat and related tools to obtain an I/O profile. For complete details about using these tools, see the Cray Research publications, UNICOS Performance Utilities Reference Manual, and the UNICOS User Commands Reference Manual, or the procview(1) man page.

The procstat tool is not available on CRAY T3E systems.

The procstat tool set does not require access to the program source files. The run-time library has built-in I/O data collection that is invoked when a program is run with procstat. The set of statistics generated usually provides enough information to tune I/O in a Fortran program without altering the source code.

The procview tool creates one or more reports from the raw output that the procstat command generates. It may also be run interactively, both in line-mode and by using the X Window System interface. The procview command presents an interactive menu when no command-line report option is included; otherwise, an output option can be specified and the report output can be redirected to a file.

To run the program under procstat, enter the following commands:

procstat -R raw a.out
procview -l -Fs raw

The -l option selects the long form report, and the -Fs option selects Fortran files sorted by maximum file size. The resulting report summaries the I/O activity of each Fortran file in the following format:

=======================================================================

Fortran Unit Number            1
File Name                      fort.1
Command Executed               t1
Date/Time at Open              05/31/91 17:00:19
Date/Time at Close             05/31/91 17:00:26
System File Descriptor         4
Type of I/O                    sequential unformatted
File Structure                 COS blocked
File Size                      8032256 (bytes)
Total data transferred         32129024 (bytes)


Fortran I/O   Count of           Real
Statement    Statements         Time
------------ ---------- --------------
READ              6000         5.3625
WRITE             2000         1.6484
REWIND               3          .0011
CLOSE                1          .0019


 4014.6    Bytes transferred per Fortran I/O statement
  87.70%   Of Fortran I/O statements did not initiate a system request

System I/O   # of    # Bytes    # Bytes     Wait Time (Clock Periods)
Function       Calls  Processed  Requested      Max       Min      Total
------------ -------  ---------  ---------  --------- --------- ---------
Read             738   24096768   24182784    9010627    135443 865007072
Write            246    8032256    8032256   10674103    133840 253750720
Seek               4        n/a        n/a      42061      3746     55067
Truncate           1        n/a        n/a      17462     17462     17462

System I/O     Avg Bytes  Percent of   Average I/O Rate
Function        Per Call  File Moved   (MegaBytes/Second)
------------ -----------  ----------  -------------------
Read             32651.4       300.0             4.643
Write            32651.4       100.0             5.276
Seek                 n/a         n/a               n/a
Truncate             n/a         n/a               n/a

=========================================================================

By examining the summary of files examined during a program, you can tell that the following types of files should be optimization targets:

Main memory is extremely fast. Cray Research provides many ways to use memory to avoid delays that are associated with transfers to and from physical devices.

The mr FFIO layer, which permits files to reside in main memory, is available on all UNICOS and UNICOS/mk systems. If the memory space is large enough, you can eliminate all system requests for I/O on a file. The previous procstat / procview report contains the following information:

To apply 8 Mbytes of memory to this file, use the following assign command and then rerun the job:

assign -F blocked,mr::1961 u:1

The maximum size of 1961 is calculated by dividing the file size of 8,032,256 bytes by the sector size of 4096 bytes.

The -F option invokes FFIO. The blocked,mr specification selects the blocked layer followed by the mr layer of FFIO. The u:1 argument specifies unit 1. Figure 13-2 shows I/O data movement when you use the assign command.

Figure 13-2. I/O data movement

The data only moves to and from the buffer of the mr layer during the operation of the READ, WRITE, and REWIND I/O statements. It gets moved from disk during OPEN processing if it exists and when SCRATCH is not specified. It gets moved to disk only during CLOSE processing when DELETE is not specified. When the program is rerun under procview, the procview report is as follows:

=======================================================================

Fortran Unit Number     1
File Name               fort.1
Command Executed        a.out
Date/Time at Open       09/04/91 17:29:38
Date/Time at Close      09/04/91 17:29:39
System File Descriptor  4
Type of I/O             sequential unformatted
File Structure          COS blocked
File Size               8032256 (bytes)
Total data transferred  8032256 (bytes)
Assign attributes       -F blocked, mr::1961

    Fortran I/O    Count of         Real
    Statement    Statements         Time
   ------------ ---------- --------------
    READ               6000         .1663
    WRITE              2000         .0880
    REWIND                3         .0005
    CLOSE                 1         .9055

    1003.7   Bytes transferred per Fortran I/O statement
    99.99%  Of Fortran I/O statements did not initiate a system request

   System I/O  # of     # Bytes     # Bytes    Wait Time   (Clock Periods)
   Function    Calls   Processed   Requested     Max       Min       Total
   ----------- -------  ---------  ---------  ---------  ---------  ---------
   Write         1      8032256    8032256    150197242  150197242  150197242
   Seek          2        n/a        n/a         3655       3654       7309
   Truncate      1        n/a        n/a         5207       5207       5207

   System I/O     Avg Bytes   Percent of     Average I/O Rate
   Function        Per Call   File Moved     (MegaBytes/Second)
   ------------ -----------   ----------    -------------------
      Write       8032256.0      100.0            8.913
      Seek          n/a           n/a               n/a
      Truncate      n/a           n/a               n/a

===========================================================================

In the new report, notice the following:

The optional solid-state storage device (SSD) is the fastest I/O device. The SSD stores data in memory chips and operates at speeds about as fast as main memory or 10 to 50 times faster than magnetic disks.

Because SSD capacity is usually much larger than main memory, SSD is used when not enough main memory is available to store all of the possible data.

You can access the SSD through ldcache. The system uses SSD to cache the data from file systems that the system administrator selects. Caching is automatic for files in these file systems and their subdirectories.

You can also access the SSD with the FFIO sds layer. When this layer is present, library routines use the SSD to hold the file between open and close. You should use the FFIO sds layer for files that are larger than the amount of ldcache available for the file.

The SDSLIMIT and SDSINCR environment variables may have significant impact if all subfields are not specified after the SDS keyword (use of these variables is not recommended).

The following timings from a CRAY Y-MP/8 system show the typical effects of optimization on the program used in “The MR Feature”. In that example, the program writes a file and reads it three times. Because it is unnecessary to save the file afterward, the .scr type (scratch file) can be used. See “Scratch Files”, for more information about scratch files. Some of the following commands appear to produce a range because of the fluctuation in results.

You can use the sds layer and ldcache in conjunction with the mr layer. For example, to allocate 2048 Mbytes (512 sectors) of main memory for the file, with the remainder on SSD, use the following assign(1) command:

assign -F mr.scr:512:512:0,sds.scr

The first 512 blocks of the file reside in main memory and the remainder of the blocks reside on SSD.

Generally, the combination of the mr and sds layers makes the maximum amount of high performance storage available to the program. The SSD is typically used in case the file size exceeds the estimated amount of main memory you can access.

The following timings from a CRAY Y-MP/8 system show the typical effects of optimization on the program used in “The MR Feature”. The program writes a file and reads it three times. Because it is not necessary to save the file afterward, you can use the .scr (scratch file) type. See “Scratch Files”, for more information about scratch files.

The FFIO cache layer keeps recently used data in fixed size main memory or SDS buffers or cache pages in order to reuse the data directly from these buffers in subsequent references. It can be tuned by selecting the number of cache pages and the size of these pages.

The use of the cache layer is especially effective when access to a file is localized to some regions of the whole file. Well-tuned cached I/O can be an order of magnitude faster than the default I/O.

Even when access is sequential, the cache layer can improve the I/O performance. For good performance, use page sizes large enough to hold the largest records.

The cache layers work with the standard Fortran I/O types and the Cray Research extensions of BUFFER IN/OUT, READMS/WRITMS, and GETWA/PUTWA.

The following assign command requests 100 pages of 42 blocks each:

assign -F cache:42:100 f:filename

Specifying cache pages of 42 blocks matches the track size of a DD-49 disk.

It is a good idea to preallocate space; this saves system overhead by making fewer system requests for allocation, and may reduce the number of physical I/O requests. You can allocate space by using the default value from the -A and -B options for the mkfs(8) command, or by using the assign(1) command with the -n option, as follows:

assign -n sz[:st] -q ocblks

The sz argument specifies the decimal number of 512-word blocks reserved for the data file. If this option is used on an existing file, sz 512-word blocks are added to the end of the file. The -qocblks option specifies the number of 512-word blocks to be allocated per file system partition. These options are generally used with the -p option to do user-level striping. The st (stride) argument to the -n option is obsolete and should not be used; it specifies the allocation increment when allocating sz blocks.

---

Note: For immediate preallocation, use the setf(1) command because assign does not preallocate space until the file is opened.

---

Use the -c option on the assign or setf command to get contiguous allocation of space so that disk heads do not have to reposition themselves as frequently. It is important to note that if contiguous allocation is unavailable, the request fails and the process might abort also.

Generally, most users should not do user-level striping (the -p option on the assign and setf commands), because it requires disk head seek operations on multiple devices. Only jobs performing I/O with large record lengths can benefit from user-level striping. Large records are those in excess of several times the size of IOS read-ahead/write-behind segments (this varies with the disk device, but it is usually at least 16 sectors), or several times the disk track size (this varies with the disk device). In addition, asynchronous I/O has a much higher payoff with user-level striping than synchronous I/O.

The assign and setf commands have a partition option, -p, that is very important for applications that perform multifile asynchronous I/O. By placing different files on different partitions (which must be on different physical devices), multiple I/O requests can be made from a job, thus increasing the I/O bandwidth to the job. The -c option has no effect without the -n option.

When a file system is composed of partitions on more than one disk, major performance improvements can result from using the disks at the same time. This technique is called disk striping.

For example, if the file system spans three disks, partitions 0, 1, and 2, it may be possible to increase performance by spreading the file over all three equally. Although 300 sequential writes may be required, only 100 must go to each disk, and the disks may be writing simultaneously. You can specify striping in the following two ways, using the assign command:

assign -p 0-2 -n 300 -q 48 -b 144  f:filename
assign -p 0:1:2 -n 300 -q 48 -F cos:144 f:filename

The previous example also specifies a larger buffer size (144), which is three tracks (one per disk) if there are 48 sectors per track.

Using the bufa layer enhances the usefulness of user striping because bufa issues asynchronous I/O system calls, which are handled more efficiently by the kernel for user-striped files. In addition, the double buffering helps load balance the CPU and I/O processing. Using the previous example, better performance could be obtained from the bufa layer by using the following:

assign -p 0-2 -n 1000 -q 48 -F bufa:144:6

or

assign -p 0-2 -n 1000 -q 16 -F bufa:48:6

See “The bufa and cachea Layers” in Chapter 11, for information about the bufa layers.

Other factors, such as channel capacity, may limit the benefit of striping. Disk space on each partition should be contiguous and preallocated for maximum benefit.

Use striping only for very large records because all of the disk heads must do seeks on every transfer.

Use the df(1) command to list the partitions of a file system. For more information about the df command, see the UNICOS User Commands Reference Manual.

Scratch files are temporary and are deleted when they are closed. To decrease I/O time, move applications' scratch files from user file systems to high-speed file systems, such as /tmp, secondary data segments (SDS), or /ssd.

When optimizing, you should avoid writing the data to disk. This is especially important if most of the data can be held in SDS or main memory.

Fortran lets you open a file with STATUS='SCRATCH'. It also lets you close temporary files by using a STATUS='DELETE'. These files are placed on disk, unless the .scr specification for FFIO or the assign -t command is specified for the file. Files specified as assign -t or .scr are deleted when they are closed. The following assign commands are examples of using these options:

assign  -t  f:filename
assign  -F mr.scr  f:filename
assign  -F sds.scr  f:filename
assign  -F cos,sds.scr  f:filename

You can think of the program's file as a scratch file and avoid flushing it at CLOSE by using the following command:

assign -F mr.scr u:1

Figure 13-3 shows the program's current data movement:

Figure 13-3. I/O data movement (current)

The following procview report shows the difference in I/O times; the last two lines of the report indicate that both the Fortran WRITE statement time and system I/O write () time were reduced to 0.

==================================================================

    Fortran Unit Number       1
    File Name                 fort.1
    Command Executed          a.out
    Date/Time at Open         09/04/91 17:31:38
    System File Descriptor    -1
    Type of I/O               sequential unformatted
    File Structure            COS blocked - 'blocked'
    Assign attributes         -F blocked,mr.scr


    Fortran I/O    Count of         Real
    Statement    Statements         Time
   ------------ ---------- --------------
    READ               6000         .1622
    WRITE              2000         .0862
    REWIND                3         .0005
    CLOSE                 1         .0000


    0  Bytes transferred per Fortran I/O statement
   100%  Of Fortran I/O statements did not initiate a system request

====================================================================

If unit 1 is declared as a scratch file by using the assign command, fort.1 will no longer exist after program execution.

Because the original procview report indicates that no BACKSPACE was done on the file, the program might not depend on the blocked structure. Perhaps the program reads all of the data that is on every record. If it does, you can avoid using the blocked structure and save more time. Even if you cannot be sure that you do not need the blocked structure, you can still try it by using this command:

assign -F mr.scr u:1

The program will probably fail if it does require blocked structure. If it runs successfully, you will notice that it runs faster. The layer of library processing that does the record keeping was eliminated, and the program's memory use now looks like that in Figure 13-4.

Figure 13-4. I/O processing with library processing eliminated

The program is now much faster. The time saved by using the assign commands described in this section is as follows:

Total optimization impact is I/O that is 15 times faster.

You may not see these exact improvements because many variables (such as configurations) exist that affect timings.

When writing a sequential COS blocked file, the library usually waits until its buffer is full before initiating a system request to write the data to the physical device. When the system request completes, the library resumes processing the user request.

The FFIO asynchronous COS layer divides this buffer in half and begins a write operation when the first half is full, but it continues processing the user request in the second half of the buffer while the system is writing data from the first half. When reading, the library tries to read ahead into the second half of the buffer to reduce the time the job must wait while waiting for system requests. This can be twice as fast as sequential I/O requests.

The asynchronous COS layer is specified with the assign -F command, as follows:

assign  -F cos.async  f:filename
assign  -F cos.async:96  f:filename

The second assign command specifies a larger buffer because the library requests (half the specified buffer size) should be the disk track size, which is assumed to be 48 sectors.

Several FFIO layers automatically enhance I/O performance by performing asynchronous read-ahead and write-behind. These layers include:

If records are accessed sequentially, the cos and bufa layers will automatically and asynchronously pre-read data ahead of the file position currently being accessed. This behavior can be obtained with the cachea layer with an assign option; in that case, the cachea layer will also detect sequential backward access patterns and pre-read in the reverse direction.

Many user codes access the majority of file records sequentially, even with ACCESS='DIRECT' specified. Asynchronous buffering provides maximum performance when:

Use of automatic read-ahead and write-behind may decrease execution time by half because I/O and CPU processing occur in parallel.

The following assign command specifies a specific cachea layer with 10 pages, each the size of a DD-40 track. Three pages of asynchronous read-ahead are requested. The read-ahead is performed when a sequential read access pattern is detected.

assign -F cachea:48:10:3 f:filename

This command would work for a direct access or sequential Fortran file which has unblocked file structure.

To utilize asynchronous read-ahead and write-behind with ER90 tape files, you can use the bufa and the er90 layers, as in the following example:

assign -F bufa,er90 f:filename

The bufa layer must be used with the er90 layer because it supports file types that are not seekable. The bufa layer can also be used with disk files, as in the following example:

assign -F bufa:48:10 f:filename

This command specifies the same buffer configuration as the previous cachea example. The bufa layer uses all its pages for asynchronous read-ahead and write-behind. When writing, each page is asynchronously flushed as soon as it is full.

Marking records incurs overhead. If a program reads all of the data in any record it accesses and avoids the use of BACKSPACE, you can make some minor performance savings by eliminating the overhead associated with records. This can be done in several ways, depending on the type of I/O and certain other characteristics.

For example, the following assign statements specify the unblocked file structure:

assign -s unblocked f:filename
assign -s u f:filename
assign -s bin f:filename

The libraries generally have default buffer sizes. The default is suitable for many devices, but major performance improvements can result from requesting an efficient buffer size.

The optimal buffer size for very large files is usually a multiple of a device allocation for the disk. This may be the size of a track on the disk. The df -p command lists thresholds for big file allocations. If optimal size buffers are used and the file is contiguous, disk operations are very efficient. Smaller sizes require more than one operation to access all of the information on the allocation or track. Performance does not improve much with buffers larger than the optimal size, unless striping is specified.

When enough main memory is available to hold the entire file, the buffer size can be selected to be as large as the file for maximum performance.

The maximum length of a formatted record depends on the size of the buffer that the I/O library uses for a file. The size of the buffer depends on the following:

On UNICOS systems, the RECL parameter on the OPEN statement is accepted by the Fortran library for sequential access files. For a sequential access file, RECL is defined as the maximum record size that can be read or written. Thus, the RECL parameter on the OPEN statement can be used to adjust the maximum length of formatted records that can be read or written for that file.

If RECL is not specified, the following default maximum record lengths apply:

After a request is made, the library usually copies data between its own buffers and the user data area. For small requests, this may result in the blocking of many requests into fewer system requests, but for large requests when blocking is not needed, this is inefficient. You can achieve performance gains by bypassing the library buffers and making system requests to the user data directly.

To bypass the library buffers and to specify a direct system interface, use the assign -s u option or specify the FFIO system, or syscall layer, as is shown in the following assign command examples:

assign  -s u  f:filename
assign  -F system  f:filename
assign  -F syscall  f:filename

The user data should be in multiples of the disk sector size (usually 4096 bytes) for best disk I/O performance.

If library buffers are bypassed, the user data should be on a sector boundary to prevent I/O performance degradation.

When a program produces a large amount of output used only as input to another program consider using pipes. If both programs can run simultaneously, data can flow directly from one to the next by using a pipe. It is unnecessary to write the data to the disk. See Chapter 4, “Tape and Named Pipe Support ”, for details about pipes.

Major performance improvements can result from overlapping CPU work and I/O work. This approach can be used in many high-volume applications; it simultaneously uses as many independent devices as possible.

To use this method, start some I/O operations and then immediately begin computational work without waiting for the I/O operations to complete. When the computational work completes, check on the I/O operations; if they are not completed yet, you must wait. To repeat this cycle, start more I/O and begin more computations.

As an example, assume that you must compute a large matrix. Instead of computing the entire matrix and then writing it out, a better approach is to compute one column at a time and to initiate the output of each column immediately after the column is computed. An example of this follows:

     dimension a(1000,2000)
     do 20 jcol= 1,2000
       do 10 i= 1,1000
         a(i,jcol)= sqrt(exp(ranf()))
10     continue
20   continue
     write(1) a
     end

First, try using the assign -F cos.async f:filename command. If this is not fast enough, rewrite the previous program to overlap I/O with CPU work, as follows:

      dimension a(1000,2000)
      do 20 jcol= 1,2000
        do 10 i= 1,1000
          a(i,jcol)= sqrt(exp(ranf()))
10      continue
        BUFFER OUT(1,0) (a(1,jcol),a(1000,jcol) )
20    continue
      end

The following Fortran statements and library routines can return control to the user after initiating I/O without requiring the I/O to complete: