Most human-readable files are in text format; this format contains records comprised of ASCII characters with each record terminated by an ASCII line-feed character, which is the newline character in UNIX terminology. The FFIO specification that selects this file structure is assign -F text.

The FFIO package is seldom required to handle text files. In the following types of cases, however, using FFIO may be necessary:

I/O speed is important when optimizing text file access. Using assign -F text is expensive in terms of CPU time, but it lets you use memory-resident and SDS files, which can reduce or eliminate I/O wait time.

The FFIO system also can process text files that have embedded EOF records. The ~e string alone in a text record is used as an EOF record. Editors such as sed(1) or other standard utilities can process these files, but it is sometimes easier with the FFIO system.

On UNICOS and UNICOS/mk systems, the text layer is also useful in conjunction with the fdcp(1) command. The text layer provides a standard output format. Many forms of data that are not considered foreign are sometimes encountered in a heterogeneous computing environment. If a record format can be described with an FFIO specification, it can usually be converted to text format by using the following script:

OTHERSPEC=$1
INFILE=$2
OUTFILE=$3
assign -F ${OTHERSPEC} ${INFILE}
assign -F text ${OUTFILE}
fdcp ${INFILE} ${OUTFILE}

Use the fdcp command to copy files while converting record blocking.

The simplest form of data file format is the simple binary stream or unblocked data. It contains no record marks, file marks, or control words. This is usually the fastest way to move large amounts of data, because it involves a minimal amount of CPU and system overhead.

The FFIO package provides several layers designed specifically to handle this binary stream of data. These layers are syscall, sds, and mr. These layers behave the same from the user's perspective; they only use different system resources. The unblocked binary stream is usually used for unformatted data transfer. It is not usually useful for text files or when record boundaries or backspace operations are required. The complete burden is placed on the application to know the format of the file and the structure and type of the data contained in it.

This lack of structure also allows flexibility; for example, a file declared with one of these layers can be manipulated as a direct-access file with any desired record length.

In this context, fdcp can be called to do the equivalent of the cp(1) command only if the input file is a binary stream and to remove blocking information only if the output file is a binary stream.

The most common use for fixed-length record files is for Fortran direct access. Both unformatted and formatted direct-access files use a form of fixed-length records. The simplest way to handle these files with the FFIO system is with binary stream layers, such as system, syscall, cache, cachea, (all available on UNICOS and UNICOS/mk systems and IRIX systems) and sds, and mr (available only on UNICOS and UNICOS/mk systems). These layers allow any requested pattern of access and also work with direct-access files. The syscall and system layers, however, are unbuffered and do not give optimal performance for small records.

The FFIO system also directly supports some fixed-length record formats.

The COS blocking format is the default file structure for all Fortran sequential unformatted files on UNICOS and UNICOS/mk systems, except tape files. The cos layer is provided to handle these files. It provides for COS blocked files on disk and on magnetic tape and it supports multifile COS blocked datasets.

The cos layer must be specified for COS blocked files. If COS is not the default file structure, or if you specify another layer, such as sds, you may have to specify a cos layer to get COS blocking.

In the FFIO system, buffering is the responsibility of the individual layers; therefore, you must understand the individual layers in order to control the use and size of buffers.

The cos layer has high payoff potential to the user who wants to extract top performance by manipulating buffer sizes. As the following example shows, the cos layer accepts a buffer size as the first numeric parameter:

assign -F cos:42 u:1

The preceding example declares a working buffer size for the cos layer of forty-two 4096-byte blocks. This is an excellent size for a file that resides on a DD-49 disk drive because a track on a DD-49 disk drive is comprised of forty-two 4096-byte blocks (sectors).

If the buffer is sufficiently large, the cos layer also lets you keep an entire file in the buffer and avoid almost all I/O operations.

I/O optimization usually consists of reducing overhead. One part of the overhead in doing I/O is the CPU time spent in record blocking. For many files in many programs, this blocking is unnecessary. If this is the case, the FFIO system can be used to deselect record blocking and thus obtain appropriate performance advantages.

The following layers offer unblocked data transfer:

You can use any of these layers alone for any file that does not require the existence of record boundaries. This includes any applications that are written in C that require a byte stream file.

The syscall layer offers a simple direct system interface with a minimum of system and library overhead. If requests are larger than approximately 32 Kbytes, this method can be appropriate, especially if the requests are a uniform multiple of 4096 bytes.

The other layers are discussed in the following sections.

The bufa layer and cachea layer permits efficient file processing. Both layers provide library-managed asynchronous buffering, and the cachea layer allows recently accessed parts of a file to be cached either in main memory or in a secondary data segment.

The number of buffers and the size of each buffer is tunable. In the bufa:bs:nbufs or cachea:bs:nbufs FFIO specifications, the bs argument specifies the size in 4096-byte blocks of each buffer. The default on UNICOS systems and UNICOS/mk systems depends on the st_oblksize field returned from a stat(2) system call of the file; if this return value is 0, the default is 489 for ER90 files and 8 for all other files. The nbufs argument specifies the number of buffers to use.

The sds layer is not available on UNICOS/mk systems or on IRIX systems. It is only available on UNICOS systems.

The sds layer lets you use the secondary data segment (SDS) feature as an I/O device for almost any file. SDS is one use of the solid-state storage device (SSD). SDS as a device is described in the UNICOS File Formats and Special Files Reference Manual. If SDS is available, the sds layer can yield very high performance. The sds transfer rate can approach 2 Gbit/s.

Used in combination with the other layers, COS blocked files, text files, and direct-access files can reside in SDS without recoding. This can provide excellent performance for any file or part of a file that can reside in SDS.

The sds layer offers the capability to declare a file to be SDS resident. It features both scratch and save mode, and it performs overflow to the next lower layer (usually disk) automatically. You can declare that a file should reside in SDS to the extent possible. The simplest specification is assign -F sds fort.1.

This specification assumes default values for all options on the sds layer. By default, the sds layer is in save mode, which makes the SDS appear like an ordinary file. Because save is the assumed mode, any existing file is loaded into SDS when the file is opened. When the file is closed, the data is written back to the disk if the data was changed.

The sds layer overflows if necessary. Data that does not fit in the SDS space overflows to the next lower-level layer. This happens regardless of the reason for insufficient SDS space. For example, if you are not validated to use SDS, all of the files that are declared to be SDS-resident immediately overflow. In the previous assign(1) example, the overflow goes to disk file fort.1. The part of the file that fits in SDS remains there until the file is closed, but the overflowed portion resides on disk.

The assign -F command specifies the entire set of processing steps that are performed when I/O is requested. You can use other layers in conjunction with the sds layer to produce the desired file structures.

In the previous example, no specification exists for blocking on the file. Therefore, the resulting file structure is identical to the following:

assign -s u fort.1

This is also identical to the following:

assign -F syscall fort.1

If a file is COS blocked, a specification must be used that handles block and record control words. The following three examples produce identical files:

assign -s cos fort.1
assign -F cos fort.1
assign -F cos,sds fort.1

If the file is read or written more than once, adding sds to the assign command provides speed.

If SDS space is unlimited, almost any unformatted sequential file referenced from Fortran I/O can be declared by using the following command:

assign -F cos,sds unf_seq

Any formatted sequential file could be declared by using the following command:

assign -F text,sds fmt_seq

Record blocking is not required for unformatted direct-access files; therefore, any unformatted direct-access file can be declared by using the following command:

assign -F sds fort.1

In many cases, the cos specification is not necessary, but that decision must be made based on the specifics of the particular file and program.

All SDS space that the sds layer uses is obtained from the sdsalloc(3) library routines. Parameters, environment variables, and rules that pertain to these routines are fully applicable to this I/O technique.

For information about possible fragmentation with SDS, see the ldcache(8) man page.

“Sample Programs for UNICOS Systems”, contains several sds layer examples.

The mr layer lets you use main memory as an I/O device for many files. Used in combination with the other layers, COS blocked files, text files, and direct-access files can all reside in memory without recoding. This can result in excellent performance for any file or part of a file that can reside in memory.

If the file is small enough to fit in memory and is traversed many times, the wall-clock time can be reduced dramatically by using the mr layer to keep the file entirely in memory.

The mr layer lets you declare that a file is memory resident. It features both scratch and save mode, and it performs overflow to the next lower layer (usually disk) automatically.

Memory-resident files can run either in interactive or batch mode. The format for the mr layer on the assign(1) command is as follows:

assign -F mr.
savscr.ovfopt:min:max:incr

The assign -F command specifies the entire set of processing steps that are performed when I/O is requested. If the mr layer is specified alone, the resulting file structure is identical to the following:

assign -s unblocked fort.1

If a file is COS blocked, you must specify the handling of block and record control words as in the following example:

assign -s cos fort.1

The previous assign specification is identical to both of the following:

assign -F cos fort.1
assign -F cos,mr fort.1

“Sample Programs for UNICOS Systems”, contains several mr program examples.

The cache layer permits efficient file processing for repeated access to one or more regions of a file. It is a library-managed buffer cache that contains a tunable number of pages of tunable size.

To specify the cache layer, use the following option:

assign -F cache[:[bs][:[nbufs]]]

The bs argument specifies the size in 4096-byte blocks of each cache page; the default is 8. The nbufs argument specifies the number of cache pages to use. The default is 4. You can achieve improved I/O performance by using one or more of the following strategies:

If the number of regions accessed within a file is known, the number of cache pages can be chosen first. To determine the cache page size, divide the amount of memory to be used by the number of cache pages. For example, suppose a program uses direct access to read 10 vectors from a file and then writes the sum to a different file:

integer VECTSIZE, NUMCHUNKS, CHUNKSIZE
parameter(VECTSIZE=1000*512)
parameter(NUMCHUNKS=100)
parameter(CHUNKSIZE=VECTSIZE/HUMCHUNKS)
read a(CHUNKSIZE), sum(CHUNKSIZE)
open(11,access='direct',recl=CHUNKSIZE*8)
call asnunit (2,'-s unblocked',ier)
open (2,form='unformatted')
do i = 1,NUMCHUNKS
  sum = 0.0
  do j = 1,10
    read(11,rec=(j-1)*NUMCHUNKS+i)a
    sum=sum+a
  enddo
  write(2) sum
enddo
end

If 4 Mbytes of memory are allocated for buffers for unit 11, 10 cache pages should be used, each of the following size:

4MB/10 = 40000 bytes = 97 blocks

Make the buffer size an even multiple of the record length of 40960 bytes by rounding it up to 100 blocks (= 40960 bytes), then use the following assign command:

assign -F cache:100:10 u:11