Directives placed in a file prior to program code are called global directives. The compiler interprets them as if they appeared at the top of each program unit in the file.

Directives appearing anywhere else in the file apply only until the end of the current program unit. The compiler resets the value of the directive to the global value at the start of the next program unit.

To continue the loop-level multiprocessing directives onto another line, use !$& as the first characters in the continued line(s). For example:

!$DOACROSS share(ALPHA, BETA, GAMMA, DELTA,
!$&  EPSILON, OMEGA), LASTLOCAL(I, J, K, L, M, N),
!$&  LOCAL(XXX1, XXX2, XXX3, XXX4, XXX5, XXX6, XXX7,
!$&  XXX8, XXX9)

To continue the PCF directives onto another line, begin the continued line with the characters !$PAR&.

The basis for the loop-level multiprocessing directives is the DOACROSS directive. This directive indicates to the compiler that it should run iterations of the subsequent DO loop in parallel. This directive must appear directly before the loop that is to be operated on, and it remains in effect for that loop only.

The format of this directive is as follows:

!$DOACROSS [clause[, clause] ...]

The sections that follow describe the DOACROSS directive clauses.

Appendix B, “Debugging and Profiling Multiprocessed Programs”, contains information on debugging when DOACROSS directives are used.

---

Note: The Fortran compiler does not support direct nesting of DOACROSS loops. For example, the following is illegal and generates a compilation error: !$DOACROSS LOCAL(I) DO I = 1, N !$DOACROSS LOCAL(J) DO J = 1, N A(I,J) = B(I,J) END DO END DO However, to simplify separate compilation, a different form of nesting is allowed. A routine that uses !$DOACROSS can be called from within a multiprocessed region. This can be useful if a single routine is called from several different places: sometimes from within a multiprocessed region, sometimes not. Nesting does not increase the parallelism. When the first !$DOACROSS loop is encountered, that loop is run in parallel. While in the parallel loop, if a call is made to a routine that itself has a !$DOACROSS, the subsequent loop is executed serially.

---

BLOCKED (int_expr)
CHUNK = int_expr

IF (logical_expr)

LASTLOCAL var[ , var ... ]
LOCAL var[ , var ... ]
SHARED var[ , var ... ]

MP_SCHEDTYPE = mode

REDUCTION var[ , var]...

!$DOACROSS LOCAL(I), REDUCTION(A(1))
      DO I = 2,N
         A(1) = A(1) + A(I)
      END DO

The CHUNK directive breaks work up into pieces. Like the MP_SCHEDTYPE directive, the CHUNK directive acts as an implicit clause, in this case a CHUNK clause, for all DOACROSS directives in the scope. The CHUNK directive is in effect from the place it occurs in the source until another corresponding directive is encountered or the end of the procedure is reached.

The format of this directive is as follows:

!$CHUNK=int_expr

The CHUNK clause to the DOACROSS directive also divides work. For more information, see “BLOCKED and CHUNK Clauses”.

The MP_SCHEDTYPE directive affects the way the compiler schedules work among the participating tasks in a loop. Like the CHUNK directive, the MP_SCHEDTYPE directive acts as an implicit clause, in this case an MP_SCHEDTYPE clause, for all DOACROSS directives in the scope. The MP_SCHEDTYPE directive is in effect from the place it occurs in the source until another corresponding directive is encountered or the end of the procedure is reached.

The MP_SCHEDTYPE directive specifies the scheduling type to be used for subsequent !$DOACROSS directives that are specified without an explicit scheduling type.

The format of this directive is as follows:

!$MP_SCHEDTYPE mode

The MP_SCHEDTYPE clause to the DOACROSS directive also divides work. For more information, see “MP_SCHEDTYPE Clause”.

The !$ directive, which is really only a prefix, precedes code that should be recognized only when multiprocessing is enabled. Multiprocessing is enabled when either -pfa or -mp is specified on the f90(1) command line.

These directive lines are considered comment lines except when multiprocessing. A line beginning with !$ is treated as a conditionally compiled Fortran statement.

The format of this directive is as follows:

!$ statement

The following code demonstrates the use of the !$ directive:

!$    PRINT 10
!$ 10 FORMAT('BEGIN MULTIPROCESSED LOOP')
!$DOACROSS LOCAL(I), SHARED(A,B)
        DO I = 1, 100
           CALL COMPUTE(A, B, I)
        END DO

This section contains examples of DOACROSS directives.

Example 1. Simple DOACROSS directive. Consider the following code fragment:

     DO 10 I = 1, 100
        A(I) = B(I)
10   CONTINUE

By inserting a directive, it can be multiprocessed:

!$DOACROSS LOCAL(I), SHARED(A, B)
     DO 10 I = 1, 100
        A(I) = B(I)
10   CONTINUE

Here, the defaults are sufficient provided that A and B are mentioned in a nonparallel region or in another SHARED list. The following code will then work:

!$DOACROSS
     DO 10 I = 1, 100
        A(I) = B(I)
10   CONTINUE

Example 2. A DOACROSS directive with a LOCAL clause. Consider the following code fragment:

     DO 10 I = 1, N
        X = SQRT(A(I))
        B(I) = X*C(I) + X*D(I)
10   CONTINUE

The following code shows this fragment rewritten for multiprocessing using explicit clauses:

!$DOACROSS LOCAL(I, X), SHARED(A, B, C, D, N)
     DO 10 I = 1, N
        X = SQRT(A(I))
        B(I) = X*C(I) + X*D(I)
10   CONTINUE

The following code shows the fragment rewritten for multiprocessing using the default settings:

!$DOACROSS LOCAL(X)
     DO 10 I = 1, N
        X = SQRT(A(I))
        B(I) = X*C(I) + X*D(I)
10   CONTINUE

Example 3. A DOACROSS directive with a LASTLOCAL clause. Consider the following code fragment:

     DO 10 I = M, K, N
        X = D(I)**2
        Y = X + X
        DO 20 J = I, MAX
           A(I,J) = A(I,J) + B(I,J) * C(I,J) * X + Y
20   CONTINUE
10   CONTINUE
     PRINT*, I, X

In this example, the final values of I and X are needed after the loop completes. A correct directive is shown in the following:

!$DOACROSS LOCAL(Y,J), LASTLOCAL(I,X),
!$& SHARED(M,K,N,ITOP,A,B,C,D)
     DO 10 I = M, K, N
         X = D(I)**2
         Y = X + X
         DO 20 J = I, ITOP
             A(I,J) = A(I,J) + B(I,J) * C(I,J) *X + Y
20       CONTINUE
10   CONTINUE
     PRINT*, I, X

You can also use the defaults:

!$DOACROSS LOCAL(Y,J), LASTLOCAL(X)
     DO 10 I = M, K, N
         X = D(I)**2
         Y = X + X
         DO 20 J = I, MAX
             A(I,J) = A(I,J) + B(I,J) * C(I,J) *X + Y
20       CONTINUE
10   CONTINUE
     PRINT*, I, X

In the preceding code example, I is a loop index variable for the DOACROSS loop, so it is LASTLOCAL by default. Even though J is a loop index variable, it is not the loop index of the loop being multiprocessed and has no special status. If it is not declared, it is assigned the default value of SHARED, which produces an incorrect answer.

The BARRIER directive ensures that each process waits until all processes reach the barrier before proceeding.

This directive has the following format:

!$PAR BARRIER

The CRITICAL SECTION and END CRITICAL SECTION directives ensure that the enclosed block of code is executed by only one process (thread) at a time. Another process attempting to gain entry to the critical section must wait until the previous process has exited. Threads do not synchronize at the bottom of a critical section.

The critical section construct can appear anywhere in a program, including inside and outside a parallel region and within a DOACROSS loop.

These directives have the following format:

!$PAR CRITICAL SECTION [ (lock_variable) ]
!$PAR END CRITICAL SECTION

The PARALLEL and END PARALLEL directives enclose a parallel region that includes work-sharing constructs and critical sections. It signifies the boundary within which slave threads execute. A user program can contain any number of parallel regions.

These directives have the following format:

!$PAR PARALLEL [clause[,clause]...]
!$PAR END PARALLEL

The preferred form of the directive has no commas between the clauses.

In the following code, all threads enter the parallel region and call routine FOO:

          SUBROUTINE EX1(INDEX)
          INTEGER I
!$PAR PARALLEL LOCAL(I)
          I = MP_MY_THREADNUM()
          CALL FOO(I)
!$PAR END PARALLEL
          END

The PARALLEL DO directive indicates that the iterations of the subsequent DO loop should be executed by different processes. This directive produces the same effect as the DOACROSS directive, and it is conceptually the same as a parallel region containing exactly one PDO construct and no other code. Each thread inside the enclosing parallel region executes separate iterations of the loop within the parallel DO construct. This directive must not appear within a parallel region.

This directive has the following format:

!$PAR PARALLELDO [clause[,clause] ...]

The PDO and END PDO directives surround a loop and indicate that the iterations of the enclosed loop should be executed by different processes. These directives must be enclosed within a parallel region delimited by PARALLEL and END PARALLEL directives.

Within a parallel region, each thread inside the region executes a separate iteration of a loop within a PDO construct.

These directives have the following format:

!$PAR PDO [clause[, clause]...]

[!$PAR END PDO [NOWAIT]]

It is legal to declare a data item as LOCAL in a PDO directive even if it was declared as SHARED in the enclosing parallel region.

The END PDO directive is optional. If specified, this directive must appear immediately after the end of the DO loop. The optional NOWAIT clause specifies that each process should proceed directly to the code immediately following the directive. If you do not specify NOWAIT, the processes wait until all have reached the directive before proceeding.

---

Note: Localized ALLOCATABLE or POINTER arrays are not supported on the PDO directive.

---

The code in the following example is equivalent to a DOACROSS loop. In fact, the compiler recognizes this as a special case and generates the same (more efficient) code as for a DOACROSS directive.

          SUBROUTINE EX2(A,N)
          REAL A(N)
!$PAR PARALLEL LOCAL(I) SHARED(A)
!$PAR PDO
          DO I = 1, N
            A(I) = A(I) + 1.0
          END DO
!$PAR END PARALLEL
          END

The PSECTION[S] and END PSECTION[S] directives delimit a parallel section construct and distribute code blocks to processes. These directives have an effect that is similar to the Fortran 90 SELECT construct. Each block of code is parceled out in turn to a separate thread.

The SECTION directive indicates a starting line for an individual section within a parallel section.

These directives must be enclosed within a parallel region delimited by PARALLEL and END PARALLEL directives.

These directives have the following format:

!$PAR PSECTION[S][LOCAL var[, var] ...]
[!$PAR SECTION]
!$PAR END PSECTION[S][NOWAIT]

The optional NOWAIT clause specifies that each process should proceed directly to the code immediately following the directive. If you do not specify NOWAIT, the processes wait until all have reached the END PSECTION directive before proceeding.

Parallel sections can contain critical section constructs, but they cannot contain any of the following types of constructs:

Each code block is executed in parallel (depending on the number of processes available). The code blocks are assigned to threads one at a time, in the order specified. Each code block is executed by only one thread.

For example, consider the following code:

          SUBROUTINE EX3(A,N1,B,N2,C,N3)
          REAL A(N1), B(N2), C(N3)
!$PAR PARALLEL LOCAL(I) SHARED(A,B,C)
!$PAR PSECTIONS
!$PAR SECTION
          DO I = 1, N1
            A(I) = 0.0
          END DO
!$PAR SECTION
          DO I = 1, N2
            B(I) = 0.5
          END DO
!$PAR SECTION
          CALL NORMALIZE(C,N3)
          DO I = 1, N3
            C(I) = C(I) + 1.0
          END DO
!$PAR END PSECTION
!$PAR END PARALLEL
          END

The first thread to enter the parallel section construct executes the first block, the second thread executes the second block, and so on. This example has only three sections, so if more than three threads are in the parallel region, the fourth and higher threads wait at the !$PAR END PSECTION directive until all threads are finished. If the parallel region is being executed by only two threads, whichever thread finishes its block first continues and executes the remaining block.

This example uses DO loops, but a parallel section can be any arbitrary block of code. Parallel constructs have significant overhead. Make sure the amount of work performed is enough to outweigh the extra overhead.

The sections within a parallel section construct are assigned to threads one at a time, from the top down. There is no other implied ordering to the operations within the sections. In particular, a later section cannot depend on the results of an earlier section, unless some form of explicit synchronization is used. If there is such explicit synchronization, you must be sure that the lexical ordering of the blocks is a legal order of execution.

The SINGLEPROCESS and END SINGLEPROCESS directives enclose a block of code that should be executed by only one process. These directives must be enclosed within a parallel region delimited by PARALLEL and ENDPARALLEL directives.

These directives have the following format:

!$PAR SINGLEPROCESS  [LOCAL var[, var] ...]
!$PAR END SINGLEPROCESS [NOWAIT]

The optional NOWAIT clause specifies that each process should proceed directly to the code immediately following the directive. If you do not specify NOWAIT, the processes wait until all have reached the END SINGLEPROCESS directive before proceeding.

This construct is semantically equivalent to a parallel section construct with only one section. The single process construct provides a more descriptive syntax.

The first thread to reach a single process section executes the code in that block. All other threads wait at the end of the block until the code has been executed.

Notice the use of the repetition of the IF test in the first parallel loop:

             IF (A(I,J) .GT. CUR_MAX) THEN
!$PAR CRITICAL SECTION
               IF (A(I,J) .GT. CUR_MAX) THEN

This practice is called test&test&set. It is a multiprocessing optimization. The following straightforward code segment is incorrect:

          DO I = 1, N
            IF (A(I,J) .GT. CUR_MAX) THEN
!$PAR CRITICAL SECTION
                  INDEX_X = I
                  INDEX_Y = J
                  CUR_MAX = A(I,J)
!$PAR END CRITICAL SECTION
            ENDIF
          ENDDO

Because many threads execute the loop in parallel, there is no guarantee that once inside the critical section, CUR_MAX still has the same value it did in the IF test outside the critical section (some other thread may have updated it). In particular, CUR_MAX may now have a value that is larger than A(I,J). Therefore, the critical section must be locked before testing the value of CUR_MAX. Changing the previous code into the following code works correctly, but suffers from a serious performance penalty: the critical section lock must be acquired and released (an expensive operation) for each element of the array:

            DO I = 1, N
!$PAR CRITICAL SECTION
                IF (A(I,J) .GT. CUR_MAX) THEN
                  INDEX_X = I
                  INDEX_Y = J
                  CUR_MAX = A(I,J)
                ENDIF
!$PAR END CRITICAL SECTION
         ENDDO

Because the values are rarely updated, this process involves a lot of wasted effort. It is almost certainly slower than just executing the loop serially.

Combining the two methods, as in the original example, produces code that is both fast and correct. If the IF test outside of the critical section fails, you can be certain that the values will not be updated and can proceed. You can expect that the outside IF test will account for the majority of cases. If the outer IF test passes, then the values might be updated, but you cannot always be certain. To ensure correctness, you must perform the test again after acquiring the critical section lock.

You can prefix one of the two identical IF tests with !$ to reduce overhead in the non-multiprocessed case.

Lastly, note the difference between the single process and critical section constructs. If several processes arrive at a critical section construct, they execute the code one at a time. However, they will all execute the code. If several processes arrive at a single process construct, only one process executes the code. The other processes bypass the code and wait at the end of the construct for the chosen process to finish.

The three work-sharing constructs, PDO, PSECTION, and SINGLEPROCESS, must be executed by all the threads executing in the parallel region or by none of the threads. The following is illegal:

      ...
!$PAR PARALLEL
          IF (MP_MY_THREADNUM() .GT. 5) THEN
!$PAR SINGLE PROCESS
              MANY_PROCESSES = .TRUE.
!$PAR END SINGLE PROCESS
          ENDIF
           ...

The preceding code cannot run successfully when more than six processors are used. One or more processes will be stuck at the !$PAR ENDSINGLEPROCESS directive waiting for all the threads to arrive. Because some of the threads never took the appropriate branch, they will never encounter the construct. However, the following kind of simple looping is supported:

       ...
!$PAR PARALLEL LOCAL(I,J) SHARED(A)
          DO I= 1,N
!$PAR PDO
            DO J = 2,N
       ...

The distinction here is that all of the threads encounter the work-sharing construct. They all complete it, and they all loop around and encounter it again.

This restriction does not apply to the critical section construct, which operates on one thread at a time without regard to any other threads.

Parallel regions cannot be nested inside of other parallel regions, nor can work-sharing constructs be nested. However, as an aid to writing library code, you can call an external routine that contains a parallel region even from within a parallel region. In this case, only the first region is actually run in parallel. Therefore, you can create a parallelized routine without accounting for whether it will be called from within an already parallelized routine.