The IO4 hardware problem involves a subtle interaction between two IO4 boards when they perform DMA to the identical cache line of memory. If one IO4 performs a partial update of a 128-byte cache line, and another IO4 accesses the same cache line for DMA between partial updates, the second IO4 can suffer a change to a different, unrelated cache line in its on-board cache. That modified cache line may not be used again, but if it is used, invalid data can be transferred.

It is important to note that the IO4 problem is specific to interactions between multiple IO4 boards. It does not affect memory interactions between CPUs, or between CPUs and IO4s. Cache coherency is properly maintained in these cases.

An unusual coincidence is required to trigger the modification of the IO4 cache memory; then the modified cache line must be used for output before the error has any effect. The right combinations are sufficiently rare that many systems with multiple IO4 boards have never encountered it. For example, the problem has occurred on a system that acted as a network gateway between ATM and FDDI network, with ATM and FDDI adapters on different IO4 boards; and it has been seen when “raw” (not filesystem) disk input was copied to a tape on a different IO4.

A kernel-level device driver for a device that uses DMA in a Challenge-architecture system probably needs to make one change to guard against the IO4 problem.

In order to preclude any chance of data corruption, drivers that are affected must ensure that the IO4 flushes its cache following any DMA write to memory (input from a device). This is done by calling a new kernel function, io4_flush_cache(), in the interrupt routine immediately following completion of any DMA.

The prototype of io4_flush_cache() is

int io4_flush_cache(caddr_t any_PIO_mapaddr);

The argument to the function is any value returned by pio_mapaddr() that is related to the device doing the DMA. The kernel uses this address to locate the IO4 involved. The returned value is 0 when the operation is successful (or was not needed). It is 1 when the argument is not a valid address returned by pio_mapaddr().

The function should be called immediately after the completion of a DMA input to memory. Typically the device produces an interrupt at the end of a DMA, and the function can be called from the interrupt handler. However, some devices can complete more than one DMA transaction per interrupt, and io4_flush_cache() should be called when each DMA completes. Put another way, if it is possible that a data transfer completed after an interrupt, then the driver should call io4_flush_cache() before marking the transaction as complete.

The io4_flush_cache() function does nothing and returns immediately in a machine that has only one IO4 board, and in a machine in which all IO4 boards have the hardware fix.

The kernel's VME interrupt handler calls io4_flush_cache() once on each VME interrupt. Thus a VME device driver only needs to call io4_flush_cache() in the event that it handles the completion of more than DMA transaction per interrupt. For example, a VME-based network driver that handles multiple packets per interrupt should call io4_flush_cache() once for each packet that completes.

Since this problem only affects Challenge/Onyx systems (including Power Challenge, Power Onyx, and Power Challenge R10000), the software fix can and should be conditionally compiled on the compiler variable EVEREST, which is set by /var/sysgen/Makefile.kernio for the affected machines (see “Using /var/sysgen/Makefile.kernio” in Chapter 9).

The following is a skeletal example of fix code for a hypothetical driver:

#ifdef EVEREST
extern void io4_flush_cache(void* anyPIOMapAddr);
#endif
caddr_t some_PIO_map_addr;
hypothetical_edtinit(...)
{
...
   some_PIO_map_addr = pio_mapaddr(my_piomap, some_dev_addr)
...
}
hypothetical_intr(...)
{
...
#ifdef EVEREST
   io4_flush_cache(some_PIO_map_addr);
#endif
...
}

For another example, see the code of the example VME device driver under “Sample VME Device Driver” in Chapter 13.