Technical Q&A DV34

Secondary Interrupts on the Page Fault Path

Q: I'd like to use secondary interrupts in my SCSI Interface Module (SIM) but doing so causes strange deadlocks. Is it legal to use secondary interrupts in a SIM?

A: In general, using secondary interrupts on the page fault path is not legal. There are, however, circumstances under which it works. As with most of my Q&A, this takes some explanation.

The two common reasons for using secondary interrupts in a device driver are:

1. Reducing interrupt latency -- By deferring complex work to a secondary interrupt, you can reduce the amount of time your driver spends at hardware interrupt time, and thereby reduce the overall interrupt latency of the system.

2. Concurrency guarantee -- Secondary interrupts are guaranteed to be serialized; only one secondary interrupt handler can be running at any point in time. This is a very useful concurrency control mechanism. If you always access global data structures from your secondary interrupt handler, you can be assured that only one thread of execution is reading or writing those global data structures at a time.

However, secondary interrupts can cause problems for devices on the page fault path. Imagine the following scenario:

1. Your SIM takes a hardware interrupt, schedules a secondary interrupt (SIH A), and then returns from its hardware interrupt handler. For your SIM to make forward progress, its secondary interrupt handler must execute to completion.

2. The interrupt systems notices that it is returning to interrupt level 0 and begins processing secondary interrupts. It enables interrupts and starts running secondary interrupt handlers.

3. While SIH A is running, a Time Manager interrupt occurs. The timer task executes, schedules a deferred task, and then returns.

4. The interrupt system notices that it is returning from the Time Manager interrupt to interrupt level 0. There are deferred tasks to run, so it enables interrupts and starts running deferred tasks.

5. One of the deferred tasks causes a page fault.

6. The Virtual Memory Manager intercepts the page fault, and makes a synchronous request to the disk device driver to read the page contents from the backing store.

7. The disk device driver calls SCSI Manager to execute a SCSI command, which in turn calls your SIM. Your SIM starts the command and returns, expecting a hardware interrupt to complete the command.

8. Your SIM's hardware interrupt fires and schedules a secondary interrupt (SIH B) to complete the command.

The system is now deadlocked. The SIH B cannot run because secondary interrupt handlers must be single-threaded and SIH A is already running. But the SIH A cannot run because it has been interrupted by a page fault, which is sitting in a synchronous wait loop waiting for SIH B to run.

The solution is to not use secondary interrupt handlers in your SIM, or any software entity on the page fault path.

Newer versions of Mac OS provide another workaround for this problem. On such systems, the OS detects these potential deadlock cases (waiting for a synchronous device request with an interrupt mask of 0 and with secondary interrupt handlers queued) and calls your SIMInterruptPoll routine. Your interrupt poll routine can detect that it has queued a secondary interrupt that has not yet run, and perform the appropriate action directly. Typically this involves calling your secondary interrupt handler routine directly.

This approach has a number of important caveats:

1. It does not work on older systems. You can check for the presence of secondary interrupt polling using the code in Listing 1. You should find it available on Mac OS 8.5 and later.

2. Secondary interrupt polling may undermine the concurrency guarantees provided by secondary interrupts. In the above example, look carefully at what happens during the deadlock recovery process. The system calls your interrupt poll routine and your interrupt poll routine calls SIH B directly. At this point, you have two threads of execution inside secondary interrupt handlers, SIH A and SIH B. This could cause problems if you designed your SIM to rely on the concurrency guarantees provided by secondary interrupts. If you want to use secondary interrupts in your SIM, you must take this potential reentrancy into account.

3. Secondary interrupt polling is only useful for devices with an interrupt poll routine, such as SIMs and AIMs. Other native drivers, such as disk drivers, do not have an interrupt poll routine and cannot use secondary interrupts if they are on the page fault path.

Because of these caveats, it may just be easier to do everything in your SIMs hardware interrupt handler. SIMs are mostly I/O bound, so the interrupt latency increase of doing everything in your hardware interrupt handler is minimal.

You can detect whether secondary interrupt polling is implemented using Gestalt, as shown in Listing 1.

Listing 1. Determining whether secondary interrupt polling is available

static Boolean HasSecondaryInterruptPolling(void)
{
    UInt32 response;

    return (Gestalt(gestaltSCSI, (SInt32 *) &response) == noErr) &&
            ((response & (1 << gestaltSCSIPollSIH)) != 0);
}

In summary:

• Prior to Mac OS 8.5, you must never use secondary interrupts on the page fault path.

• With Mac OS 8.5 and beyond, SIMs and AIMs may use secondary interrupts, but they must take special action in their interrupt poll routine.

• Despite the difficulties using secondary interrupts in a SIM, DTS recommends that SIM developers seriously consider adopting this technique. The primary advantage, lower system interrupt latency, is a significant benefit to real-time applications such as QuickTime.

For more information about this mind-manglingly complex part of the Mac OS I/O subsystem, see:

• Designing PCI Cards and Drivers for Power Macintosh Computers

• Inside Macintosh: Devices (especially the SCSI Manager 4.3 chapter)

• Technote 1094 Virtual Memory Application Compatibility

• Technote 1104 Interrupt-Safe Routines

• Technote 1137 Disabling Interrupts on the Traditional Mac OS

• DTS Q&A DV 32 PrepareMemoryForIO and Execution Levels