Technical Note TN2118

Kernel Core Dumps

This technote explains how you can enable remote kernel core dumps on Mac OS X 10.3 and later. Kernel core dumps allow you to examine the state of the kernel after a kernel panic or hang. You can use this to debug a kernel problem when circumstances prevent you from using the two-machine kernel debugger. This technote is targeted at two distinct audiences. Kernel extension developers can use this information to debug kernel panics or hangs that they can't reproduce locally. In addition, system administrators can use this information to record details about which machines are panicking and why, and to perform offline debugging on high-availability systems.

Introduction Configuring the Server Creating a Core Dump Directory Activating the Kernel Dump Server Process Configuring the Kernel Core Dump Server for Mac OS X prior to 10.5 Configuring the Kernel Core Dump Server for Mac OS X 10.4.x Configuring the Kernel Core Dump Server for Mac OS X 10.3.x Configure xinetd Signal xinetd Confirm Configuration Configuring a Client Debug Flags in Depth Testing Your Configuration Triggering a Kernel Panic with DTrace Triggering a Kernel Panic with the Instant Panic Kernel Extension The Kernel Panic Core Dump File Debugging with Kernel Core Dumps Kernel Core Dump Options Configuring the Client and Server UDP Port Assignment Configuring the Client UDP Port Assignment Configuring the Server UDP Port Assignment Configuring the Client Router Usage Specifying the Ethernet Interface Conclusion Further Reading Downloadables Document Revision History

Introduction

The Mac OS X kernel should never panic because, when it does, it seriously inconveniences the user. Thus, a kernel panic is always the result of a bug, either in Apple's code or in the code of some third party kernel extension. Such bugs need to be investigated and resolved.

There are a number of circumstances in which the ability to capture a kernel core dump is useful.

• When you're writing a kernel extension and you encounter a kernel panic, you can usually debug the problem with the two-machine kernel debugger. However, there are circumstances where this isn't possible. For example, if a tester or end user reports a problem that you can't reproduce on your desktop—either because it happens infrequently or because it only happens with obscure hardware or in a non-standard environment—you will not be able to debug using the standard tools. In these circumstances it's helpful if you can capture a core dump of the panicked kernel and debug using that core dump.

• If you manage a large group of Macintosh computers, you might want to monitor which computers are panicking and why. You can use this information to determine how frequently kernel panics occurs, whether there are any common symptoms, and, most importantly, whether any third party kernel extensions are involved.

• Finally, if you manage a high-availability Macintosh server and you have problems with the server panicking, you can capture a kernel core dump, immediately restart the server, and then debug the problem offline.

To assist you in these situations, Apple provides a remote kernel core dump facility which has been present since Mac OS X 10.3 for PowerPC-based Macintosh systems, and since Mac OS X 10.4.7 for Intel-based Macintosh systems. You can configure a Mac OS X computer so that, when the machine panics, it transmits a core dump of the kernel to a server via TCP/IP. The core dump server is a daemon that collects the kernel core dump from the client and writes it to disk. You can then analyze the core dump using a variety of tools, most notably GDB.

There are two components of core dumping, the server side and client side. The server side is the destination of the core dump file, the client side enables the kernel on the machine which panics, to dump its core to the server.

This technical note focuses on describing the core dump process across an internet connection. A FireWire connection can also be used to transmit the core dump. FireWire is a useful alternative when the kernel panic on the client system involves the built-in Ethernet driver, or some other network code such that the core dump cannot be sent across an Ethernet connection.

The Read Me file in the FireWire SDK for Mac OS X describes the setup process for using FireWire to transmit a core dump. You can access the FireWire SDK from the Apple Developer Hardware & Drivers - Downloads web page.

Once you succeeded in saving the core dump across the FireWire connection, you can learn more about analyzing the core dump by reading the section Debugging with Kernel Core Dumps.

This technical note provides information on the following tasks.

• Setting up a kernel core dump server.

• Configuring a client computer to dump its core to the core dump server when the client panics.

• Testing your setup.

• Understanding and debugging the resulting kernel core dumps.

• Understanding optional settings for kernel core dumps.

Back to Top

Configuring the Server

The first step in collecting kernel core dumps is to set up a kernel core dump server. To start you must choose a machine to act as the server, taking into account the following points.

• The kernel core dump server is not computationally expensive; it will run easily on any machine capable of running Mac OS X.

• Kernel core dumps are large, typically running to the order of 200-500+ MB (this varies based on the client's kernel map size, physical memory size, usage patterns, and so on). Your server should have a lot of free disk space.

• The server must have a static IP address.

• The server's IP address must be visible to all of the clients. You can't place the server behind a NAT or firewall unless your clients are also behind the same NAT or firewall.

To enable the kernel core dump server, execute the following steps:

• Create the core dump directory as described in Creating a Core Dump Directory.

• Activate the Kernel Dump Server Process as described in Activating the Kernel Dump Server Process.

• For Mac OS X 10.4, configure and activate the Kernel Dump Server Process as described in Configuring the Kernel Core Dump Server for Mac OS X 10.4.x.

• For Mac OS X 10.3, configure and activate the Kernel Dump Server Process as described in Configuring the Kernel Core Dump Server for Mac OS X 10.3.x.

Creating a Core Dump Directory

To start, you must create a directory into which the server writes core dumps. That directory must be writable by the program that's dumping the core. The easiest way to guarantee this is to make it writable by everyone. We recommend that you create this directory with the commands shown in Listing 1.

Listing 1: Creating the core dump directory.

server$ sudo mkdir /PanicDumps
Password:********
server$ sudo chown root:wheel /PanicDumps
server$ sudo chmod 1777 /PanicDumps

Activating the Kernel Dump Server Process

For Mac OS X 10.5 and greater, a disabled launchd property list file is present as part of the basic system installation. The com.apple.kdump.plist property list file assumes that you will use the /PanicDumps directory to store the kernel core dumps. If you want to use a different directory, create the directory using similar instructions as shown in Listing 1 then modify the /System/Library/LaunchDaemons/com.apple.kdump.plist property list file and update the ProgramArguments property with the full path to the desired directory.

To activate the server, use launchctl shown in Listing 2. This will update the property list file so that the core dump server will always startup automatically across system restarts.

Listing 2: Activating the core dump server.

server$ sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.kdumpd.plist
Password:********

You can verify that the core dump server is active by using the launchctl list command as shown in Listing 3.

Listing 3: Verifying the core dump server is active.

server$ sudo launchctl list | grep kdump
Password:********
 - 0     com.apple.kdumpd

There may be no PID associated with the com.apple.kdumpd launchd job label as the process may not be active just yet. The status 0 value indicates that the server is ready for on-demand use.

Once you have the server configured correctly, you can proceed on to the next step, Configuring a Client.

Back to Top

Configuring the Kernel Core Dump Server for Mac OS X prior to 10.5

If you have server machine with a version of Mac OS X prior to 10.5, there are different instructions to follow. For a server machine with Mac OS X 10.4, follow the instructions in the section, Configuring the Kernel Core Dump Server for Mac OS X 10.4.x. For a server machine with Mac OS X 10.3, follow the instructions in the section, Configuring the Kernel Core Dump Server for Mac OS X 10.3.x.

Configuring the Kernel Core Dump Server for Mac OS X 10.4.x

The procedure for setting up the Kernel Core Dump server under Mac OS X 10.4.x is similar to the procedure for Mac OS X 10.5. The only difference is that the launchd property list file is not installed by default, so you have to create it yourself. Create the launchd property list file shown in Listing 4. These instructions assume that you are using the /PanicDumps directory to store the kernel core dumps. If you use a different directory, you will have to substitute the full path to that directory in place of the ProgramArguments string property /PanicDumps.

Listing 4: Creating the Server launchd property list file

server$ sudo cat > /tmp/com.mycompany.kdump.plist
Password: ********
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>InitGroups</key>
    <true/>
    <key>Label</key>
    <string>com.mycompany.kdumpd</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/libexec/kdumpd</string>
        <string>/PanicDumps</string>
    </array>
    <key>Sockets</key>
    <dict>
        <key>Listener</key>
        <dict>
            <key>SockServiceName</key>
            <string>1069</string>
            <key>SockType</key>
            <string>dgram</string>
        </dict>
    </dict>
    <key>UserName</key>
    <string>nobody</string>
    <key>inetdCompatibility</key>
    <dict>
        <key>Wait</key>
        <true/>
    </dict>
</dict>
</plist>
^D
client$ ls -l /tmp/com.mycompany.kdump.plist
-rw-r--r-- 1 admin  wheel  831 Sep 10 15:25 /tmp/com.mycompany.kdump.plist
client$ sudo cp /tmp/com.mycompany.kdump.plist /Library/LaunchDaemons/
Password: ********
client$ ls -l /Library/LaunchDaemons/com.mycompany.kdump.plist
-rw-r--r-- 1 root  wheel  831 Sep 10 15:26 /Library/LaunchDaemons/com.mycompany.kdump.plist

Once you have the created the launchd property list, restart the system. The core dump server process should be active on restart. You can verify that the core dump server process is active by using the launchctl list command as shown in Listing 5.

Listing 5: Verifying the core dump server is active for Mac OS X 10.4.x

server$ sudo launchctl list | grep kdump
Password:********
com.mycompany.kdumpd

The presence of the com.mycompany.kdumpd launchd job label indicates that the core dump server is ready for on-demand use.

With the server configured, configure the client as described in Configuring a Client.

Back to Top

Configuring the Kernel Core Dump Server for Mac OS X 10.3.x

Follow these instructions to configure the Kernel Core Dump server for Mac OS X 10.3. These instructions assume that you are using the /PanicDumps directory to store the kernel core dumps. If you use a different directory, you will have to substitute the full path to that directory in place of /PanicDumps in subsequent steps.

Configure xinetd

Configure the extended Internet services daemon, xinetd, to run the server when a client connects. Copy the text from Listing 6 into a file called macosxkdump in the /etc/xinetd.d directory. Listing 7 shows one way to do this.

Listing 6: Contents of the 'macosxkdump' configuration file

service macosxkdump
{
    disable     = no
    type        = UNLISTED
    socket_type = dgram
    protocol    = udp
    port        = 1069
    user        = nobody
    groups      = yes
    server      = /usr/libexec/kdumpd
    server_args = /PanicDumps
    wait        = yes
}

Listing 7: Creating the 'macosxkdump' file

server$ cat > /tmp/macosxkdump
service macosxkdump
{
    disable     = no
    type        = UNLISTED
    socket_type = dgram
    protocol    = udp
    port        = 1069
    user        = nobody
    groups      = yes
    server      = /usr/libexec/kdumpd
    server_args = /PanicDumps
    wait        = yes
}
^D
server$ ls -l /tmp/macosxkdump
-rw-r--r-- 1 quinn  staff  278 14 Jul 15:25 /tmp/macosxkdump
server$ sudo cp /tmp/macosxkdump /etc/xinetd.d 
Password: ********
server$ ls -l /etc/xinetd.d/macosxkdump 
-rw-r--r-- 1 root  wheel  278 14 Jul 15:26 /etc/xinetd.d/macosxkdump

Signal xinetd

You must send the SIGHUP signal to the xinetd daemon for it to recognize your configuration changes. Listing 8 shows how to do this. When entering the first command, make sure to use backquotes (ASCII 96), not single quotes (ASCII 39).

Listing 8: Signalling xinetd

server$ sudo kill -HUP `cat /var/run/xinetd.pid`
Password: ********

Alternatively, you can simply reboot the machine, which also causes xinetd to recognize the new configuration.

Confirm Configuration

You can confirm that everything is configured correctly using one of two methods.

• If you look in the system log you will see text similar to that shown in Listing 9 indicating that xinetd has started a new service. You can view the system log manually (/var/log/system.log) or in the Console application.

• You can send a SIGUSR1 signal to xinetd to ask it to dump its current configuration. Listing 10 shows how to do this. The daemon appends the information to /var/run/xinetd.dump. You should see a record indicating that the macosxkdump service is active, similar to the one shown in Listing 11.

Listing 9: System log confirmation

Jul 14 15:40:57 localhost xinetd[349]: Starting reconfiguration
Jul 14 15:40:57 localhost xinetd[349]: readjusting service ssh
Jul 14 15:40:57 localhost xinetd[349]: Reconfigured: new=1 old=1 dropped=0 (services)

Listing 10: Signalling xinetd to dump its configuration

server$ sudo kill -USR1 `cat /var/run/xinetd.pid`
Password: ********

Listing 11: xinetd dump confirmation

Service = macosxkdump
    State = Active
    Service configuration: macosxkdump
        id = macosxkdump
        flags = IPv4
        type = UNLISTED
        socket_type = dgram
        Protocol (name,number) = (udp,17)
        port = 1069
        wait = yes
        user = -2
        Groups = yes
        PER_SOURCE = -1
        Bind = All addresses.
        Server = /usr/libexec/kdumpd
        Server argv = kdumpd /PanicDumps
        Only from: All sites
        No access: No blocked sites
        Logging to syslog. Facility = daemon, level = info
        Log_on_success flags = HOST PID HOST
        Log_on_failure flags = HOST
    running servers = 0
    retry servers = 0
    attempts = 0
    service fd = 6

Once you have the server configured, you can configure the client as described in Configuring a Client.

Back to Top

Configuring a Client

To enable kernel core dumps on a client machine, you must modify the boot-args NVRAM variable to include two arguments.

• Set the 0x0400 flag in the debug argument. Furthermore, there are a number of other useful flags for this argument which are described in detail below.

• Set the _panicd_ip argument to the IP address of the kernel core dump server. You must use an IPv4 address in dotted decimal notation; IPv6 and DNS addresses are not supported.

• If you are configuring a Mac OS X Server system as a core dump client, you must limit or turn off watchdogtimerd. On the Mac OS X Server system the watchdog timer reboots the machine after a panic occurs. You must limit or disable watchdogtimerd so that the server system will not reboot while the client is in the process of sending the core dump to the server.

  Note: On a Mac OS X Server system you can turn off watchdogtimerd using the Energy Saver preference pane Options tab, or from the command line. Alternatively, you can modify the amount of time watchdogtimerd waits before rebooting. See watchdogtimerd for further information on modification of the watchdog behavior.

Listing 12 shows an example of how to set the boot-args firmware variable, assuming that the kernel core dump server's IP address is 10.0.40.2.

Listing 12: Setting the boot-args firmware variable

client$ sudo nvram boot-args="debug=0xd44 _panicd_ip=10.0.40.2"
Password: ********

You must restart to enable this setting.

Debug Flags in Depth

You can use the boot-args debug flags to control various aspects of kernel debugging. Many of these flags are documented in existing documentation. Table 1 describes the flags most relevant to kernel core dumps. Table 2 shows how to combine these flags to effect various useful behaviors.

Table 1 : Debug flags

Table 2 : Useful debug flag combinations

Back to Top

Testing Your Configuration

Once you have enabled the server and configured the client, it's time to test things to make sure they're working as expected. To test the system you need to trigger a kernel panic on the client system. Two methods are presented here.

1. You can trigger a panic using DTrace on client systems running Mac OS X 10.5 or greater.

2. You can trigger a panic using the InstantPanic kernel extension which works for all releases of Mac OS X.

Triggering a Kernel Panic with DTrace

For Mac OS X 10.5 or later, use dtrace in a Terminal window as shown in Listing 13 to trigger a kernel panic on the client system.

Listing 13: Using DTrace to trigger a panic on the client system

client$ sudo dtrace -w -n "BEGIN{ panic();}"

Back to Top

Triggering a Kernel Panic with the Instant Panic Kernel Extension

For all versions of Mac OS X, this technical note includes a simple kernel extension, InstantPanic, that panics the kernel as soon as you load it. You can get the source and binary for this kernel extension from the Downloadables section.

Start by downloading and unarchiving the kernel extension on your desktop. Load the kernel extension using the commands shown in Listing 14.

Listing 14: Triggering a panic by loading the 'InstantPanic' kernel extension

client$ cd ~/Desktop/InstantPanic/build/
client$ sudo cp -r InstantPanic.kext /
Password: ********
client$ sudo kextload /InstantPanic.kext

Back to Top

The Kernel Panic Core Dump File

Once a panic occurs, if all goes well, the client will transmit a kernel core dump to the server. You can see the transmission progress on the screen of the client (assuming that you set the DB_LOG_PI_SCRN debug flag).

When the transfer is complete you can list the /PanicDumps directory on the server to see the new panic dump file.

Listing 15: An example kernel core dump

server$ ls -l /PanicDumps
total 216872
-rw-rw---- 1 nobody  wheel  300245144 Sep 19 12:51 paniclog-xnu-1228.5.20-10.0.40.7-3a061187

The name of this file includes the kernel version—as displayed by uname -a, in this case 1228.5.20-and the client's IP address (10.0.40.7), along with a unique timestamp.

The section Debugging with Kernel Core Dumps explains how you can debug a kernel panic using the core dump file. Alternatively, if you had set the DB_PANICLOG_DUMP flag, the /PanicDumps directory will contain a panic log file, much like the file you'd get in /Library/Logs after a panic. Listing 16 shows an example of this.

Listing 16: An example panic log file

server$ ls -l /PanicDumps
total 216872
-rw-rw---- 1 nobody  wheel       300245144 Sep 19 12:51 paniclog-xnu-1228.5.20-10.0.40.7-3a061187
server$ sudo cat /PanicDumps/paniclog-xnu-1228.5.20-10.0.40.7-3a061187 
panic(cpu 0 caller 0x21058018): InstantPanic: Just add water!
Backtrace, Format - Frame : Return Address (4 potential args on stack) 
0x1b38fc18 : 0x12b0fa (0x4592a4 0x1b38fc4c 0x133243 0x0) 
0x1b38fc68 : 0x21058018 (0x21058053 0x0 0x2a57e38 0x2e4f4a0) 
0x1b38fc88 : 0x18f24d (0x21058080 0x0 0x0 0x0) 
0x1b38fcc8 : 0x18f51e (0x6f 0x1 0x2e4f4a4 0x2e4f4c0) 
0x1b38fd68 : 0x147c60 (0x53b8e0 0x6f 0x1 0x2e4f4a4) 
0x1b38fdb8 : 0x12d17e (0x2e4f488 0x3300790 0x1b38fdf8 0x11ee14) 
0x1b38fdf8 : 0x126257 (0x2e4f400 0x2a50844 0x3338e88 0x0) 
0x1b38ff08 : 0x1973dd (0x1b38ff44 0x0 0x0 0x0) 
0x1b38ffc8 : 0x19f3b3 (0x33cfc34 0x0 0x1a20b5 0x2f73790) 
No mapping exists for frame pointer
Backtrace terminated-invalid frame pointer 0xbffff458
      Kernel loadable modules in backtrace (with dependencies):
         com.apple.dts.kext.InstantPanic(1.0)@0x21057000->0x21058fff

BSD process name corresponding to current thread: kextload

Mac OS version:
9E17

Kernel version:
Darwin Kernel Version 9.4.0: Mon Jun  9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386
System model name: iMac4,1 (Mac-F42787C8)

For more information about kernel panic logs, see Technical Note TN2063: Understanding and Debugging Kernel Panics.

Back to Top

Debugging with Kernel Core Dumps

If you're a kernel programmer, you can use kernel core dumps to debug kernel panics and hangs. Before you start, you should collect together the following helpful resources.

• The Kernel Debug Kit for the kernel that panicked. You can get Kernel Debug Kits from the Apple developer web site.

  The rest of this section assumes that you have mounted the correct Kernel Debug Kit disk image on your machine.

• The Darwin source code for the kernel that panicked. You can get Darwin source code from the Darwin Releases page on the Apple developer site. The source for the kernel is held in the xnu project.

  The Darwin kernel source is virtually identical to the source used to build the Mac OS X kernel. In almost all cases you can use the Darwin source to do meaningful source-level debugging of the Mac OS X kernel.

  The rest of this section assumes that the machine that panicked is running Mac OS X 10.5.4, which corresponds to xnu-1228.5.20. Furthermore, it assumes that you have downloaded the xnu-1228.5.20 source and that you unarchived it to a folder on your desktop named xnu-1228.5.20.

The first step to debugging with a kernel core file is to open that file in GDB using the -c option. Listing 17 shows an example of this. Once you've opened the core file in GDB, you can inspect the state of the kernel using standard GDB commands. This example uses bt to display a backtrace of the stack.

Listing 17: Opening a kernel core file in GDB

server$ sudo gdb -c /PanicDumps/core-xnu-1228.5.20-10.0.40.7-10499ce2 
Password: ********
GNU gdb 6.3.50-20050815 (Apple version gdb-768) (Tue Oct  2 04:07:49 UTC 2007)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-apple-darwin".
#0  0x001ae4cd in ?? ()

As you can see, this backtrace contains no symbolic information. You can fix this by loading kernel symbols from the Kernel Debug Kit. Listing 18 shows an example of this. Once you load the kernel symbols, the backtrace is now much more helpful.

Listing 18: Loading kernel symbols

(gdb) add-symbol-file /Volumes/KernelDebugKit/mach_kernel
add symbol table from file "/Volumes/KernelDebugKit/mach_kernel"? (y or n) y
Reading symbols from /Volumes/KernelDebugKit/mach_kernel...Reading symbols from /Volumes/KernelDebugKit/mach_kernel.dSYM/Contents/Resources/DWARF/mach_kernel...done.
done.
(gdb) bt
#0  Debugger (message=0x4592a4 "panic") at /SourceCache/xnu/xnu-1228.5.20/osfmk/i386/AT386/model_dep.c:786
#1  0x0012b0fa in panic (str=0x210c7053 "InstantPanic: Just add water!") at /SourceCache/xnu/xnu-1228.5.20/osfmk/kern/debug.c:274
#2  0x210c7018 in ?? ()

Now that you have symbols, you're only a short step away from source-level debugging. As you can see from frame 1 of the backtrace, GDB expects to find the kernel source in the directory /SourceCache/xnu/xnu-1228.5.20. You can meet that expectation by virtue of a well-placed symbolic link. Listing 19 shows how to create this link.

Listing 19: Creating a symlink so that GDB can find the source

server$ mkdir -p /SourceCache/xnu
server$ ln -s ~/Desktop/xnu-1228.5.20 /SourceCache/xnu

Now you can look back up the stack (using the frame command) and get actual source code listings (using the list command). Listing 20 shows an example of this.

Listing 20: Source-level debugging

(gdb) frame 0
#0  Debugger (message=0x4592a4 "panic") at /SourceCache/xnu/xnu-1228.5.20/osfmk/i386/AT386/model_dep.c:786
warning: Source file is more recent than executable.
786		hw_atomic_sub(&debug_mode, 1); 
(gdb) frame 1
#1  0x0012b0fa in panic (str=0x210c7053 "InstantPanic: Just add water!") at /SourceCache/xnu/xnu-1228.5.20/osfmk/kern/debug.c:274
warning: Source file is more recent than executable.
274		Debugger("panic");

Last, but certainly not least, you can use the kernel debugging macros on a kernel core dump in the same way you would on a live kernel. Listing 21 shows how to load up the kernel debugging macros and execute two of the most useful macros.

• paniclog prints the standard panic information

• showallstacks prints a backtrace for everything thread running within the kernel

Listing 21: Kernel debugging macros in action

(gdb) source /Volumes/KernelDebugKit/kgmacros 
Loading Kernel GDB Macros package. Type "help kgm" for more info.
(gdb) paniclog
panic(cpu 0 caller 0x21114018): InstantPanic: Just add water!
Backtrace, Format - Frame : Return Address (4 potential args on stack) 
0x20ed7c18 : 0x12b0fa (0x4592a4 0x20ed7c4c 0x133243 0x0) 
0x20ed7c68 : 0x21114018 (0x21114053 0x0 0x2a57e38 0x538eca0) 
0x20ed7c88 : 0x18f24d (0x21114080 0x0 0x0 0x0) 
0x20ed7cc8 : 0x18f51e (0x72 0x1 0x538eca4 0x538ecc0) 
0x20ed7d68 : 0x147c60 (0x53b8e0 0x72 0x1 0x538eca4) 
0x20ed7db8 : 0x12d17e (0x538ec88 0x5399590 0x20ed7df8 0x11ee14) 
0x20ed7df8 : 0x126257 (0x538ec00 0x2a50508 0x39cf5a0 0x0) 
0x20ed7f08 : 0x1973dd (0x20ed7f44 0x0 0x0 0x0) 
0x20ed7fc8 : 0x19f3b3 (0x40c5e8c 0x19ecd7 0x8 0x40c5e8c) 
No mapping exists for frame pointer
Backtrace terminated-invalid frame pointer 0xbffff458
      Kernel loadable modules in backtrace (with dependencies):
         com.apple.dts.kext.InstantPanic(1.0)@0x21113000->0x21114fff

BSD process name corresponding to current thread: kextload

Mac OS version:
9E17

Kernel version:
Darwin Kernel Version 9.4.0: Mon Jun  9 19:30:53 PDT 2008; root:xnu-1228.5.20~1/RELEASE_I386
System model name: iMac4,1 (Mac-F42787C8)

(gdb) showallstacks
[...]
task        vm_map      ipc_space  #acts   pid  proc        command
0x0345182c  0x039cf5a0  0x02a50508    1    576  0x0333b4a0  kextload
            thread      processor   pri  state  wait_queue  wait_event
            0x05358a78  0x0053b0c0   31  R
        kernel_stack=0x20ed4000
        stacktop=0x20ed7c18
        0x20ed7c18  0x12b0fa <panic+422>
        0x20ed7c68  0x21114018 <com.apple.dts.kext.InstantPanic + 0x1018>
        0x20ed7c88  0x18f24d <kmod_start_or_stop+213>
        0x20ed7cc8  0x18f51e <kmod_control+112>
        0x20ed7d68  0x147c60 <_Xkmod_control+240>
        0x20ed7db8  0x12d17e <ipc_kobject_server+247>
        0x20ed7df8  0x126257 <mach_msg_overwrite_trap+752>
        0x20ed7f08  0x1973dd <mach_call_munger+529>
        0x20ed7fc8  0x19f3b3 <lo_mach_scall+227>
        stackbottom=0x20ed7fc8

You can learn more about the kernel debugging macros in the I/O Kit documentation.

Back to Top

Kernel Core Dump Options

The information above is sufficient for the needs of most developers to obtain kernel panic core dumps. However, there are options to address some networking issues.

• If a client or server system has an existing network process which uses UDP port 1069, it is possible in some situations to alter the UDP port assignment for the client and server system. Read the section Configuring the Client and Server UDP Port Assignment for customizing the core dump port assignment use.

• A client system may connected to a local area network where there is more than one router through which the client can send the core dump traffic to the core dump server. it can be useful in this network topology to specify that the client system use a specific router to send the core dump packets to the server. Read the section Configuring the Client Router Usage to set the client to use a specific router.

• By default, the client system will use the built-in Ethernet interface en0 for kernel debugging, including kernel core dumps. On some machines like the Mac OS X Server system, and those desktop Macintosh system which have two built-in Ethernet ports, you might want the kernel core dump to be sent out the Ethernet interface en1. Read the section Specifying the Ethernet Interface to set the client to use a specific built-in Ethernet interface.

Configuring the Client and Server UDP Port Assignment

The kernel dump protocol uses UDP on port 1069 by default. The port assignment is configurable for all Mac OS X releases on the kernel dump server side. On the client side, for the Intel-based Macintosh systems you must have Mac OS X 10.4.7 or greater to configure a custom port assignment. For a PowerPC-based Macintosh system, you must have Mac OS X 10.5 or greater to configure a custom port assignment. If you want the core dump server to collect core dumps from PowerPC clients running Mac OS X 10.4 or earlier, or from Intel-based client running Mac OS X 10.4.6, you must leave the server port assignment set to the default value of 1069.

Configuring the Client UDP Port Assignment

To customize the client port assignment to 12345, add _panicd_port=12345 as a boot-args argument. Listing 22 shows an example of setting the boot-args to include the custom _panicd_port setting.

Listing 22: Setting the client boot-args firmware variable to include the port assignment

client$ sudo nvram boot-args="debug=0xd44 _panicd_ip=10.0.40.2 _panicd_port=12345"
Password: ********

You must restart to enable this setting.

Back to Top

Configuring the Server UDP Port Assignment

For Mac OS X 10.5 or greater, to customize the server port assignment, edit the /System/Library/LaunchDaemon/com.apple.kdump.plist launchd property list file. Change the SockServiceName string property from 1069 to the desired port assignment. Restart the system for the change to take effect.

For a core dump server running under Mac OS X 10.4.x, to modify the port assignment refer to Listing 4 and modify the SockServiceName string property from 1069 to the desired port assignment.

For a core dump server running under Mac OS X 10.3.x, to modify the port assignment refer to Listing 7 and modify the port value from 1069 to the desired port assignment.

Back to Top

Configuring the Client Router Usage

There are some situations where you want to force the core dump client to communicate to the core dump server using a specific router. This option exists for Intel-based clients systems with Mac OS X 10.4.7 or greater present, and for PowerPC-based clients with Mac OS X 10.5 or greater present. To customize the client router address assignment to 10.0.40.1, add _router_ip=10.0.40.1 as a boot-args argument. Listing 23 shows an example of setting the boot-args to include the custom _router_ip setting.

Listing 23: Setting the client boot-args firmware variable to include the router address assignment

client$ sudo nvram boot-args="debug=0xd44 _panicd_ip=10.0.40.2 _router_ip=10.0.40.1"
Password: ********

You must restart to enable this setting.

Back to Top

Specifying the Ethernet Interface

You can force the system to use a particular port by setting the kdp_match_name boot argument. For example, to set the system to always use "en1" for kernel debugging, add kdp_match_name=en1 to your boot-args setting. Refer to Listing 24 for an example of setting the firmware kdp_match_name=en1 property.

Listing 24: Setting the boot-args firmware variable to include the kdp_match_name assignment

client$ sudo nvram boot-args="debug=0xd44 _panicd_ip=10.0.40.2 kdp_match_name=en1"
Password: ********

You must restart to enable this setting.

Back to Top

Conclusion

The kernel core dump facility is a useful debugging tool for both kernel extension developers and users with large or complex Macintosh installations. Using this facility, you can capture information about kernel panics (and kernel hangs) where it's not possible to use the two-machine kernel debugger.

Back to Top

Further Reading

• Hello Debugger: Debugging a Device Driver With GDB

• Kernel Programming, especially the When Things Go Wrong: Debugging the Kernelsection

• Packaging Your KEXT for Distribution and Installation

• Mac OS X Hardware & Drivers download page

• Kernel Debug Kits

• Technical Note TN2063: Understanding and Debugging Kernel Panics

• Technical Note TN2163: Building Universal I/O Kit Drivers

• Technical Q&A QA1264: Generating a Non-Maskable Interrupt (NMI)

• kdumpd man page

• Darwin source code, especially the kernel source (module xnu) and the kdumpd source (module network_cmds)

• xinetd man page only present on Mac OS X 10.4 and earlier.

• watchdogtimerd man page only present on Mac OS X Server systems.

Back to Top

Downloadables

• A kernel extension that will panic your system. ("tn2118_InstantPanic.zip", 24.1K)

Back to Top

Document Revision History

Posted: 2008-11-12