Archive for April 22nd, 2008

Crash Dump Analysis Patterns (Part 59)

Tuesday, April 22nd, 2008

David V provided an idea and a user dump for the next pattern which I call Missing Component. Sometimes the code raises an exception when certain DLL is missing. We need to guess that component name if we don’t have symbols and source code. This can be done by inspecting raw stack data in the close proximity of the exception ESP/RSP.

Consider the crash dump of Zune.exe with the following incomplete unmanaged stack trace:

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 76f442eb (kernel32!RaiseException+0x00000058)
   ExceptionCode: c06d007f
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0024f21c

0:000> kL
ChildEBP RetAddr 
0024f1f8 6eb1081e kernel32!RaiseException+0x58
WARNING: Stack unwind information not available. Following frames may be wrong.
0024f260 6eac62fb ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f2ac 6ea9e269 ZuneNativeLib!ZuneLibraryExports::Phase2Initialization+0x24c9
0024f32c 79e74d79 ZuneNativeLib!ZuneLibraryExports::QueryDatabase+0x99da
0024f3d4 664bd6af mscorwks!MethodTable::IsValueType+0x35
0024f3e8 319cec9e ZuneShell_ni+0x2d6af
0024f3f4 31a15d19 UIX_ni+0x1ec9e
0024f3f8 00000000 UIX_ni+0x65d19

We can try to interpret the crash as Managed Code Exception but let’s first to check the exception code. Google search shows that the error code c06d007f means “DelayLoad Export Missing” and this definitely has to do with some missing DLL. It is not possible to tell which one was missing from the stack trace output. Additional digging is required.

Let’s look at the raw stack. First, we can try to see whether there are any calls to LoadLibrary on thread raw stack data:

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0024f8c4
    StackBase:            00250000
    StackLimit:           00249000

    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             000012f4 . 00001080
    RpcHandle:            00000000
    Tls Storage:          004e8a18
    PEB Address:          7ffde000
    LastErrorValue:       126
    LastStatusValue:      c0000135
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 00249000 00250000
00249000 00000000
00249004 00000000
00249008 00000000
0024900c 00000000
00249010 00000000
00249014 00000000
00249018 00000000
[...]
0024f1a0 00000000
0024f1a4 00000000
0024f1a8 c06d007f
0024f1ac 00000000
0024f1b0 00000000
0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0x11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0x15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
[...]

There are no such calls in our crash dump. Then we can try to interpret raw stack data as a byte stream to see “.dll” strings:

0:000> db 00249000 00250000
00249000 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249020 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249030 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
[...]

There are no such strings except “user32.dll”.

Now we can try to interpret every double word as a pointer to a Unicode string:

0:000> dpu 00249000 00250000
[...]

There are no strings with “.dll” inside. Finally, if we try to interpret every double word as a pointer to an ASCII string we get a few references to “ZuneService.dll”:

0:000> dpa 00249000 00250000
[...]
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 "MZ."
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8 ""
0024f1dc 76fb8e8f "..t-.E."
0024f1e0 6ecb9b40 “ZuneService.dll”
0024f1e4 0024f21c “$”
0024f1e8 0024f200 “.”
0024f1ec 6ec74e2a “..^.._]..”
0024f1f0 6ecb9b40 “ZuneService.dll”
0024f1f4 6ecb9ff0 “CreateServiceInstance”
0024f1f8 0024f260 “..$”
0024f1fc 6eb1081e “.]…….e.”
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268 “..$”
0024f210 00000000
0024f214 0024f2c8 “…n ..n<.$”
0024f218 6ecbe220 “”
0024f21c 00000024
0024f220 6ecb9960 “.”
0024f224 6ecbe05c “.c.n.2.n”
0024f228 6ecb9b40 “ZuneService.dll”
0024f22c 00000001
0024f230 6ecb9ff0 “CreateServiceInstance”
0024f234 ffffffff
0024f238 00000000

If we search for 0024f1e0 pointer in dps WinDbg command output we would see that it is in a close proximity to RaiseException call and it seems that all our pointers to “ZuneService.dll” string fall into ZuneNativeLib address range:

0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0×11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0×6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268
0024f210 00000000
0024f214 0024f2c8
0024f218 6ecbe220 ZuneNativeLib!ShutdownSingletonMgr+0×15f704
0024f21c 00000024
0024f220 6ecb9960 ZuneNativeLib!ShutdownSingletonMgr+0×15ae44
0024f224 6ecbe05c ZuneNativeLib!ShutdownSingletonMgr+0×15f540
0024f228 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f22c 00000001
0024f230 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f234 ffffffff
0024f238 00000000

When examining the system it was found that ZuneService.dll was missing there indeed.

- Dmitry Vostokov @ DumpAnalysis.org -

Bugchecks: SYSTEM_SERVICE_EXCEPTION

Tuesday, April 22nd, 2008

Bugcheck 0×3B is forced on x64 Windows platforms when an exception happens during a system service and unwind leads to a transition from a kernel to a user mode. Let’s see this in a complete memory dump:

SYSTEM_SERVICE_EXCEPTION (3b)
An exception happened while executing a system service routine.
Arguments:
Arg1: 00000000c0000005, Exception code that caused the bugcheck
Arg2: fffff80001048a1d, Address of the exception record for the exception that caused the bugcheck
Arg3: fffffade643f6870, Address of the context record for the exception that caused the bugcheck
Arg4: 0000000000000000, zero.

CONTEXT: fffffade643f6870 -- (.cxr 0xfffffade643f6870)
rax=005300450053005c rbx=0000000000000048 rcx=0000000000000020
rdx=fffffa8007c9da20 rsi=0000000000000048 rdi=fffffade643f71d0
rip=fffff80001048a1d rsp=fffffade643f7088 rbp=0000000000000000
 r8=0000000000000048 r9=0000000000000002 r10=00490046002d0054
r11=0000000000000000 r12=fffffadf19744010 r13=fffffade643f7a78
r14=0000000000000800 r15=fffffadf1da71ee8
iopl=0 nv up ei pl nz na pe nc
cs=0010 ss=0018 ds=002b es=002b fs=0053 gs=002b efl=00010202
nt!memmove+0xbd:
fffff800`01048a1d 488941e0 mov qword ptr [rcx-20h],rax ds:002b:00000000`00000000=????????????????

0: kd> kL
Child-SP RetAddr Call Site
fffffade`643f5eb8 fffff800`0104e834 nt!KeBugCheckEx
fffffade`643f5ec0 fffff800`0104e2fb nt!KiBugCheckDispatch+0x74
fffffade`643f6040 fffff800`0105c09d nt!KiSystemServiceHandler+0x7b
fffffade`643f6080 fffff800`01031561 nt!RtlpExecuteHandlerForException+0xd
fffffade`643f60b0 fffff800`010174fa nt!RtlDispatchException+0x2c0
fffffade`643f6770 fffff800`0104e92f nt!KiDispatchException+0xd9
fffffade`643f6d70 fffff800`0104d7e1 nt!KiExceptionExit
fffffade`643f6ef0 fffff800`01048a1d nt!KiPageFault+0x1e1
fffffade`643f7088 fffff800`01025977 nt!memmove+0xbd
fffffade`643f7090 fffffadf`101f858d nt!RtlAppendUnicodeStringToString+0x67
fffffade`643f70c0 fffffadf`101f8a1d driver+0x558d
fffffade`643f7a20 fffff800`012c3b21 driver+0x5a1d
fffffade`643f7a70 fffff800`012c3bd6 nt!IopXxxControlFile+0xa6b
fffffade`643f7b90 fffff800`0104e5fd nt!NtDeviceIoControlFile+0x56
fffffade`643f7c00 00000000`77ef12ca nt!KiSystemServiceCopyEnd+0×3
00000000`00e6ba08 00000000`77d67963 ntdll!ZwDeviceIoControlFile+0xa
00000000`00e6ba10 00000000`6340239f kernel32!DeviceIoControl+0×237
00000000`00e6bbf0 00000000`0000000e application!DllUnregisterServer+0×40f
[…]

On x64 Windows platforms KiSystemServiceCopyEnd has the similar purpose as KiFastSystemCallRet on x86 platforms.

We see that the chain of exception handlers spans protection boundary where KiSystemServiceCopyEnd has KiSystemServiceHandler as its exception handler:

0: kd> !exchain
100 stack frames, scanning for handlers...
Frame 0x03: nt!RtlpExecuteHandlerForException+0xd (fffff800`0105c09d)
  ehandler nt!RtlpExceptionHandler (fffff800`0105c060)
Frame 0x05: nt!KiDispatchException+0xd9 (fffff800`010174fa)
  ehandler nt!_C_specific_handler (fffff800`010356e0)
Frame 0x0a: driver+0x558d (fffffadf`101f858d)
  ehandler driver+0x1242 (fffffadf`101f4242)
Frame 0x0c: nt!IopXxxControlFile+0xa6b (fffff800`012c3b21)
  ehandler nt!_C_specific_handler (fffff800`010356e0)
Frame 0×0e: nt!KiSystemServiceCopyEnd+0×3 (fffff800`0104e5fd)
  ehandler nt!KiSystemServiceHandler (fffff800`0104e280)

Frame 0×10: kernel32!DeviceIoControl+0×237 (00000000`77d67963)
  ehandler kernel32!_C_specific_handler (00000000`77d92200)

If we disassemble KiSystemServiceHandler we get this code with bugcheck 3B branch:

kd> uf nt!KiSystemServiceHandler
nt!KiSystemServiceHandler:
[...]
fffff800`01040ddc cmp     byte ptr [rax+153h],0
fffff800`01040de3 je      nt!KiSystemServiceHandler+0×7b (fffff800`01040dfb)

nt!KiSystemServiceHandler+0x65:
fffff800`01040de5 xor     r10,r10
fffff800`01040de8 mov     r9,r8
fffff800`01040deb mov     r8,qword ptr [rcx+10h]
fffff800`01040def mov     edx,dword ptr [rcx]
fffff800`01040df1 mov     ecx,3Bh
fffff800`01040df6 call    nt!KiBugCheckDispatch (fffff800`01041300)

nt!KiSystemServiceHandler+0x7b:
fffff800`01040dfb mov     eax,1
fffff800`01040e00 add     rsp,38h
fffff800`01040e04 ret
[...]

Here we see that the code checks if the previous mode for a thread was UserMode and if this is the case it bugchecks the system because transitioning back to  user space in exception unwind would have had disastrous consequences. The system wants to save a controlled crash dump for later problem analysis:

kd> dt _KTHREAD
ntdll!_KTHREAD
   +0x000 Header           : _DISPATCHER_HEADER
   +0x018 MutantListHead   : _LIST_ENTRY
   +0x028 InitialStack     : Ptr64 Void
   +0x030 StackLimit       : Ptr64 Void
   +0x038 KernelStack      : Ptr64 Void
[...]
   +0×153 PreviousMode     : Char
[…]

Note that _KTHREAD.PreviousMode should not be confused with _KTRAP_FRAME.PreviousMode. The latter has KernelMode value if an exception happened while CPU was in kernel mode but the former structure field shows the previous CPU mode of a thread, for example, it has UserMode value if a user space thread called a system service.

kd> dt _KTRAP_FRAME
ntdll!_KTRAP_FRAME
  +0x000 P1Home : Uint8B
  +0x008 P2Home : Uint8B
  +0x010 P3Home : Uint8B
  +0x018 P4Home : Uint8B
  +0x020 P5 : Uint8B
  +0×028 PreviousMode : Char
  +0×029 PreviousIrql : UChar
  +0×02a FaultIndicator : UChar
[…]

I put all of this on a colored sequence UML diagram:

 

- Dmitry Vostokov @ DumpAnalysis.org -