Crash Dump Analysis Patterns (Part 59)

David V provided an idea and a user dump for the next pattern which I call Missing Component. Sometimes the code raises an exception when certain DLL is missing. We need to guess that component name if we don’t have symbols and source code. This can be done by inspecting raw stack data in the close proximity of the exception ESP/RSP.

Consider the crash dump of Zune.exe with the following incomplete unmanaged stack trace:

EXCEPTION_RECORD:  ffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 76f442eb (kernel32!RaiseException+0x00000058)
   ExceptionCode: c06d007f
  ExceptionFlags: 00000000
NumberParameters: 1
   Parameter[0]: 0024f21c

0:000> kL
ChildEBP RetAddr 
0024f1f8 6eb1081e kernel32!RaiseException+0x58
WARNING: Stack unwind information not available. Following frames may be wrong.
0024f260 6eac62fb ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f2ac 6ea9e269 ZuneNativeLib!ZuneLibraryExports::Phase2Initialization+0x24c9
0024f32c 79e74d79 ZuneNativeLib!ZuneLibraryExports::QueryDatabase+0x99da
0024f3d4 664bd6af mscorwks!MethodTable::IsValueType+0x35
0024f3e8 319cec9e ZuneShell_ni+0x2d6af
0024f3f4 31a15d19 UIX_ni+0x1ec9e
0024f3f8 00000000 UIX_ni+0x65d19

We can try to interpret the crash as Managed Code Exception but let’s first to check the exception code. Google search shows that the error code c06d007f means “DelayLoad Export Missing” and this definitely has to do with some missing DLL. It is not possible to tell which one was missing from the stack trace output. Additional digging is required.

Let’s look at the raw stack. First, we can try to see whether there are any calls to LoadLibrary on thread raw stack data:

0:000> !teb
TEB at 7ffdf000
    ExceptionList:        0024f8c4
    StackBase:            00250000
    StackLimit:           00249000

    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdf000
    EnvironmentPointer:   00000000
    ClientId:             000012f4 . 00001080
    RpcHandle:            00000000
    Tls Storage:          004e8a18
    PEB Address:          7ffde000
    LastErrorValue:       126
    LastStatusValue:      c0000135
    Count Owned Locks:    0
    HardErrorMode:        0

0:000> dds 00249000 00250000
00249000 00000000
00249004 00000000
00249008 00000000
0024900c 00000000
00249010 00000000
00249014 00000000
00249018 00000000
[...]
0024f1a0 00000000
0024f1a4 00000000
0024f1a8 c06d007f
0024f1ac 00000000
0024f1b0 00000000
0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0x11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0x15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0x15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0x6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
[...]

There are no such calls in our crash dump. Then we can try to interpret raw stack data as a byte stream to see “.dll” strings:

0:000> db 00249000 00250000
00249000 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249010 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249020 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00249030 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
[...]

There are no such strings except “user32.dll”.

Now we can try to interpret every double word as a pointer to a Unicode string:

0:000> dpu 00249000 00250000
[...]

There are no strings with “.dll” inside. Finally, if we try to interpret every double word as a pointer to an ASCII string we get a few references to “ZuneService.dll”:

0:000> dpa 00249000 00250000
[...]
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 "MZ."
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8 ""
0024f1dc 76fb8e8f "..t-.E."
0024f1e0 6ecb9b40 “ZuneService.dll”
0024f1e4 0024f21c “$”
0024f1e8 0024f200 “.”
0024f1ec 6ec74e2a “..^.._]..”
0024f1f0 6ecb9b40 “ZuneService.dll”
0024f1f4 6ecb9ff0 “CreateServiceInstance”
0024f1f8 0024f260 “..$”
0024f1fc 6eb1081e “.]…….e.”
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268 “..$”
0024f210 00000000
0024f214 0024f2c8 “…n ..n<.$”
0024f218 6ecbe220 “”
0024f21c 00000024
0024f220 6ecb9960 “.”
0024f224 6ecbe05c “.c.n.2.n”
0024f228 6ecb9b40 “ZuneService.dll”
0024f22c 00000001
0024f230 6ecb9ff0 “CreateServiceInstance”
0024f234 ffffffff
0024f238 00000000

If we search for 0024f1e0 pointer in dps WinDbg command output we would see that it is in a close proximity to RaiseException call and it seems that all our pointers to “ZuneService.dll” string fall into ZuneNativeLib address range:

0024f1b4 76f442eb kernel32!RaiseException+0x58
0024f1b8 00000001
0024f1bc 0024f21c
0024f1c0 00000000
0024f1c4 00000000
0024f1c8 00000000
0024f1cc 00000000
0024f1d0 76f00000 kernel32!_imp___aullrem (kernel32+0x0)
0024f1d4 f7bd2a5d
0024f1d8 0024f1e8
0024f1dc 76fb8e8f kernel32!LookupHandler+0x10
0024f1e0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1e4 0024f21c
0024f1e8 0024f200
0024f1ec 6ec74e2a ZuneNativeLib!ShutdownSingletonMgr+0×11630e
0024f1f0 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f1f4 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f1f8 0024f260
0024f1fc 6eb1081e ZuneNativeLib!ZuneLibraryExports::InteropNotifyUnAdvise+0×6aa9
0024f200 c06d007f
0024f204 00000000
0024f208 00000001
0024f20c 0024f268
0024f210 00000000
0024f214 0024f2c8
0024f218 6ecbe220 ZuneNativeLib!ShutdownSingletonMgr+0×15f704
0024f21c 00000024
0024f220 6ecb9960 ZuneNativeLib!ShutdownSingletonMgr+0×15ae44
0024f224 6ecbe05c ZuneNativeLib!ShutdownSingletonMgr+0×15f540
0024f228 6ecb9b40 ZuneNativeLib!ShutdownSingletonMgr+0×15b024
0024f22c 00000001
0024f230 6ecb9ff0 ZuneNativeLib!ShutdownSingletonMgr+0×15b4d4
0024f234 ffffffff
0024f238 00000000

When examining the system it was found that ZuneService.dll was missing there indeed.

- Dmitry Vostokov @ DumpAnalysis.org -

6 Responses to “Crash Dump Analysis Patterns (Part 59)”

  1. Marc Sherman Says:

    When searching the stack, how did you decide to start at 00249000? The lowest address given by kL is 0024f1f8 (child ebp for kernel32!RaiseException). The difference of the two is over 24K.

    thanks,
    Marc

  2. Dmitry Vostokov Says:

    I forgot to include !teb in the post. Corrected it now:

    0:000> !teb
    TEB at 7ffdf000
    ExceptionList: 0024f8c4
    StackBase: 00250000
    StackLimit: 00249000
    SubSystemTib: 00000000
    FiberData: 00001e00
    ArbitraryUserPointer: 00000000
    Self: 7ffdf000
    EnvironmentPointer: 00000000
    ClientId: 000012f4 . 00001080
    RpcHandle: 00000000
    Tls Storage: 004e8a18
    PEB Address: 7ffde000
    LastErrorValue: 126
    LastStatusValue: c0000135
    Count Owned Locks: 0
    HardErrorMode: 0

  3. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 59b) Says:

    […] I introduced Missing Component pattern where the example and emphasis was on dynamically loaded modules. In this part I cover […]

  4. Software Generalist » Blog Archive » Reading Notebook: 16-Feb-09 Says:

    […] problems (pp. 122 -127) - For Windows, I’ve identified 2 patterns so far: http://www.dumpanalysis.org/blog/index.php/2008/04/22/crash-dump-analysis-patterns-part-59/ and […]

  5. Crash Dump Analysis » Blog Archive » DLL Link Patterns Says:

    […] Missing Component (general) […]

  6. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 92) Says:

    […] we should check if it is exited but references to it exist or there are any missing threads or components inside it, any suspended threads and special processes like a postmortem debugger. We […]

Leave a Reply

You must be logged in to post a comment.