Crash Dump Analysis Patterns (Part 8)

Today I will talk about another pattern occurring frequently and I call it Hidden Exception. You run !analyze -v command and you don’t see an exception or you see only a breakpoint hit. In this case manual analysis is required. This happens sometimes because of another pattern: Multiple Exceptions. In other cases an exception happens and it is handled by an exception handler dismissing it and a process continues execution slowly accumulating corruption inside its data leading to a new crash or hang. Sometimes you see a process hanging during its termination like the case I present here.

We have a process dump with only one thread:

0:000> kv
ChildEBP RetAddr
0096fcdc 7c822124 ntdll!KiFastSystemCallRet
0096fce0 77e6baa8 ntdll!NtWaitForSingleObject+0xc
0096fd50 77e6ba12 kernel32!WaitForSingleObjectEx+0xac
0096fd64 67f016ce kernel32!WaitForSingleObject+0x12
0096fd78 7c82257a component!DllInitialize+0xc2
0096fd98 7c8118b0 ntdll!LdrpCallInitRoutine+0x14
0096fe34 77e52fea ntdll!LdrShutdownProcess+0x130
0096ff20 77e5304d kernel32!_ExitProcess+0x43
0096ff34 77bcade4 kernel32!ExitProcess+0x14
0096ff40 77bcaefb msvcrt!__crtExitProcess+0x32
0096ff70 77bcaf6d msvcrt!_cinit+0xd2
0096ff84 77bcb555 msvcrt!_exit+0x11
0096ffb8 77e66063 msvcrt!_endthreadex+0xc8
0096ffec 00000000 kernel32!BaseThreadStart+0x34

We can look at its raw stack and try to find the following address:

KiUserExceptionDispatcher

This function calls RtlDispatchException:

0:000> !teb
TEB at 7ffdc000
    ExceptionList:        0096fd40
    StackBase:            00970000
    StackLimit:           0096a000
    SubSystemTib:         00000000
    FiberData:            00001e00
    ArbitraryUserPointer: 00000000
    Self:                 7ffdc000
    EnvironmentPointer:   00000000
    ClientId:             00000858 . 000008c0
    RpcHandle:            00000000
    Tls Storage:          00000000
    PEB Address:          7ffdd000
    LastErrorValue:       0
    LastStatusValue:      c0000135
    Count Owned Locks:    0
    HardErrorMode:        0

0:000>dds 0096a000 00970000
...
...
...
0096c770  7c8140cc ntdll!RtlDispatchException+0x91
0096c774  0096c808
0096c778  0096ffa8
0096c77c  0096c824
0096c780  0096c7e4
0096c784  77bc6c74 msvcrt!_except_handler3
0096c788  00000000
0096c78c  0096c808
0096c790  01030064
0096c794  00000000
0096c798  00000000
0096c79c  00000000
0096c7a0  00000000
0096c7a4  00000000
0096c7a8  00000000
0096c7ac  00000000
0096c7b0  00000000
0096c7b4  00000000
0096c7b8  00000000
0096c7bc  00000000
0096c7c0  00000000
0096c7c4  00000000
0096c7c8  00000000
0096c7cc  00000000
0096c7d0  00000000
0096c7d4  00000000
0096c7d8  00000000
0096c7dc  00000000
0096c7e0  00000000
0096c7e4  00000000
0096c7e8  00970000
0096c7ec  00000000
0096c7f0  0096caf0
0096c7f4  7c82ecc6 ntdll!KiUserExceptionDispatcher+0xe
0096c7f8  0096c000
0096c7fc  0096c824 ; a pointer to an exception context
0096c800  0096c808
0096c804  0096c824
0096c808  c0000005
0096c80c  00000000
0096c810  00000000
0096c814  77bd8df3 msvcrt!wcschr+0×15
0096c818  00000002
0096c81c  00000000
0096c820  01031000
0096c824  0001003f
0096c828  00000000
0096c82c  00000000
0096c830  00000000
0096c834  00000000
0096c838  00000000
0096c83c  00000000

A second parameter to both functions is a pointer to a so called exception context (processor state when an exception occurred). We can use .cxr command to change thread execution context to what it was at the time of exception:

After changing the context we can see thread stack prior to that exception:

0:000> kL
ChildEBP RetAddr
0096caf0 67b11808 msvcrt!wcschr+0×15
0096cb10 67b1194d component2!function1+0×50
0096cb24 67b11afb component2!function2+0×1a
0096eb5c 67b11e10 component2!function3+0×39
0096ed94 67b14426 component2!function4+0×155
0096fdc0 67b164b7 component2!function5+0×3b
0096fdcc 00402831 component2!function6+0×5b
0096feec 0096ff14 program!function+0×1d1
0096ffec 00000000 kernel32!BaseThreadStart+0×34

We see that the exception happened when component2 was searching a Unicode string for a character (wcschr). Most likely the string was not zero terminated:

To summarize and show you the common exception handling path in user space here is another thread stack taken from a different dump:

ntdll!KiFastSystemCallRet
ntdll!NtWaitForMultipleObjects+0xc
kernel32!UnhandledExceptionFilter+0×746
kernel32!_except_handler3+0×61
ntdll!ExecuteHandler2+0×26
ntdll!ExecuteHandler+0×24
ntdll!RtlDispatchException+0×91
ntdll!KiUserExceptionDispatcher+0xe
ntdll!RtlpCoalesceFreeBlocks+0×36e ; crash is here
ntdll!RtlFreeHeap+0×38e
msvcrt!free+0xc3
msvcrt!_freefls+0×124
msvcrt!_freeptd+0×27
msvcrt!__CRTDLL_INIT+0×1da
ntdll!LdrpCallInitRoutine+0×14
ntdll!LdrShutdownThread+0xd2
kernel32!ExitThread+0×2f
kernel32!BaseThreadStart+0×39

When RtlpCoalesceFreeBlocks (this function compacts heap and it is called from RtlFreeHeap) does an illegal memory access then this exception is first processed in kernel and because it happened in user space and mode the execution is transferred to RtlDispatchException which searches for exception handler and in this case there is a default one installed: UnhandledExceptionFilter.

If you see this function on call stack you can also manually get an exception context and a thread stack leading to it like in this example below taken from other dump:

The most likely reason of this crash is an instance of Dynamic Memory Corruption pattern - heap corruption.

- Dmitry Vostokov -

18 Responses to “Crash Dump Analysis Patterns (Part 8)”

  1. Tal Rosen Says:

    I just got a similar dump.
    The event log had no ‘application error’ recorded.
    Lucky for us a minidump was generated. Opening the dump you see just two threads, one thread shows the ExitProcess call stack.
    The cause was a 3rdParty library, calling abort() or exit() when encountering a memory corruption. Their bad exit, raised an exception in Microsoft DB DLL, this is how we got the mini dump.

  2. Dmitry Vostokov Says:

    We can also search for exception codes like c0000005 using scripts to dump raw stack data:

    http://www.dumpanalysis.org/blog/index.php/2007/08/28/raw-stack-dump-from-all-threads/
    http://www.dumpanalysis.org/blog/index.php/2007/12/24/raw-stack-dump-of-all-threads-part-2/

    For example:

    007cfa40 017d0000
    007cfa44 007cfd90
    007cfa48 7c82855e ntdll!KiUserExceptionDispatcher+0xe
    007cfa4c 7c826d9b ntdll!NtContinue+0xc
    007cfa50 7c82856c ntdll!KiUserExceptionDispatcher+0x1c
    007cfa54 007cfa78
    007cfa58 00000000
    007cfa5c c0000005
    007cfa60 00000000
    007cfa64 00000000
    007cfa68 0100e076 component!foo+0x1c4
    007cfa6c 00000002
    007cfa70 00000001
    007cfa74 00000000
    007cfa78 0001003f
    007cfa7c 00000003
    007cfa80 000000b0
    007cfa84 00000001
    007cfa88 00000000
    007cfa8c 00000000
    007cfa90 00000155
    007cfa94 ffff027f
    007cfa98 ffff0000
    007cfa9c ffffffff
    007cfaa0 00000000
    007cfaa4 00000000
    007cfaa8 00000000
    007cfaac ffff0000
    007cfab0 00000000
    007cfab4 00000000
    007cfab8 00000000

    1: kd> .cxr 007cfa78
    eax=01073bb0 ebx=7ffd9000 ecx=00000050 edx=01073bb0 esi=000003e5 edi=00000000
    eip=0100e076 esp=007cfd44 ebp=007cfd90 iopl=0 nv up ei pl zr na pe nc
    cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
    component!foo+0x1c4:
    001b:0100e076 891a mov dword ptr [edx],ebx ds:0023:01073bb0=????????

  3. KappA Says:

    Dmitry -

    Yet another live-saver tip! This has helped me with a critical issue today! Many thanks for this wealth of knowledge.

    KappA

  4. Dmitry Vostokov Says:

    Thanks! The presence of unloaded fault handling modules can be the sign of hidden exceptions too

    Unloaded modules:
    697b0000 697c7000 faultrep.dll
    Timestamp: Fri Mar 25 02:11:44 2005 (42437360)
    Checksum: 0001DC38

  5. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 60) Says:

    […] Residue patterns and one of them is Exception Handling Residue we can use to check for hidden exceptions and differentiate between 1st and 2nd chance exceptions. Code residues are very powerful in […]

  6. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 59b) Says:

    […] see exception dispatching calls highlighted above. One of their parameters is an exception record and we try to get […]

  7. Crash Dump Analysis » Blog Archive » Heuristic Stack Trace in WinDbg 6.9.3.113 Says:

    […] another 64-bit example of Hidden Exception pattern where looking at raw stack data helps in problem identification. Opening the dump in […]

  8. Crash Dump Analysis » Blog Archive » WinDbg shortcuts: .exptr Says:

    […] looking at hidden exceptions and manual crash dumps we need information from this structure and this command provides a […]

  9. !analyze -v : Crash Dump Analysis Patterns (Part 8) Says:

    […] 원문 http://www.dumpanalysis.org/blog/index.php/2007/02/02/crash-dump-analysis-patterns-part-8/ […]

  10. Crash Dump Analysis » Blog Archive » Manual dump, virtualized process, stack trace collection, multiple exceptions, optimized code, wild code pointer, incorrect stack trace and hidden exception: pattern cooperation Says:

    […] or the case of incorrect stack trace. However using techniques to get exception context from hidden exceptions we get the following stack […]

  11. Crash Dump Analysis » Blog Archive » WOW64 process, NULL data pointer, stack overflow, main thread, incorrect stack trace, nested exceptions, hidden exception, manual dump, multiple exceptions and virtualized system: pattern cooperation Says:

    […] addresses we used to try for .exr and .cxr commands in hidden exception pattern are beyond user space and we therefore conclude that somehow such structures or pointers […]

  12. Crash Dump Analysis » Blog Archive » Statement current, coupled processes, wait chain, spiking thread, hidden exception, and not my version: memory dump and trace analysis pattern cooperation Says:

    […] Looking at the raw stack data (using !teb and dds WinDbg commands) we see a hidden processed exception: […]

  13. Crash Dump Analysis » Blog Archive » Icons for Memory Dump Analysis Patterns (Part 12) Says:

    […] we introduce an icon for Hidden Exception […]

  14. Crash Dump Analysis » Blog Archive » Strong process coupling, stack trace collection, critical section coruption and wait chains, message box, self-diagnosis and hidden exception and dynamic memory corruption: pattern cooperation Says:

    […] was an exception indeed diagnosed by FilterException call. The exception is probably hidden somewhere on the raw […]

  15. Crash Dump Analysis » Blog Archive » Crash Dump Analysis Patterns (Part 105) Says:

    […] see any exception thread it doesn’t mean that no exception had occurred. There could be hidden exceptions on raw stack […]

  16. Crash Dump Analysis » Blog Archive » Coupled processes, wait chains, message box, waiting thread time, paged out data, incorrect stack trace, hidden exception, unknown component and execution residue: pattern cooperation Says:

    […] We also see that stack trace is incorrect and we try to reconstruct the point of exception by looking at thread raw stack and searching for any hidden exception:  […]

  17. Dmitry Vostokov Says:

    Sometimes we can spot 0001003f and its address can be the beginning of a context record:

    […]
    0070f668 00000000
    0070f66c 00000000
    0070f670 00000000
    0070f674 00000000
    0070f678 00000000
    0070f67c 0001003f
    […]

  18. Dmitry Vostokov Says:

    On Windows 10 RSP below KiUserExceptionDispatch can be used as an address for .cxr command:

    […]
    00000023`d432f4e8 00000023`d3e3c190
    00000023`d432f4f0 00000000`00000000
    00000023`d432f4f8 00007ffa`e5c5577a ntdll!KiUserExceptionDispatch+0×3a
    00000023`d432f500 00000000`00000000
    00000023`d432f508 00000000`00000810
    […]

    0:001> .cxr 00000023`d432f500
    rax=0000000000000000 rbx=0000000000000000 rcx=00007ff676f399b0
    rdx=0000000000000000 rsi=00000023d3e3c190 rdi=00007ff676f211e0
    rip=00007ff676f2120d rsp=00000023d432fc30 rbp=0000000000000000
    r8=00006f6d5c4f5ead r9=0000000000000032 r10=0000000000000032
    r11=00000023d432f960 r12=0000000000000000 r13=0000000000000000
    r14=0000000000000000 r15=0000000000000000
    iopl=0 nv up ei pl nz na po nc
    cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
    Application+0×120d:
    00007ff6`76f2120d c70000000000 mov dword ptr [rax],0 ds:00000000`00000000=????????

Leave a Reply