Crash Dump Analysis Patterns (Part 8)
Today I will talk about another pattern occurring frequently and I call it Hidden Exception. You run !analyze -v command and you don’t see an exception or you see only a breakpoint hit. In this case manual analysis is required. This happens sometimes because of another pattern: Multiple Exceptions. In other cases an exception happens and it is handled by an exception handler dismissing it and a process continues execution slowly accumulating corruption inside its data leading to a new crash or hang. Sometimes you see a process hanging during its termination like the case I present here.
We have a process dump with only one thread:
0:000> kv
ChildEBP RetAddr
0096fcdc 7c822124 ntdll!KiFastSystemCallRet
0096fce0 77e6baa8 ntdll!NtWaitForSingleObject+0xc
0096fd50 77e6ba12 kernel32!WaitForSingleObjectEx+0xac
0096fd64 67f016ce kernel32!WaitForSingleObject+0x12
0096fd78 7c82257a component!DllInitialize+0xc2
0096fd98 7c8118b0 ntdll!LdrpCallInitRoutine+0x14
0096fe34 77e52fea ntdll!LdrShutdownProcess+0x130
0096ff20 77e5304d kernel32!_ExitProcess+0x43
0096ff34 77bcade4 kernel32!ExitProcess+0x14
0096ff40 77bcaefb msvcrt!__crtExitProcess+0x32
0096ff70 77bcaf6d msvcrt!_cinit+0xd2
0096ff84 77bcb555 msvcrt!_exit+0x11
0096ffb8 77e66063 msvcrt!_endthreadex+0xc8
0096ffec 00000000 kernel32!BaseThreadStart+0x34
We can look at its raw stack and try to find the following address:
KiUserExceptionDispatcher
This function calls RtlDispatchException:
0:000> !teb
TEB at 7ffdc000
ExceptionList: 0096fd40
StackBase: 00970000
StackLimit: 0096a000
SubSystemTib: 00000000
FiberData: 00001e00
ArbitraryUserPointer: 00000000
Self: 7ffdc000
EnvironmentPointer: 00000000
ClientId: 00000858 . 000008c0
RpcHandle: 00000000
Tls Storage: 00000000
PEB Address: 7ffdd000
LastErrorValue: 0
LastStatusValue: c0000135
Count Owned Locks: 0
HardErrorMode: 0
0:000>dds 0096a000 00970000
...
...
...
0096c770 7c8140cc ntdll!RtlDispatchException+0x91
0096c774 0096c808
0096c778 0096ffa8
0096c77c 0096c824
0096c780 0096c7e4
0096c784 77bc6c74 msvcrt!_except_handler3
0096c788 00000000
0096c78c 0096c808
0096c790 01030064
0096c794 00000000
0096c798 00000000
0096c79c 00000000
0096c7a0 00000000
0096c7a4 00000000
0096c7a8 00000000
0096c7ac 00000000
0096c7b0 00000000
0096c7b4 00000000
0096c7b8 00000000
0096c7bc 00000000
0096c7c0 00000000
0096c7c4 00000000
0096c7c8 00000000
0096c7cc 00000000
0096c7d0 00000000
0096c7d4 00000000
0096c7d8 00000000
0096c7dc 00000000
0096c7e0 00000000
0096c7e4 00000000
0096c7e8 00970000
0096c7ec 00000000
0096c7f0 0096caf0
0096c7f4 7c82ecc6 ntdll!KiUserExceptionDispatcher+0xe
0096c7f8 0096c000
0096c7fc 0096c824 ; a pointer to an exception context
0096c800 0096c808
0096c804 0096c824
0096c808 c0000005
0096c80c 00000000
0096c810 00000000
0096c814 77bd8df3 msvcrt!wcschr+0×15
0096c818 00000002
0096c81c 00000000
0096c820 01031000
0096c824 0001003f
0096c828 00000000
0096c82c 00000000
0096c830 00000000
0096c834 00000000
0096c838 00000000
0096c83c 00000000
A second parameter to both functions is a pointer to a so called exception context (processor state when an exception occurred). We can use .cxr command to change thread execution context to what it was at the time of exception:
After changing the context we can see thread stack prior to that exception:
0:000> kL
ChildEBP RetAddr
0096caf0 67b11808 msvcrt!wcschr+0×15
0096cb10 67b1194d component2!function1+0×50
0096cb24 67b11afb component2!function2+0×1a
0096eb5c 67b11e10 component2!function3+0×39
0096ed94 67b14426 component2!function4+0×155
0096fdc0 67b164b7 component2!function5+0×3b
0096fdcc 00402831 component2!function6+0×5b
0096feec 0096ff14 program!function+0×1d1
0096ffec 00000000 kernel32!BaseThreadStart+0×34
We see that the exception happened when component2 was searching a Unicode string for a character (wcschr). Most likely the string was not zero terminated:
To summarize and show you the common exception handling path in user space here is another thread stack taken from a different dump:
ntdll!KiFastSystemCallRet
ntdll!NtWaitForMultipleObjects+0xc
kernel32!UnhandledExceptionFilter+0×746
kernel32!_except_handler3+0×61
ntdll!ExecuteHandler2+0×26
ntdll!ExecuteHandler+0×24
ntdll!RtlDispatchException+0×91
ntdll!KiUserExceptionDispatcher+0xe
ntdll!RtlpCoalesceFreeBlocks+0×36e ; crash is here
ntdll!RtlFreeHeap+0×38e
msvcrt!free+0xc3
msvcrt!_freefls+0×124
msvcrt!_freeptd+0×27
msvcrt!__CRTDLL_INIT+0×1da
ntdll!LdrpCallInitRoutine+0×14
ntdll!LdrShutdownThread+0xd2
kernel32!ExitThread+0×2f
kernel32!BaseThreadStart+0×39
When RtlpCoalesceFreeBlocks (this function compacts heap and it is called from RtlFreeHeap) does an illegal memory access then this exception is first processed in kernel and because it happened in user space and mode the execution is transferred to RtlDispatchException which searches for exception handler and in this case there is a default one installed: UnhandledExceptionFilter.
If you see this function on call stack you can also manually get an exception context and a thread stack leading to it like in this example below taken from other dump:
The most likely reason of this crash is an instance of Dynamic Memory Corruption pattern - heap corruption.
- Dmitry Vostokov -
February 8th, 2007 at 8:00 am
I just got a similar dump.
The event log had no ‘application error’ recorded.
Lucky for us a minidump was generated. Opening the dump you see just two threads, one thread shows the ExitProcess call stack.
The cause was a 3rdParty library, calling abort() or exit() when encountering a memory corruption. Their bad exit, raised an exception in Microsoft DB DLL, this is how we got the mini dump.
January 3rd, 2008 at 2:24 pm
We can also search for exception codes like c0000005 using scripts to dump raw stack data:
http://www.dumpanalysis.org/blog/index.php/2007/08/28/raw-stack-dump-from-all-threads/
http://www.dumpanalysis.org/blog/index.php/2007/12/24/raw-stack-dump-of-all-threads-part-2/
For example:
007cfa40 017d0000
007cfa44 007cfd90
007cfa48 7c82855e ntdll!KiUserExceptionDispatcher+0xe
007cfa4c 7c826d9b ntdll!NtContinue+0xc
007cfa50 7c82856c ntdll!KiUserExceptionDispatcher+0x1c
007cfa54 007cfa78
007cfa58 00000000
007cfa5c c0000005
007cfa60 00000000
007cfa64 00000000
007cfa68 0100e076 component!foo+0x1c4
007cfa6c 00000002
007cfa70 00000001
007cfa74 00000000
007cfa78 0001003f
007cfa7c 00000003
007cfa80 000000b0
007cfa84 00000001
007cfa88 00000000
007cfa8c 00000000
007cfa90 00000155
007cfa94 ffff027f
007cfa98 ffff0000
007cfa9c ffffffff
007cfaa0 00000000
007cfaa4 00000000
007cfaa8 00000000
007cfaac ffff0000
007cfab0 00000000
007cfab4 00000000
007cfab8 00000000
1: kd> .cxr 007cfa78
eax=01073bb0 ebx=7ffd9000 ecx=00000050 edx=01073bb0 esi=000003e5 edi=00000000
eip=0100e076 esp=007cfd44 ebp=007cfd90 iopl=0 nv up ei pl zr na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000246
component!foo+0x1c4:
001b:0100e076 891a mov dword ptr [edx],ebx ds:0023:01073bb0=????????
January 4th, 2008 at 1:29 am
Dmitry -
Yet another live-saver tip! This has helped me with a critical issue today! Many thanks for this wealth of knowledge.
KappA
January 7th, 2008 at 5:56 pm
Thanks! The presence of unloaded fault handling modules can be the sign of hidden exceptions too
Unloaded modules:
697b0000 697c7000 faultrep.dll
Timestamp: Fri Mar 25 02:11:44 2005 (42437360)
Checksum: 0001DC38
April 29th, 2008 at 11:28 am
[…] Residue patterns and one of them is Exception Handling Residue we can use to check for hidden exceptions and differentiate between 1st and 2nd chance exceptions. Code residues are very powerful in […]
June 12th, 2008 at 4:49 pm
[…] see exception dispatching calls highlighted above. One of their parameters is an exception record and we try to get […]
June 26th, 2008 at 2:11 am
[…] another 64-bit example of Hidden Exception pattern where looking at raw stack data helps in problem identification. Opening the dump in […]
July 26th, 2008 at 8:54 pm
[…] looking at hidden exceptions and manual crash dumps we need information from this structure and this command provides a […]
September 18th, 2008 at 6:24 am
[…] 원문 http://www.dumpanalysis.org/blog/index.php/2007/02/02/crash-dump-analysis-patterns-part-8/ […]
June 4th, 2009 at 10:34 am
[…] or the case of incorrect stack trace. However using techniques to get exception context from hidden exceptions we get the following stack […]
July 12th, 2009 at 5:59 pm
[…] addresses we used to try for .exr and .cxr commands in hidden exception pattern are beyond user space and we therefore conclude that somehow such structures or pointers […]
October 12th, 2009 at 7:11 pm
[…] Looking at the raw stack data (using !teb and dds WinDbg commands) we see a hidden processed exception: […]
March 25th, 2010 at 3:55 pm
[…] we introduce an icon for Hidden Exception […]
April 26th, 2010 at 8:00 pm
[…] was an exception indeed diagnosed by FilterException call. The exception is probably hidden somewhere on the raw […]
August 5th, 2010 at 8:07 pm
[…] see any exception thread it doesn’t mean that no exception had occurred. There could be hidden exceptions on raw stack […]
August 13th, 2010 at 7:09 pm
[…] We also see that stack trace is incorrect and we try to reconstruct the point of exception by looking at thread raw stack and searching for any hidden exception: […]
May 15th, 2015 at 3:16 pm
Sometimes we can spot 0001003f and its address can be the beginning of a context record:
[…]
0070f668 00000000
0070f66c 00000000
0070f670 00000000
0070f674 00000000
0070f678 00000000
0070f67c 0001003f
[…]
May 7th, 2016 at 3:10 pm
On Windows 10 RSP below KiUserExceptionDispatch can be used as an address for .cxr command:
[…]
00000023`d432f4e8 00000023`d3e3c190
00000023`d432f4f0 00000000`00000000
00000023`d432f4f8 00007ffa`e5c5577a ntdll!KiUserExceptionDispatch+0×3a
00000023`d432f500 00000000`00000000
00000023`d432f508 00000000`00000810
[…]
0:001> .cxr 00000023`d432f500
rax=0000000000000000 rbx=0000000000000000 rcx=00007ff676f399b0
rdx=0000000000000000 rsi=00000023d3e3c190 rdi=00007ff676f211e0
rip=00007ff676f2120d rsp=00000023d432fc30 rbp=0000000000000000
r8=00006f6d5c4f5ead r9=0000000000000032 r10=0000000000000032
r11=00000023d432f960 r12=0000000000000000 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
Application+0×120d:
00007ff6`76f2120d c70000000000 mov dword ptr [rax],0 ds:00000000`00000000=????????