Virtualized process, incorrect stack trace, stack trace collection, multiple exceptions, optimized code and C++ exception: pattern cooperation

It happened to me that I accidentally opened a YouTube page and it started playing video so I tried to either stop it or pushed the back browser button. Then immediately WER dialog appeared ruining my intricate multi-tab window. Nevertheless I looked at this event from the positive perspective as an opportunity to analyze a dump file and write a case study. I saved the crash dump using Windows Server 2008 Task Manager for later postmortem analysis.

Today I had a chance to look at it. When we open the dump file we see that it is the dump of a WOW64 process:

0:000> !analyze -v

EXCEPTION_RECORD:  ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000000000000
   ExceptionCode: 80000003 (Break instruction exception)
  ExceptionFlags: 00000000
NumberParameters: 0

STACK_TEXT: 
00000000`0016ed78 00000000`75fcab46 : 00000000`77ddc990 00000000`0016fd20 00000000`75efa2b1 00000000`0016f760 : wow64cpu!WaitForMultipleObjects32+0×3a
00000000`0016ee20 00000000`75fca14c : 00000000`00000000 00000000`00000000 00000000`75fc3258 00000000`7ffe0030 : wow64!RunCpuSimulation+0xa
00000000`0016ee50 00000000`77d052d3 : 00000000`77cd0000 00000000`00000000 00000000`77de61b0 00000000`77cd0000 : wow64!Wow64LdrpInitialize+0×4b4

00000000`0016f3b0 00000000`77d05363 : 00000000`00000000 00000000`00000000 00000000`7efdf000 00000000`77d07ca7 : ntdll!LdrpInitializeProcess+0×14ac
00000000`0016f660 00000000`77cf85ce : 00000000`0016f760 00000000`00000000 00000000`7efdf000 00000000`00000000 : ntdll! ?? ::FNODOBFM::`string’+0×1ff19
00000000`0016f710 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!LdrInitializeThunk+0xe

FOLLOWUP_IP:
wow64cpu!WaitForMultipleObjects32+3a
00000000`75ec374f 418bbda0000000  mov     edi,dword ptr [r13+0A0h]

MODULE_NAME: wow64cpu 

Then we load a WOW64 extension that allows us to analyze 32-bit parts of saved 64-bit dumps and we retry the default analysis:

0:000> .load wow64exts; .effmach x86
Effective machine: x86 compatible (x86)

0:000:x86> !analyze -v

IP_ON_HEAP:  ffffffffc0d84c02
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

STACK_TEXT: 
00000000`003b0068 iexplore!__dyn_tls_init_callback <PERF> +0×0
ffffffff`c0d84c02 unknown+0×0
ffffffff`aacf27dc unknown+0×0
ffffffff`c1d84c02 unknown+0×0
ffffffff`b3cf38dc unknown+0×0
ffffffff`c2d84c02 unknown+0×0
00000000`3ecf38dc unknown+0×0
[…]
00000000`7aca19dd unknown+0×0
ffffffff`e0d84c02 unknown+0×0
ffffffff`a7ca19dd unknown+0×0
ffffffff`

FOLLOWUP_IP:
iexplore!__dyn_tls_init_callback <PERF> (iexplore+0x80068)+0
003b0068 70f7            jo      iexplore!__dyn_tls_init_callback <PERF> (iexplore+0x80061) (003b0061)

Clearly the shown above stack trace is incorrect but at the same time the first thread stack looks consistent:

0:000:x86> k 100
ChildEBP RetAddr 
0051d930 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15
0051d9cc 77888f76 kernel32!WaitForMultipleObjectsEx+0x11d
0051da20 7617ab28 user32!RealMsgWaitForMultipleObjectsEx+0x14d
0051da48 7617ac88 ole32!CCliModalLoop::BlockFn+0x97
0051da70 7617ad74 ole32!ModalLoop+0x5b
0051da80 76287c28 ole32!SwitchSTA+0x21
0051daac 762889d4 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x127
0051db8c 7617ad2e ole32!CRpcChannelBuffer::SendReceive2+0xef
0051dba8 7617ace0 ole32!CCliModalLoop::SendReceive+0x1e
0051dc20 7619e688 ole32!CAptRpcChnl::SendReceive+0x73
0051dc74 779f1074 ole32!CCtxComChnl::SendReceive+0x1c5
0051dc90 779f102b rpcrt4!NdrProxySendReceive+0x49
0051dc9c 779f0146 rpcrt4!NdrpProxySendReceive+0xb
0051e0a4 779f11ee rpcrt4!NdrClientCall2+0x18f
0051e0c8 779649c2 rpcrt4!ObjectStublessClient+0x90
0051e0d8 74bc86e3 rpcrt4!ObjectStubless+0xf
WARNING: Frame IP not in any known module. Following frames may be wrong.
0051e0e4 74bda9bb ieframe+0x1986e3
0051e0f0 74b020d9 ieframe+0x1aa9bb
0051e164 74adbb6a ieframe+0xd20d9
0051e18c 77888807 ieframe+0xabb6a
0051e1b8 77888962 user32!InternalCallWinProc+0x23
0051e230 7788c4b6 user32!UserCallWinProcCheckWow+0x109
0051e274 7788c517 user32!SendMessageWorker+0x55b
0051e298 75b3f74a user32!SendMessageW+0x7f
0051e2c0 75b3f63b comctl32!CToolbar::TBOnLButtonUp+0x12f
0051e368 75b37020 comctl32!CToolbar::ToolbarWndProc+0xaa9
0051e388 77888807 comctl32!CToolbar::s_ToolbarWndProc+0x9b
0051e3b4 77888962 user32!InternalCallWinProc+0x23
0051e42c 77888aad user32!UserCallWinProcCheckWow+0x109
0051e490 77888b00 user32!DispatchMessageWorker+0x380
0051e4a0 74ae07ce user32!DispatchMessageW+0xf
0051e4d4 74a7ffce ieframe+0xb07ce
0051e4e0 74a6f579 ieframe+0x4ffce
0051e504 74a6f4c7 ieframe+0x3f579
0051f574 74a5d1ba ieframe+0x3f4c7
0051f7a4 003333c3 sxs!_hmod__RPCRT4_dll <PERF> (sxs+0x8d1ba)
0051fbe8 0033325a iexplore!wWinMain+0x27b
0051fc7c 775de4a5 iexplore!_initterm_e+0x1b1
0051fc88 77eecfed kernel32!BaseThreadInitThunk+0xe
0051fcc8 77eed1ff ntdll_77e70000!__RtlUserThreadStart+0x23
0051fce0 00000000 ntdll_77e70000!_RtlUserThreadStart+0x1b

We look at the stack trace collection to find another exception and we see it indeed on 7th stack trace:

0:000:x86> ~*k 100

[...]

   7  Id: b44.b1c Suspend: 0 Teb: 7efad000 Unfrozen
ChildEBP RetAddr 
05baee9c 7756e91a ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baef38 775649d9 kernel32!WaitForMultipleObjectsEx+0x11d
05baef54 7761573d kernel32!WaitForMultipleObjects+0x18
05baefc0 77615969 kernel32!WerpReportFaultInternal+0x16d
05baefd4 775ec66f kernel32!WerpReportFault+0×70
05baf060 77eed03e kernel32!UnhandledExceptionFilter+0×1b5
05baf068 77ebf2d0 ntdll_77e70000!__RtlUserThreadStart+0×6f
05baf07c 77f229b3 ntdll_77e70000!_EH4_CallFilterFunc+0×12
05baf0a4 77e93099 ntdll_77e70000!_except_handler4+0×8e
05baf0c8 77e9306b ntdll_77e70000!ExecuteHandler2+0×26
05baf178 77e92eff ntdll_77e70000!ExecuteHandler+0×24
05baf198 7757f328 ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf4fc 7155dead kernel32!RaiseException+0×58

WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d Flash10b!DllUnregisterServer+0×1adbe0
05baf550 7150ca25 Flash10b!DllUnregisterServer+0×1aa2d0
05baf564 7150b5b5 Flash10b!DllUnregisterServer+0×15c758
05baf588 7150b90a Flash10b!DllUnregisterServer+0×15b2e8
00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

[...]

  97  Id: b44.ff4 Suspend: 1 Teb: 7ee9f000 Unfrozen
ChildEBP RetAddr 
7387facc 77561270 ntdll_77e70000!ZwWaitForSingleObject+0x15
7387fb3c 775611d8 kernel32!WaitForSingleObjectEx+0xbe
7387fb50 713e0a11 kernel32!WaitForSingleObject+0x12
WARNING: Stack unwind information not available. Following frames may be wrong.
7387fb74 713e0d34 Flash10b!DllUnregisterServer+0x30744
7387fb78 775de4a5 Flash10b!DllUnregisterServer+0x30a67
7387fb84 77eecfed kernel32!BaseThreadInitThunk+0xe
7387fbc4 77eed1ff ntdll_77e70000!__RtlUserThreadStart+0x23
7387fbdc 00000000 ntdll_77e70000!_RtlUserThreadStart+0x1b

Then we switch to this thread and try to find and set the new exception context:

0:000:x86> ~7s
ntdll_77e70000!NtWaitForMultipleObjects+0x15:
77e90bc5 c21400          ret     14h

0:007:x86> kv
ChildEBP RetAddr  Args to Child             
05baee9c 7756e91a 00000002 05baeeec 00000001 ntdll_77e70000!NtWaitForMultipleObjects+0x15
05baef38 775649d9 05baeeec 05baef88 00000000 kernel32!WaitForMultipleObjectsEx+0x11d
05baef54 7761573d 00000002 05baef88 00000000 kernel32!WaitForMultipleObjects+0x18
05baefc0 77615969 05baf090 00000001 00000001 kernel32!WerpReportFaultInternal+0x16d
05baefd4 775ec66f 05baf090 00000001 19e00de6 kernel32!WerpReportFault+0x70
05baf060 77eed03e 00000000 77ebf2d0 00000000 kernel32!UnhandledExceptionFilter+0×1b5
05baf068 77ebf2d0 00000000 05bafacc 77e9da38 ntdll_77e70000!__RtlUserThreadStart+0×6f
05baf07c 77f229b3 00000000 00000000 00000000 ntdll_77e70000!_EH4_CallFilterFunc+0×12
05baf0a4 77e93099 fffffffe 05bafabc 05baf1e0 ntdll_77e70000!_except_handler4+0×8e
05baf0c8 77e9306b 05baf190 05bafabc 05baf1e0 ntdll_77e70000!ExecuteHandler2+0×26
05baf178 77e92eff 01baf190 05baf1e0 05baf190 ntdll_77e70000!ExecuteHandler+0×24
05baf198 7757f328 00000003 19930520 05baf544 ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf4fc 7155dead e06d7363 00000001 00000003 kernel32!RaiseException+0×58
WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d 05baf544 715e7954 715e5ac0 Flash10b!DllUnregisterServer+0×1adbe0
05baf550 7150ca25 002c1f7c 4237d4f8 4237c598 Flash10b!DllUnregisterServer+0×1aa2d0
05baf564 7150b5b5 002c1f60 00000010 423d0b28 Flash10b!DllUnregisterServer+0×15c758
05baf588 7150b90a 4237cb38 7150ba0b 00000000 Flash10b!DllUnregisterServer+0×15b2e8
00000000 00000000 00000000 00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

Unfortunately a pointer to an exception pointers structure is NULL (probably because of optimized code) and we cannot use .exptr command. However we can still use a technique described in Hidden Exception pattern using 32-bit raw stack to set the context and display the exception record:

0:007:x86> !wow64exts.info

PEB32: 0x7efde000
PEB64: 0x7efdf000

Wow64 information for current thread:

TEB32: 0x7efaf000
TEB64: 0x7efad000

32 bit, StackBase   : 0×5bb0000
        StackLimit  : 0×5b95000

        Deallocation: 0×5ab0000

64 bit, StackBase   : 0x30ffd20
        StackLimit  : 0x30f8000
        Deallocation: 0x30c0000

0:007:x86> dps 0×5b95000 0×5bb0000
05b95000  00000000
05b95004  00000000
05b95008  00000000
[…]
05baf160  00000009
05baf164  00000000
05baf168  00000000
05baf16c  05bb0000
05baf170  05b95000
05baf174  00000000
05baf178  05baf4fc
05baf17c  77e92eff ntdll_77e70000!KiUserExceptionDispatcher+0xf
05baf180  01baf190
05baf184  05baf1e0 ; .cxr
05baf188  05baf190 ; .exr
05baf18c  05baf1e0
05baf190  e06d7363
[…]

0:007:x86> .cxr 05baf1e0
eax=05baf4ac ebx=002c1f60 ecx=00000003 edx=00000000 esi=716d9c54 edi=00000010
eip=7757f328 esp=05baf4ac ebp=05baf4fc iopl=0 nv up ei pl nz na pe nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206
kernel32!RaiseException+0×58:
7757f328 c9              leave

0:007:x86> kv
  *** Stack trace for last set context - .thread/.cxr resets it
ChildEBP RetAddr  Args to Child             
05baf4fc 7155dead e06d7363 00000001 00000003 kernel32!RaiseException+0×58
WARNING: Stack unwind information not available. Following frames may be wrong.
05baf534 7155a59d 05baf544 715e7954 715e5ac0 Flash10b!DllUnregisterServer+0×1adbe0
05baf550 7150ca25 002c1f7c 4237d4f8 4237c598 Flash10b!DllUnregisterServer+0×1aa2d0
05baf564 7150b5b5 002c1f60 00000010 423d0b28 Flash10b!DllUnregisterServer+0×15c758
05baf588 7150b90a 4237cb38 7150ba0b 00000000 Flash10b!DllUnregisterServer+0×15b2e8
00000000 00000000 00000000 00000000 00000000 Flash10b!DllUnregisterServer+0×15b63d

0:007:x86> .exr 05baf190
ExceptionAddress: 000000007757f328 (kernel32!RaiseException+0×0000000000000058)
   ExceptionCode: e06d7363 (C++ EH exception)
  ExceptionFlags: 00000001
NumberParameters: 3
   Parameter[0]: 0000000019930520
   Parameter[1]: 0000000005baf544
   Parameter[2]: 00000000715e7954

We double check that Flash10b code raised the exception indeed:

0:007:x86> .asm no_code_bytes
Assembly options: no_code_bytes

0:007:x86> ub 7155dead
Flash10b!DllUnregisterServer+0×1adbc4:
7155de91 je      Flash10b!DllUnregisterServer+0×1adbcd (7155de9a)
7155de93 mov     dword ptr [ebp-0Ch],1994000h
7155de9a lea     eax,[ebp-0Ch]
7155de9d push    eax
7155de9e push    dword ptr [ebp-10h]
7155dea1 push    dword ptr [ebp-1Ch]
7155dea4 push    dword ptr [ebp-20h]
7155dea7 call    dword ptr [Flash10b!DllUnregisterServer+0×1ba0a3 (7156a370)]
 

0:007:x86> dps 7156a370
7156a370  7757f2cf kernel32!RaiseException
7156a374  77562728 kernel32!WideCharToMultiByte
7156a378  7756c60b kernel32!GlobalFree
7156a37c  77562ddf kernel32!GetCurrentThread
7156a380  775739fe kernel32!SetThreadAffinityMask
7156a384  775654c9 kernel32!VirtualQuery
7156a388  77562e9c kernel32!IsDBCSLeadByte
7156a38c  77561ce5 kernel32!GetACP
7156a390  77565ed6 kernel32!GetCPInfo
7156a394  77561b36 kernel32!MultiByteToWideChar
7156a398  7756eb64 kernel32!ResetEvent
7156a39c  77562c88 kernel32!CreateEventA
7156a3a0  77561a3a kernel32!CloseHandle
7156a3a4  775649c1 kernel32!WaitForMultipleObjects
7156a3a8  77562c64 kernel32!SetEvent
7156a3ac  77560e18 kernel32!InterlockedExchange
7156a3b0  77560e2c kernel32!InterlockedCompareExchange
7156a3b4  77560c79 kernel32!Sleep
7156a3b8  77e935c0 ntdll_77e70000!RtlLeaveCriticalSection
7156a3bc  77e93580 ntdll_77e70000!RtlEnterCriticalSection
7156a3c0  77ea27fe ntdll_77e70000!RtlDeleteCriticalSection
7156a3c4  77ea2512 ntdll_77e70000!RtlInitializeCriticalSection
7156a3c8  77568c28 kernel32!WriteConsoleA
7156a3cc  7757a6dc kernel32!SetStdHandle
7156a3d0  77568c0b kernel32!GetConsoleMode
7156a3d4  77568c75 kernel32!GetConsoleCP
7156a3d8  77564ab7 kernel32!GetStringTypeW
7156a3dc  77576a56 kernel32!GetStringTypeA
7156a3e0  77562d90 kernel32!InitializeCriticalSectionAndSpinCount
7156a3e4  775608d0 kernel32!GetSystemTimeAsFileTime
7156a3e8  77565c3b kernel32!GetEnvironmentStringsW
7156a3ec  77565c13 kernel32!FreeEnvironmentStringsA

I believe the appearance of “dead” in the return address 7155dead was just a coincidence.

So we have found the 3rd-party component that raised e06d7363 C++ exception. Actually Internet search shows that e06d7363 is quite common in many applications.

- Dmitry Vostokov @ DumpAnalysis.org -

Leave a Reply

You must be logged in to post a comment.