Stack trace collection, blocked thread and coupled processes: pattern cooperation

This is a continuation of the story of a process hang simulation where I made all threads in IE7 instance frozen. I left that process frozen after my experiments and later tried to reply to one e-mail using PHP-based browser client running in another IE7 process. And I found that the mouse click on ”Reply” button didn’t bring out any GUI response. I tried to close IE7 instance: all tabs closed except the one that was hanging. So I dumped the process and found a blocking thread inside waiting for an RPC call. I made all threads in the first IE7 process unfrozen and the second hanging IE7 process immediately exited. Instead of digging into the dump further I decided to repeat the problem. First I launched the fresh instance of IE7 and opened my e-mail client. After clicking on “Reply” with success I dumped the process using Vista Task Manager and renamed the resulted memory dump as iexplore2.dmp. Then I launched another IE instance and made all threads frozen. Then I came back to the first instance of IE7 and tried to do ”Reply” again. After waiting for about 10 minutes for any response I dumped the process again and renamed the dump file as iexplore3.dmp. Comparing thread stack traces from both dump files showed one difference: the blocked OLE/RPC thread obviously processing some JavaScript code:

[Before hang]

0:000> ~[0n36]k 100
ChildEBP RetAddr 
161bfc00 76b60dde ntdll!KiFastSystemCallRet
161bfc04 705e41a1 user32!NtUserWaitMessage+0xc
161bfc68 76c24911 ieframe!CTabWindow::_TabWindowThreadProc+0x2d0
161bfc74 76fee4b6 kernel32!BaseThreadInitThunk+0xe
161bfcb4 76fee489 ntdll!__RtlUserThreadStart+0x23
161bfccc 00000000 ntdll!_RtlUserThreadStart+0x1b

[After hang]

0:000> ~[0n36]k 100
ChildEBP RetAddr 
WARNING: Stack unwind information not available. Following frames may be wrong.
161bc27c 76b60208 ntdll!KiFastSystemCallRet
161bc2d0 767fab28 user32!RealMsgWaitForMultipleObjectsEx+0x13c
161bc2f8 767fac88 ole32!CCliModalLoop::BlockFn+0×97
161bc320 76907b73 ole32!ModalLoop+0×5b
161bc33c 76908b68 ole32!ThreadSendReceive+0×12c
161bc364 769089d4 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0×194
161bc444 767fad2e ole32!CRpcChannelBuffer::SendReceive2+0xef
161bc460 767face0 ole32!CCliModalLoop::SendReceive+0×1e
161bc4d8 7681e688 ole32!CAptRpcChnl::SendReceive+0×73
161bc52c 7667264e ole32!CCtxComChnl::SendReceive+0×1c5
161bc544 766726af rpcrt4!NdrProxySendReceive+0×43
161bc9c8 76f3ad86 rpcrt4!NdrProxySendReceive+0xa4

161bc9e0 76f3ad58 oleaut32!IEnumVARIANT_RemoteNext_Proxy+0×19
161bc9fc 6c1f2a7b oleaut32!IEnumVARIANT_Next_Proxy+0×1c
161bca5c 6c1f2b9c mshtml!SearchBrowsersForWindow+0×1bd
161bca84 6c1a2932 mshtml!GetTargetWindow+0×53
161bcabc 6c1b1300 mshtml!CWindow::FindWindowByName+0xe1
161bcad4 706498d4 mshtml!CWindow::FindWindowByName+0×17
161bcaf4 70649e5a ieframe+0×1198d4
161bcb48 70649ff6 ieframe+0×119e5a
161bcbac 70649b82 ieframe+0×119ff6
161bcbe0 6c189f9b ieframe+0×119b82
161bcc10 6c119cba mshtml!COmWindowProxy::FindFrame+0×5c
161bcc44 6c18be8e mshtml!COmWindowProxy::AccessAllowedToFrame+0×7f
161bccb4 6c1c4a2e mshtml!COmWindowProxy::open+0×15b
161bcd1c 6c0371b6 mshtml!Method_IDispatchpp_oDoBSTR_oDoBSTR_oDoBSTR_oDoVARIANTBOOL+0xeb
161bcdb4 6c037493 mshtml!CBase::ContextInvokeEx+0×4ef
161bcde0 6c037607 mshtml!CBase::InvokeEx+0×25
161bce48 6c0374c2 mshtml!COmWindowProxy::InvokeEx+0×297
161bce70 6c5b348e mshtml!COmWindowProxy::subInvokeEx+0×26
161bcea8 6c5b33fe jscript!IDispatchExInvokeEx2+0xac
161bcee0 6c5b3e09 jscript!IDispatchExInvokeEx+0×56
161bcf50 6c5b30eb jscript!InvokeDispatchEx+0×78
161bcf98 6c5b18ab jscript!VAR::InvokeByName+0xba
161bcfd8 6c5b2109 jscript!VAR::InvokeDispName+0×43
161bcffc 6c5b28d8 jscript!VAR::InvokeByDispID+0xb9
161bd0b4 6c5b1019 jscript!CScriptRuntime::Run+0×167f
161bd0cc 6c5b2aa8 jscript!ScrFncObj::Call+0×8d
161bd158 6c5b00f2 jscript!NameTbl::InvokeInternal+0xe0
161bd184 6c5b28d8 jscript!VAR::InvokeByDispID+0xfd
161bd23c 6c5b1019 jscript!CScriptRuntime::Run+0×167f
161bd254 6c5b1b7f jscript!ScrFncObj::Call+0×8d
161bd2c4 6c59f9d2 jscript!CSession::Execute+0xa7
161bd314 6c59fdf7 jscript!COleScript::ExecutePendingScripts+0×147
161bd378 6c59fc46 jscript!COleScript::ParseScriptTextCore+0×243
161bd3a4 6bfcca36 jscript!COleScript::ParseScriptText+0×2b

161bd404 6c1b1931 mshtml!CScriptCollection::ParseScriptText+0×240
161bf48c 6c12adae mshtml!CWindow::ExecuteScriptUri+0×197
161bf4cc 6c1b2f77 mshtml!CWindow::NavigateEx+0×50
161bf530 6c1b3372 mshtml!CDoc::ExecuteScriptUri+0×1f7
161bf560 6c27b8ac mshtml!CDoc::ExecuteScriptURL+0×4b
161bf5a8 6c27a54c mshtml!CHyperlink::ClickAction+0×1a9
161bf5b8 6c121847 mshtml!CAnchorElement::ClickAction+0×10
161bf5e4 6c07a7ef mshtml!CElement::DoClick+0×121
161bf610 6c07a5bd mshtml!CAnchorElement::DoClick+0×4d
161bf69c 6c07f680 mshtml!CDoc::PumpMessage+0xcbd
161bf7e8 6c12a7e0 mshtml!CDoc::OnMouseMessage+0×3d7
161bf90c 6c039a11 mshtml!CDoc::OnWindowMessage+0×8f7
161bf938 76b5f8d2 mshtml!CServer::WndProc+0×78
161bf964 76b5f794 user32!InternalCallWinProc+0×23
161bf9dc 76b606f6 user32!UserCallWinProcCheckWow+0×14b
161bfa0c 76b6069c user32!CallWindowProcAorW+0×97
161bfa2c 6baad980 user32!CallWindowProcW+0×1b
161bfa98 6baa104a GoogleToolbarDynamic_F423308312A7B033+0×3d980
161bfabc 6bb67e57 GoogleToolbarDynamic_F423308312A7B033+0×3104a
161bfae8 76b5f8d2 GoogleToolbarDynamic_F423308312A7B033+0xf7e57
161bfb14 76b5f794 user32!InternalCallWinProc+0×23
161bfb8c 76b60008 user32!UserCallWinProcCheckWow+0×14b
161bfbf0 76b60060 user32!DispatchMessageWorker+0×322
161bfc00 705e42c1 user32!DispatchMessageW+0xf
161bfc68 76c24911 ieframe+0xb42c1
161bfc74 76fee4b6 kernel32!BaseThreadInitThunk+0xe

Upon seeing SendReceive2 on the latter stack trace I recalled that it is possible to know the target process PID: In Search of Lost CID. The same procedure applied here reveals PID = 0xdec:

0:000> ~[0n36]kv 9
ChildEBP RetAddr  Args to Child             
WARNING: Stack unwind information not available. Following frames may be wrong.
161bc27c 76b60208 161bc230 161bc2a4 00000000 ntdll!KiFastSystemCallRet
161bc2d0 767fab28 00000000 161bc318 00000000 user32!RealMsgWaitForMultipleObjectsEx+0x13c
161bc2f8 767fac88 161bc318 00000000 161bc328 ole32!CCliModalLoop::BlockFn+0x97
161bc320 76907b73 00000000 00000000 161bc42c ole32!ModalLoop+0x5b
161bc33c 76908b68 00000000 161bc440 00000000 ole32!ThreadSendReceive+0x12c
161bc364 769089d4 161bc42c 00000000 161bc488 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x194
161bc444 767fad2e 14f75040 161bc56c 161bc550 ole32!CRpcChannelBuffer::SendReceive2+0xef
161bc460 767face0 161bc56c 161bc550 00000000 ole32!CCliModalLoop::SendReceive+0×1e
161bc4d8 7681e688 14f75040 161bc56c 161bc550 ole32!CAptRpcChnl::SendReceive+0×73

Note: 14f75040 is 00000000 in iexplore3.dmp from ftp because the dumps were stripped from almost all process data and contain only values necessary to reconstruct stack traces. So you won’t be able to extract correct raw stack data from them.

0:000> ddp 14f75040
14f75040  76828438 76907c77 ole32!CRpcChannelBuffer::QueryInterface
14f75044  7681c7e4 7689b57c ole32!CRpcChannelBuffer::QueryInterface
14f75048  00000003
14f7504c  00000002
14f75050  00000000
14f75054  00000000
14f75058  0046ccd0 0046ce50
14f7505c  0e8de858 00000000
14f75060  1acb7310 00000044
14f75064  1acb3130 76828510 ole32!CStdIdentity::`vftable’
14f75068  7682b098 767f8066 ole32!CDestObject::QueryInterface
14f7506c  00070005 ee0100ed
14f75070  00000000
14f75074  00000000
14f75078  00000d78
14f7507c  00000000
14f75080  76828438 76907c77 ole32!CRpcChannelBuffer::QueryInterface
14f75084  7681c7e4 7689b57c ole32!CRpcChannelBuffer::QueryInterface
14f75088  00000001
14f7508c  00000024
14f75090  00000000
14f75094  00000000
14f75098  07abd4a8 07aacab0
14f7509c  00000000
14f750a0  00000000
14f750a4  1ae12b10 76828510 ole32!CStdIdentity::`vftable’
14f750a8  7682b098 767f8066 ole32!CDestObject::QueryInterface
14f750ac  00070005 ee0100ed
14f750b0  ffffffff
14f750b4  00001134
14f750b8  00001134
14f750bc  00000000

0:000> dd 0046ccd0 l4
0046ccd0  0046ce50 0046cc50 00000dec 00000000

In Task Manager I found this to be ieuser.exe process so I suspect there is a high degree of process coupling between all launched IE7 processes and ieuser.exe including COM/OLE runtime.

The stripped versions of dumps are available for practice on ftp:

ftp://dumpanalysis.org/pub/ie7_pattern_cooperation2.zip

- Dmitry Vostokov @ DumpAnalysis.org -

3 Responses to “Stack trace collection, blocked thread and coupled processes: pattern cooperation”

  1. Yuhong Bao Says:

    On the matter of hex-only address for ieframe entries:
    That was due to a missing symbols that was fixed since.

  2. Software Generalist » Blog Archive » Reading Notebook: 24-May-10 Says:

    […] Protected mode IE startup sequence (pp. 467 - 470) - ieuser.exe might block several iexplore.exe instances: http://www.dumpanalysis.org/blog/index.php/2009/02/11/stack-trace-collection-blocked-thread-and-coup… […]

  3. Crash Dump Analysis » Blog Archive » Reading Notebook: 24-May-10 Says:

    […] Protected mode IE startup sequence (pp. 467 - 470) - ieuser.exe might block several iexplore.exe instances: http://www.dumpanalysis.org/blog/index.php/2009/02/11/stack-trace-collection-blocked-thread-and-coup… […]

Leave a Reply

You must be logged in to post a comment.