Crash Dump Analysis Patterns (Part 127)

Stack Trace is a general pattern and there can always be found fine-grained patterns in stack traces as well. Here we discuss the general category of such stack trace patterns called Technology-Specific Subtrace (TSST) and give examples related to COM technology.

Consider this trace:

1: kd> k250
ChildEBP RetAddr            
8d5d2808 82a7eb15 nt!KiSwapContext+0x26
8d5d2840 82a7d403 nt!KiSwapThread+0x266
8d5d2868 82a772cf nt!KiCommitThreadWait+0x1df
8d5d28e0 82550d75 nt!KeWaitForSingleObject+0x393
8d5d293c 82550e10 win32k!xxxRealSleepThread+0x1d7
8d5d2958 824ff4b0 win32k!xxxSleepThread+0x2d
8d5d29cc 825547e8 win32k!xxxInterSendMsgEx+0xb1c
8d5d2a1c 825546a4 win32k!xxxSendMessageTimeout+0x13b
8d5d2a44 82533843 win32k!xxxSendMessage+0×28
8d5d2b08 824fd865 win32k!xxxCalcValidRects+0xf7
8d5d2b64 82502c98 win32k!xxxEndDeferWindowPosEx+0×100
8d5d2b84 825170c9 win32k!xxxSetWindowPos+0xf6
8d5d2c08 82517701 win32k!xxxActivateThisWindow+0×2b1
8d5d2c38 82517537 win32k!xxxActivateWindow+0×144
8d5d2c4c 824fd9dd win32k!xxxSwpActivate+0×44
8d5d2ca4 82502c98 win32k!xxxEndDeferWindowPosEx+0×278
8d5d2cc4 824fff82 win32k!xxxSetWindowPos+0xf6
8d5d2d10 82a5342a win32k!NtUserSetWindowPos+0×140
8d5d2d10 76ee64f4 nt!KiFastCallEntry+0×12a (TrapFrame @ 8d5d2d34)
01e2cea0 7621358d ntdll!KiFastSystemCallRet
01e2cea4 6a8fa0eb USER32!NtUserSetWindowPos+0xc
01e2cf14 6a894b13 IEFRAME!SHToggleDialogExpando+0×15a
01e2cf28 6a894d5d IEFRAME!EleDlg::ToggleExpando+0×20
01e2d74c 6a895254 IEFRAME!EleDlg::OnInitDlg+0×229
01e2d7b8 762186ef IEFRAME!EleDlg::DlgProcEx+0×189
01e2d7e4 76209eb2 USER32!InternalCallWinProc+0×23
01e2d860 7620b98b USER32!UserCallDlgProcCheckWow+0xd6
01e2d8a8 7620bb7b USER32!DefDlgProcWorker+0xa8
01e2d8c4 762186ef USER32!DefDlgProcW+0×22
01e2d8f0 76218876 USER32!InternalCallWinProc+0×23
01e2d968 76217631 USER32!UserCallWinProcCheckWow+0×14b
01e2d9a8 76209b1d USER32!SendMessageWorker+0×4d0
01e2da64 76235500 USER32!InternalCreateDialog+0xb0d
01e2da94 76235553 USER32!InternalDialogBox+0xa7
01e2dab4 76235689 USER32!DialogBoxIndirectParamAorW+0×37
01e2dad8 6a5d4952 USER32!DialogBoxParamW+0×3f
01e2db00 6a5d5024 IEFRAME!Detour_DialogBoxParamW+0×47
01e2db24 6a8956df IEFRAME!SHFusionDialogBoxParam+0×32
01e2db58 6a8957bb IEFRAME!EleDlg::ShowDialog+0×398
01e2e638 6a8959d3 IEFRAME!ShowDialogBox+0xb6
01e2eb9c 6a9013ed IEFRAME!ShowElevationPrompt+0×1dd
01e2f010 7669fc8f IEFRAME!CIEUserBrokerObject::BrokerCoCreateInstance+0×202
01e2f040 76704c53 RPCRT4!Invoke+0×2a
01e2f448 76d9d936 RPCRT4!NdrStubCall2+0×2d6
01e2f490 76d9d9c6 ole32!CStdStubBuffer_Invoke+0xb6
01e2f4d8 76d9df1f ole32!SyncStubInvoke+0×3c
01e2f524 76cb213c ole32!StubInvoke+0xb9
01e2f600 76cb2031 ole32!CCtxComChnl::ContextInvoke+0xfa
01e2f61c 76d9a754 ole32!MTAInvoke+0×1a
01e2f64c 76d9dcbb ole32!AppInvoke+0xab
01e2f72c 76d9a773 ole32!ComInvokeWithLockAndIPID+0×372

01e2f778 7669f34a ole32!ThreadInvoke+0×302
01e2f7b4 7669f4da RPCRT4!DispatchToStubInCNoAvrf+0×4a
01e2f80c 7669f3c6 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×16c
01e2f834 766a0cef RPCRT4!RPC_INTERFACE::DispatchToStub+0×8b
01e2f86c 7669f882 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0xb2
01e2f8b8 7669f7a4 RPCRT4!LRPC_SCALL::DispatchRequest+0×23b
01e2f8d8 7669f763 RPCRT4!LRPC_SCALL::QueueOrDispatchCall+0xbd
01e2f8f4 7669f5ff RPCRT4!LRPC_SCALL::HandleRequest+0×34f
01e2f928 7669f573 RPCRT4!LRPC_SASSOCIATION::HandleRequest+0×144
01e2f960 7669ee4f RPCRT4!LRPC_ADDRESS::HandleRequest+0xbd
01e2f9dc 7669ece7 RPCRT4!LRPC_ADDRESS::ProcessIO+0×50a
01e2f9e8 766a1357 RPCRT4!LrpcServerIoHandler+0×16
01e2f9f8 76ecd3a3 RPCRT4!LrpcIoComplete+0×16
01e2fa20 76ed0748 ntdll!TppAlpcpExecuteCallback+0×1c5
01e2fb88 76e11174 ntdll!TppWorkerThread+0×5a4
01e2fb94 76efb3f5 kernel32!BaseThreadInitThunk+0xe
01e2fbd4 76efb3c8 ntdll!__RtlUserThreadStart+0×70
01e2fbec 00000000 ntdll!_RtlUserThreadStart+0×1b

In the middle of the stack trace we see COM interface invocation in IEFRAME module. The similar stack trace fragment can be found in the following stack trace where COM IRemUnknown interface implementation resides in .NET CLR mscorwks module:

0:000> kL
ChildEBP RetAddr 
0018a924 68b5f8f0 mscorwks!SafeReleaseHelper+0x77
0018a958 68b04a99 mscorwks!SafeRelease+0x2f
0018a98c 68b04860 mscorwks!IUnkEntry::Free+0x68
0018a9a0 68b049b5 mscorwks!RCW::ReleaseAllInterfaces+0x18
0018a9d0 68b049e1 mscorwks!RCW::ReleaseAllInterfacesCallBack+0xbd
0018aa00 68c0a108 mscorwks!RCW::Cleanup+0x22
0018aa0c 68c0a570 mscorwks!RCWCleanupList::ReleaseRCWListRaw+0x16
0018aa3c 68bd4b3d mscorwks!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xdf
0018aa4c 75dd8c2e mscorwks!CtxEntry::EnterContextCallback+0×89
0018aa68 763c586c ole32!CRemoteUnknown::DoCallback+0×7a
0018aa84 764405f1 rpcrt4!Invoke+0×2a
0018ae88 75efd936 rpcrt4!NdrStubCall2+0×2ea
0018aed0 75efd9c6 ole32!CStdStubBuffer_Invoke+0xb6
0018af18 75efdf1f ole32!SyncStubInvoke+0×3c
0018af64 75e1223c ole32!StubInvoke+0xb9
0018b040 75e12131 ole32!CCtxComChnl::ContextInvoke+0xfa
0018b05c 75e130fa ole32!MTAInvoke+0×1a
0018b088 75efde47 ole32!STAInvoke+0×46
0018b0bc 75efdcbb ole32!AppInvoke+0xab
0018b19c 75efe34c ole32!ComInvokeWithLockAndIPID+0×372

0018b1c4 75e12ed2 ole32!ComInvoke+0xc5
0018b1d8 75e12e91 ole32!ThreadDispatch+0×23
0018b21c 75a06238 ole32!ThreadWndProc+0×161
0018b248 75a068ea user32!InternalCallWinProc+0×23
0018b2c0 75a07d31 user32!UserCallWinProcCheckWow+0×109
0018b320 75a07dfa user32!DispatchMessageWorker+0×3bc
0018b330 75ddd6be user32!DispatchMessageW+0xf
0018b360 75ddd66d ole32!CCliModalLoop::PeekRPCAndDDEMessage+0×4c
0018b390 75ddd57e ole32!CCliModalLoop::FindMessage+0×30
0018b3f0 75ddd633 ole32!CCliModalLoop::HandleWakeForMsg+0×41
0018b408 75dd1117 ole32!CCliModalLoop::BlockFn+0xc3
0018b488 68a6c905 ole32!CoWaitForMultipleHandles+0xcd
0018b4a8 68a6c866 mscorwks!NT5WaitRoutine+0×51
0018b514 68a6c7ca mscorwks!MsgWaitHelper+0xa5
0018b534 68b5fbe4 mscorwks!Thread::DoAppropriateAptStateWait+0×28
0018b5b8 68b5fc79 mscorwks!Thread::DoAppropriateWaitWorker+0×13c
0018b608 68b5fdf9 mscorwks!Thread::DoAppropriateWait+0×40
0018b664 68a1c5b6 mscorwks!CLREvent::WaitEx+0xf7
0018b678 68b1adb4 mscorwks!CLREvent::Wait+0×17
0018b6c8 68b1ab2a mscorwks!WKS::GCHeap::FinalizerThreadWait+0xfb
0018b764 08fa12c1 mscorwks!GCInterface::RunFinalizers+0×99
[…]

A TSST usually spans several modules. In any stack trace we can also find several TSST that may be overlapping. For example, in the first stack trace above we can discern fragments of COM, RPC, LPC, GUI Dialog, Window Management, and Window Messaging subtraces. In the second trace we can also see GC, Modal Loop, COM Wrapper, and Interface Management stack frames.

The closest software trace analysis pattern here is Implementation Discourse.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

One Response to “Crash Dump Analysis Patterns (Part 127)”

  1. Dmitry Vostokov Says:

    Another example:

    App!foo
    rpcrt4!Invoke
    rpcrt4!NdrStubCall2
    combase!CStdStubBuffer_Invoke
    oleaut32!CUnivStubWrapper::Invoke
    combase!SyncStubInvoke
    combase!StubInvoke
    combase!CCtxComChnl::ContextInvoke
    combase!DefaultInvokeInApartment
    combase!ClassicSTAInvokeInApartment
    combase!AppInvoke
    combase!ComInvokeWithLockAndIPID
    combase!ComInvoke
    combase!ThreadDispatch
    combase!ThreadWndProc
    user32!_InternalCallWinProc
    user32!UserCallWinProcCheckWow
    user32!DispatchMessageWorker
    user32!DispatchMessageA
    App!bar
    kernel32!BaseThreadInitThunk

Leave a Reply