Crash Dump Analysis Patterns (Part 238)

Sometimes developers introduce their own variants of synchronization code instead of using synchronization API provided by language runtime and OS. If we are lucky we can spot it in function and class method names and then use Constant Subtrace analysis pattern:

0: kd> kc
*** Stack trace for last set context - .thread/.cxr resets it
# Call Site
00 nt!KiSwapContext
01 nt!KiCommitThreadWait
02 nt!KeWaitForSingleObject
03 nt!NtWaitForSingleObject
04 nt!KiSystemServiceCopyEnd
05 ntdll!ZwWaitForSingleObject
06 KERNELBASE!WaitForSingleObjectEx
07 wbemcomn!CWbemCriticalSection::Enter
08 wbemcore!EnsureInitialized
09 wbemcore!InitAndWaitForClient
0a wbemcore!CWbemLevel1Login::ConnectorLogin
0b wbemcore!CWbemLevel1Login::NTLMLogin
0c RPCRT4!Invoke
0d RPCRT4!NdrStubCall2
0e ole32!CStdStubBuffer_Invoke
0f ole32!SyncStubInvoke
10 ole32!StubInvoke
11 ole32!CCtxComChnl::ContextInvoke
12 ole32!AppInvoke
13 ole32!ComInvokeWithLockAndIPID
14 ole32!ThreadInvoke
15 RPCRT4!DispatchToStubInCNoAvrf
16 RPCRT4!RPC_INTERFACE::DispatchToStubWorker
18 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject
19 RPCRT4!LRPC_SCALL::DispatchRequest
1a RPCRT4!LRPC_SCALL::HandleRequest
1c RPCRT4!LRPC_ADDRESS::HandleRequest
1e RPCRT4!LrpcIoComplete
1f ntdll!TppAlpcpExecuteCallback
20 ntdll!TppWorkerThread
21 kernel32!BaseThreadInitThunk
22 ntdll!RtlUserThreadStart

0: kd> kc
*** Stack trace for last set context - .thread/.cxr resets it
# Call Site
00 repdrvfs!SCachePage::operator=
01 repdrvfs!std::vector<scachepage,wbem_allocator<scachepage> >::erase
02 repdrvfs!CPageCache::Read
03 repdrvfs!CPageFile::GetPage
04 repdrvfs!ValidateBTreeAgainstObjHeap
05 repdrvfs!PerformAllValidations
06 repdrvfs!VerifyRepositoryOnline
07 repdrvfs!VerifyRepository
08 repdrvfs!CPageSource::Startup
09 repdrvfs!CPageSource::Init
0a repdrvfs!CFileCache::InnerInitialize
0b repdrvfs!CFileCache::Initialize
0c repdrvfs!CRepository::Initialize
0d repdrvfs!CRepository::Logon
0e wbemcore!CRepository::Init
0f wbemcore!InitSubsystems
10 wbemcore!ConfigMgr::InitSystem
11 wbemcore!EnsureInitialized
12 wbemcore!InitAndWaitForClient
13 wbemcore!CWbemLevel1Login::ConnectorLogin
14 wbemcore!CWbemLevel1Login::NTLMLogin
15 RPCRT4!Invoke
16 RPCRT4!NdrStubCall2
17 ole32!CStdStubBuffer_Invoke
18 ole32!SyncStubInvoke
19 ole32!StubInvoke
1a ole32!CCtxComChnl::ContextInvoke
1b ole32!AppInvoke
1c ole32!ComInvokeWithLockAndIPID
1d ole32!ThreadInvoke
1e RPCRT4!DispatchToStubInCNoAvrf
1f RPCRT4!RPC_INTERFACE::DispatchToStubWorker
21 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject
22 RPCRT4!LRPC_SCALL::DispatchRequest
23 RPCRT4!LRPC_SCALL::HandleRequest
25 RPCRT4!LRPC_ADDRESS::HandleRequest
27 RPCRT4!LrpcIoComplete
28 ntdll!TppAlpcpExecuteCallback
29 ntdll!TppWorkerThread
2a kernel32!BaseThreadInitThunk
2b ntdll!RtlUserThreadStart

These two thread stack traces were spotted from a complete memory dump Stack Trace Collection as the part of a larger ALPC Wait Chain. We switched to these threads using .thread /r /p WinDbg command to get the stripped stack trace via kc command for better illustration. We see Constant Subtrace until wbemcore!EnsureInitialized function which serves as a bifurcation stack frame. The first stack trace has “CriticalSection::Enter” after the bifurcation stack frame compared to the second stack trace which looks like Spiking Thread in user space.

There is no hidden critical section associated with that process except the one which is probably related to the spiking Variable Subtrace since it doesn’t have any LockCount:

0: kd> !cs -l -o -s
DebugInfo = 0x0000000004a774d0
Critical section = 0x000000000308d690 (+0x308D690)
LockCount = 0×0
WaiterWoken = No
OwningThread = 0×0000000000000928
RecursionCount = 0×1
LockSemaphore = 0×0
SpinCount = 0×0000000000000000
OwningThread = .thread fffffa806ebd8060
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled

We can also disassemble wbemcomn!CWbemCriticalSection::Enter and find out that it calls WaitForSingleObject once and no other synchronization API indeed:

0: kd> uf wbemcomn!CWbemCriticalSection::Enter
000007fe`f78ad1c0 mov rcx,qword ptr [rbx+10h]
000007fe`f78ad1c4 mov edx,r12d
000007fe`f78ad1c7 call qword ptr [wbemcomn!_imp_WaitForSingleObject (000007fe`f78ee4f0)]
000007fe`f78ad1cd cmp eax,esi
000007fe`f78ad1cf je wbemcomn!CWbemCriticalSection::Enter+0x52 (000007fe`f78a62a3) Branch

We add this nonstandard synchronization memory analysis pattern to Wait Chain analysis pattern category.

- Dmitry Vostokov @ + -

Leave a Reply