Software Diagnostics Library

If a thread has an associated I/O Request Packet (IRP) we may see another type of a stack trace we call I/O Request Stack Trace. It also grows bottom-up as can be seen on the diagram 3. We can see this stack trace by using !irp WinDbg command:

0: kd> !thread fffffa801827a4c0 3f
THREAD fffffa801827a4c0 Cid 06c0.50cc Teb: 000007ffffec8000 Win32Thread: fffff900c1c64010 WAIT: (Executive) KernelMode Alertable
fffffa8016f64028 SynchronizationEvent
IRP List:
fffffa80162aa230: (0006,03a0) Flags: 00000884 Mdl: 00000000
[…]
nt!KiSwapContext+0×7a
nt!KiCommitThreadWait+0×1d2
nt!KeWaitForSingleObject+0×19f
nt!FsRtlCancellableWaitForMultipleObjects+0×5e
nt!FsRtlCancellableWaitForSingleObject+0×27
fltmgr! ?? ::FNODOBFM::`string’+0×2bfa
fltmgr!FltpCreate+0×2a9
nt!IopParseDevice+0×14d3
nt!ObpLookupObjectName+0×588
nt!ObOpenObjectByName+0×306
nt!IopCreateFile+0×2bc
nt!NtCreateFile+0×78
nt!KiSystemServiceCopyEnd+0×13
ntdll!NtCreateFile+0xa
[…]

0: kd> !irp fffffa80162aa230
Irp is active with 10 stacks 10 is current (= 0xfffffa80162aa588)
No Mdl: No System Buffer: Thread fffffa801827a4c0: Irp stack trace.
cmd flg cl Device File Completion-Context
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 fffffa800cb28030 00000000 fffff880012048f0-fffffa8016f64010
\FileSystem\Ntfs fltmgr!FltpSynchronizedOperationCompletion
Args: 00000000 00000000 00000000 00000000
>[ 0, 0] 0 1 fffffa800ca00890 fffffa801060d070 00000000-00000000 pending
\FileSystem\FltMgr
Args: fffff88014450868 02000060 00000006 00000000

We see the current stack trace pointer points to the bottom I/O stack location. Non-empty top locations are analogous to Past Stack Trace. Further exploration of Device and File column information may point to further troubleshooting directions such as the Blocking File pattern example.

By analogy with Stack Trace Collection pattern that dumps stack traces from all threads based on memory dump type there is also I/O Stack Trace Collection pattern that dumps I/O request stack traces from all IRPs that were possible to find.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Rough Stack Trace is an example of more general Execution Residue pattern or Caller-n-Callee for managed space. It’s just a collection of symbolic references (may also include Coincidental Symbolic Information) from the thread stack region or its fragment. In WinDbg we can get it by using dpS command:

0:003> !teb
TEB at 000007fffffd6000
ExceptionList:        0000000000000000
StackBase:            0000000002450000
StackLimit:           000000000244b000
SubSystemTib:         0000000000000000
FiberData:            0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self:                 000007fffffd6000
EnvironmentPointer:   0000000000000000
ClientId:             00000000000047fc . 0000000000004824
RpcHandle:            0000000000000000
Tls Storage:          000007fffffd6058
PEB Address:          000007fffffda000
LastErrorValue:       0
LastStatusValue:      c0000302
Count Owned Locks:    0
HardErrorMode:        0

0:003> dpS 000000000244b000 0000000002450000
000007fe`fd4a8a2e ole32!InternalVerifyStackAvailable+0x44 [d:\winmain\minio\safealloca\alloca.c @ 317]
000007fe`fd4a8a2e ole32!InternalVerifyStackAvailable+0x44 [d:\winmain\minio\safealloca\alloca.c @ 317]
000007fe`fd4a8a2e ole32!InternalVerifyStackAvailable+0x44 [d:\winmain\minio\safealloca\alloca.c @ 317]
00000000`771d5430 ntdll!RtlpInterceptorRoutines
00000000`771134d8 ntdll!RtlAllocateHeap+0x16c
00000000`770ec9c3 ntdll!RtlAppendUnicodeStringToString+0x53
00000000`76eaebe5 kernel32!Wow64RedirectKeyPathInternal+0x2b7
00000000`770ec9c3 ntdll!RtlAppendUnicodeStringToString+0x53
00000000`771140fd ntdll!RtlFreeHeap+0x1a6
00000000`76eaec01 kernel32!ConstructKernelKeyPath+0x15f
00000000`76eaedd3 kernel32!Wow64NtOpenKey+0xee
00000000`771140fd ntdll!RtlFreeHeap+0x1a6
00000000`76ebc8aa kernel32!BaseRegOpenClassKeyFromLocation+0x3ba
00000000`76f3edf0 kernel32!`string'
00000000`771d5430 ntdll!RtlpInterceptorRoutines
00000000`76ebc9b9 kernel32!BaseRegGetUserPrefixLength+0xea
00000000`76f3ee38 kernel32!`string'
00000000`76f3edc8 kernel32!`string'
00000000`76ebc3a8 kernel32!BaseRegGetKeySemantics+0x1b8
00000000`771150d3 ntdll!RtlNtStatusToDosError+0x27
00000000`76eb36b7 kernel32!LocalBaseRegOpenKey+0x276
000007fe`fd4b6c79 ole32!GetUnquotedPath+0x29 [d:\w7rtm\com\ole32\com\objact\dllcache.cxx @ 2256]
000007fe`fd4b7019 ole32!CClassCache::CDllPathEntry::NegotiateDllInstantiationProperties2+0x145 [d:\w7rtm\com\ole32\com\objact\dllcache.cxx @ 3092]
00000000`771d5430 ntdll!RtlpInterceptorRoutines
00000000`771134d8 ntdll!RtlAllocateHeap+0x16c
00000000`77115cc4 ntdll!RtlpAllocateHeap+0xc12
000007fe`fdc10359 usp10!CUspShapingClient::AllocMem+0x49
000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12
000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12
000007fe`fdc1d4f1 usp10!UspFreeMem+0x61
000007fe`fdc4896e usp10!COtlsClient::FreeMem+0xe
000007fe`fdc6e817 usp10!ApplyFeatures+0xa17
000007fe`fdc6f2f2 usp10!ApplyLookup+0x592
000007fe`fdc48901 usp10!COtlsClient::GetDefaultGlyphs+0x131
000007fe`fdc60100 usp10!HangulEngineGetGlyphs+0x2c0
000007fe`fdc10359 usp10!CUspShapingClient::AllocMem+0x49
000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12
000007fe`fdc10359 usp10!CUspShapingClient::AllocMem+0x49
000007fe`fdc1d4f1 usp10!UspFreeMem+0x61
000007fe`fdc48942 usp10!COtlsClient::AllocMem+0x12
000007fe`fdc1d4f1 usp10!UspFreeMem+0x61
000007fe`fdc4896e usp10!COtlsClient::FreeMem+0xe
000007fe`fdc6e817 usp10!ApplyFeatures+0xa17
000007fe`fdc6aaa8 usp10!RePositionOtlGlyphs+0x238
000007fe`fdc48901 usp10!COtlsClient::GetDefaultGlyphs+0x131
000007fe`fdc60100 usp10!HangulEngineGetGlyphs+0x2c0
000007fe`fdc48798 usp10!COtlsClient::ReleaseOtlTable+0x78
000007fe`fdc6ae85 usp10!otlResourceMgr::detach+0xc5
00000000`7717c63e ntdll!EtwEventWriteNoRegistration+0xae
000007fe`fdc48a99 usp10!COtlsClient::Release+0x49
00000000`771150d3 ntdll!RtlNtStatusToDosError+0x27
00000000`7716bd85 ntdll!WaitForWerSvc+0x85
00000000`7717b94e ntdll!WerpAllocateAndInitializeSid+0xbe
00000000`7716bd90 ntdll! ?? ::FNODOBFM::`string'
00000000`77175dcf ntdll!WerpFreeSid+0x3f
00000000`7718123d ntdll!SendMessageToWERService+0x22d
00000000`77181260 ntdll! ?? ::FNODOBFM::`string'
00000000`77182308 ntdll!ReportExceptionInternal+0xc8
000007fe`fd061430 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`76ec1723 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`76f3b5e5 kernel32!WerpReportFaultInternal+0x215
00000000`76f3b767 kernel32!WerpReportFault+0x77
00000000`76f3b7bf kernel32!BasepReportFault+0x1f
00000000`76f3b9dc kernel32!UnhandledExceptionFilter+0x1fc
00000000`77118d7e ntdll!RtlpFindUnicodeStringInSection+0x50e
00000000`771198fc ntdll!LdrpFindLoadedDll+0x10c
00000000`770e9caa ntdll!RtlDecodePointer+0x2a
00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`771e8180 ntdll!`string'+0xc040
00000000`771e818c ntdll!`string'+0xc04c
00000000`77153398 ntdll! ?? ::FNODOBFM::`string'+0x2365
00000000`770d85c8 ntdll!_C_specific_handler+0x8c
00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`770ec541 ntdll!RtlUserThreadStart+0x1d
00000000`770e9d2d ntdll!RtlpExecuteHandlerForException+0xd
00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0)
00000000`770d91cf ntdll!RtlDispatchException+0x45a
00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0)
00000000`7711920a ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3da
00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0)
00000000`771e8180 ntdll!`string'+0xc040
00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`770ec541 ntdll!RtlUserThreadStart+0x1d
00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0)
00000000`771d7718 ntdll!LdrpDefaultExtension
00000000`770d852c ntdll!_C_specific_handler
00000000`771e8180 ntdll!`string'+0xc040
000007fe`ff3625c0 msctf!s_szCompClassName
000007fe`fd602790 ole32!`string'
00000000`770e7a33 ntdll!LdrpFindOrMapDll+0x138
00000000`771192a8 ntdll!LdrpApplyFileNameRedirection+0x2d3
000007fe`fd602848 ole32!`string'
00000000`771d5430 ntdll!RtlpInterceptorRoutines
00000000`77113448 ntdll!RtlAllocateHeap+0xe4
00000000`76fd88b8 user32!GetPropW+0x4d
00000000`76fd88b8 user32!GetPropW+0x4d
00000000`76fd7931 user32!IsWindow+0x9
00000000`76fd7931 user32!IsWindow+0x9
00000000`770f41c8 ntdll!RtlpReAllocateHeap+0x178
000007fe`fb601381 uxtheme!CThemeWnd::_PreDefWindowProc+0x31
00000000`76eb59e0 kernel32!BaseThreadInitThunk
00000000`ffdbdb32 calc!CTimedCalc::Start+0xa9
00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0)
00000000`ffe0ac64 calc!_dyn_tls_init_callback <PERF> (calc+0x7ac64)
00000000`76ea0000 kernel32!TestResourceDataMatchEntry <PERF> (kernel32+0x0)
00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0)
00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0)
00000000`76fd760e user32!RealDefWindowProcW+0x5a
000007fe`fb600037 uxtheme!operator delete <PERF> (uxtheme+0x37)
00000000`77111248 ntdll!KiUserExceptionDispatch+0x2e
000007fe`fb63fb40 uxtheme!$$VProc_ImageExportDirectory
00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2
00000000`76fe76c2 user32!DefDlgProcW+0x36
00000000`76fd9bef user32!UserCallWinProcCheckWow+0x1cb
00000000`76fd9b43 user32!UserCallWinProcCheckWow+0x99
00000000`76fd9bef user32!UserCallWinProcCheckWow+0x1cb
00000000`76fd72cb user32!DispatchClientMessage+0xc3
00000000`770e46b4 ntdll!NtdllDialogWndProc_W
00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2
00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2
00000000`77101530 ntdll!NtdllDispatchMessage_W
00000000`76fe505b user32!DialogBox2+0x2ec
00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0)
00000000`76fe4edd user32!InternalDialogBox+0x135
00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0)
00000000`76fe4f52 user32!DialogBoxIndirectParamAorW+0x58
00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0)
00000000`76fdd476 user32!DialogBoxParamW+0x66
00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`ffdbdafa calc!CTimedCalc::WatchDogThread+0x72
00000000`76eb59ed kernel32!BaseThreadInitThunk+0xd
00000000`770ec541 ntdll!RtlUserThreadStart+0x1d
00000000`76f3b7e0 kernel32!UnhandledExceptionFilter
00000000`76f3b7e0 kernel32!UnhandledExceptionFilter

The name for this pattern comes from rough sets in mathematics.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Although in the case of system hangs we, usually, recommend dumping Stack Trace Collection, in some cases it is very time-consuming, especially when it involves thousands of processes such as in modern terminal services environments. In such a case, if the problem description indicates the last action such as a not progressing user logon or a recently launched process we first check the tail of the corresponding linked list where Last Object is usually added to the tail of the list:

Sometimes we can simply check the end of some enumerated collection such as sessions (dotted lines represent ALPC Wait Chains):

This analysis pattern can be added to the first tier of RSDP. If nothing found around a couple of Last Objects we then resort to the analysis of entire linked lists.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here we provide another variant of a general Wait Chain pattern related to RtlAcquireResourceShared and RtlAcquireResourceExclusive calls:

THREAD fffffa8052d66060  Cid 03c0.3240  Teb: 000007fffff90000 Win32Thread: 0000000000000000 WAIT: (UserRequest) UserMode Non-Alertable
fffffa804a79ad50  Semaphore Limit 0x7fffffff
Impersonation token:  fffff8a01b19d060 (Level Impersonation)
DeviceMap                 fffff8a0035276c0
Owning Process            fffffa804a16b260       Image:         lsm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      73343513       Ticks: 1460259 (0:06:20:16.546)
Context Switch Count      17             IdealProcessor: 1
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0x000000007735fbf0)
Stack Init fffff8800e870db0 Current fffff8800e870900
Base fffff8800e871000 Limit fffff8800e86b000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`0e870940 fffff800`01c76972 nt!KiSwapContext+0x7a
fffff880`0e870a80 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2
fffff880`0e870b10 fffff800`01f7b2be nt!KeWaitForSingleObject+0x19f
fffff880`0e870bb0 fffff800`01c801d3 nt!NtWaitForSingleObject+0xde
fffff880`0e870c20 00000000`773912fa nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0e870c20)
00000000`022ae6c8 00000000`773470b4 ntdll!NtWaitForSingleObject+0xa
00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0
00000000`022ae710 00000000`ff401675 lsm!CAutoSharedLock::CAutoSharedLock+0×61
00000000`022ae7e0 00000000`ff402c68 lsm!CTSSession::getTerminal+0×21
00000000`022ae820 000007fe`fd8bff85 lsm!RpcGetEnumResult+0×202
00000000`022ae980 000007fe`fd8b4de2 RPCRT4!Invoke+0×65
00000000`022ae9e0 000007fe`fd8b17bd RPCRT4!NdrStubCall2+0×32a
00000000`022af000 000007fe`fd8b3254 RPCRT4!NdrServerCall2+0×1d
00000000`022af030 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0×14
00000000`022af060 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146
00000000`022af180 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0×149
00000000`022af260 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0×20d
00000000`022af390 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf
00000000`022af4d0 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5
00000000`022af560 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0×26b
00000000`022af5f0 00000000`7713652d ntdll!TppWorkerThread+0×3f8
00000000`022af8f0 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd
00000000`022af920 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

These functions are undocumented but ReactOS source code shows they all take a pointer to RTL_RESOURCE structure which has handles to a shared and exclusive semaphores:

RTL_CRITICAL_SECTION Lock
HANDLE SharedSemaphore
ULONG SharedWaiters
HANDLE ExclusiveSemaphore
ULONG ExclusiveWaiters
LONG NumberActive
HANDLE OwningThread
ULONG TimeoutBoost
PVOID DebugInfo

To double check that we disassemble RtlAcquireResourceShared and check the return address from NtWaitForSingleObject call (00000000`773470b4):

0: kd> .thread /r /p fffffa8052d66060
Implicit thread is now fffffa80`52d66060
Implicit process is now fffffa80`4a16b260
Loading User Symbols
..........................................

0: kd> uf ntdll!RtlAcquireResourceShared
[...]
ntdll!RtlAcquireResourceShared+0xc2:
00000000`773470a6 488b4b28 mov rcx,qword ptr [rbx+28h]
00000000`773470aa 4c8bc6 mov r8,rsi
00000000`773470ad 33d2 xor edx,edx
00000000`773470af e83ca20400 call ntdll!NtWaitForSingleObject (00000000`773912f0)
00000000`773470b4 3d02010000 cmp eax,102h
00000000`773470b9 0f8402800600 je ntdll! ?? ::FNODOBFM::`string’+0×12629 (00000000`773af0c1)
[…]
ntdll!RtlAcquireResourceShared:
00000000`77352af0 48895c2420 mov qword ptr [rsp+20h],rbx
00000000`77352af5 57 push rdi
00000000`77352af6 4883ec30 sub rsp,30h
00000000`77352afa 448b4944 mov r9d,dword ptr [rcx+44h]
00000000`77352afe 0fb6fa movzx edi,dl
00000000`77352b01 488bd9 mov rbx,rcx
00000000`77352b04 4585c9 test r9d,r9d
00000000`77352b07 0f88a7000000 js ntdll!RtlAcquireResourceShared+0×65 (00000000`77352bb4)
[…]

We see the handle is taken from [RBX+28] and we see that RBX was saved at the function prologue and then the value of RCX was assigned to RBX. RCX as the first calling convention parameter should be a pointer to RTL_RESOURCE which has RTL_CRITICAL_SECTION as the first member and its size is 0×28:

0: kd> dt ntdll!_RTL_CRITICAL_SECTION
ntdll!_RTL_CRITICAL_SECTION
+0x000 DebugInfo        : Ptr64 _RTL_CRITICAL_SECTION_DEBUG
+0x008 LockCount        : Int4B
+0x00c RecursionCount   : Int4B
+0x010 OwningThread     : Ptr64 Void
+0x018 LockSemaphore    : Ptr64 Void
+0x020 SpinCount        : Uint8B

Therefore [RBX+28] contains SharedSemaphore field which is assigned to RCX as a first parameter to NtWaitForSingleObject. The similar fragment of RtlAcquireResourceExclusive has [RBX+36] which 0×10 further than 0×28 and corresponds to ExclusiveSemaphore handle field:

ntdll!RtlAcquireResourceExclusive+0xd2:
00000000`770c2a12 488b4b38        mov     rcx,qword ptr [rbx+38h]
00000000`770c2a16 4c8bc6          mov     r8,rsi
00000000`770c2a19 33d2            xor     edx,edx
00000000`770c2a1b e8d0e80400      call    ntdll!NtWaitForSingleObject (00000000`771112f0)
00000000`770c2a20 3d02010000      cmp     eax,102h
00000000`770c2a25 0f8401c60600    je      ntdll! ?? ::FNODOBFM::`string’+0×12591 (00000000`7712f02c)

So we just need to know the vale of RBX and dump the structure to find OwningThread field. We can either calculate it from RSP or use /c switch with .frame command:

0: kd> kn
*** Stack trace for last set context - .thread/.cxr resets it
# Child-SP          RetAddr           Call Site
00 fffff880`0e870940 fffff800`01c76972 nt!KiSwapContext+0x7a
01 fffff880`0e870a80 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2
02 fffff880`0e870b10 fffff800`01f7b2be nt!KeWaitForSingleObject+0x19f
03 fffff880`0e870bb0 fffff800`01c801d3 nt!NtWaitForSingleObject+0xde
04 fffff880`0e870c20 00000000`773912fa nt!KiSystemServiceCopyEnd+0x13
05 00000000`022ae6c8 00000000`773470b4 ntdll!NtWaitForSingleObject+0xa
06 00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0
07 00000000`022ae710 00000000`ff401675 lsm!CAutoSharedLock::CAutoSharedLock+0×61
08 00000000`022ae7e0 00000000`ff402c68 lsm!CTSSession::getTerminal+0×21
09 00000000`022ae820 000007fe`fd8bff85 lsm!RpcGetEnumResult+0×202
0a 00000000`022ae980 000007fe`fd8b4de2 RPCRT4!Invoke+0×65
0b 00000000`022ae9e0 000007fe`fd8b17bd RPCRT4!NdrStubCall2+0×32a
0c 00000000`022af000 000007fe`fd8b3254 RPCRT4!NdrServerCall2+0×1d
0d 00000000`022af030 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0×14
0e 00000000`022af060 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0×146
0f 00000000`022af180 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0×149
10 00000000`022af260 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0×20d
11 00000000`022af390 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0×3bf
12 00000000`022af4d0 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5
13 00000000`022af560 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0×26b
14 00000000`022af5f0 00000000`7713652d ntdll!TppWorkerThread+0×3f8
15 00000000`022af8f0 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd
16 00000000`022af920 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0: kd> .frame /c 6
06 00000000`022ae6d0 00000000`ff4013a3 ntdll!RtlAcquireResourceShared+0xd0
rax=0000000000000000 rbx=00000000023ac128 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000077472410 rdi=0000000000000001
rip=00000000773470b4 rsp=00000000022ae6d0 rbp=0000000000000000
r8=0000000000000000  r9=0000000000000000 r10=0000000000000000
r11=0000000000000000 r12=29406b2a1a85bd43 r13=0000000000000009
r14=000000000000000c r15=00000000022aef20
iopl=0         nv up di pl nz na pe nc
cs=0000  ss=0000  ds=0000  es=0000  fs=0000  gs=0000             efl=00000000
ntdll!RtlAcquireResourceShared+0xd0:
00000000`773470b4 3d02010000      cmp     eax,102h

0: kd> dp rbx+28 L10
00000000`023ac150  00000000`00001244 00000000`000001b5
00000000`023ac160  00000000`00000f3c ffffffff`00000000
00000000`023ac170  00000000`000021a0 00000000`00000000
00000000`023ac180  00000000`02735fc0 00000000`00000001
00000000`023ac190  00000000`00000000 01cf07ac`9fa06d27
00000000`023ac1a0  00000000`00000000 00000000`00000000
00000000`023ac1b0  ffffffff`ffffffff 00000000`00000000
00000000`023ac1c0  00000000`00000000 00000000`00000000

We check all these handles (OwnerThread seems comes earlier with NumberActive field missing but that could just differences between the old x86 structure implemented in ReactOS and x64 Windows):

0: kd> !handle 00000000`00001244

PROCESS fffffa804a16b260
SessionId: 0  Cid: 03c0    Peb: 7fffffdc000  ParentCid: 0350
DirBase: 195950000  ObjectTable: fffff8a0032424e0  HandleCount: 5252.
Image: lsm.exe

Handle table at fffff8a0032424e0 with 5252 entries in use

1244: Object: fffffa804a79ad50  GrantedAccess: 00100003 Entry: fffff8a022b39910
Object: fffffa804a79ad50  Type: (fffffa8048fc8790) Semaphore
ObjectHeader: fffffa804a79ad20 (new version)
HandleCount: 1  PointerCount: 438

0: kd> !handle 00000000`00000f3c

PROCESS fffffa804a16b260
SessionId: 0  Cid: 03c0    Peb: 7fffffdc000  ParentCid: 0350
DirBase: 195950000  ObjectTable: fffff8a0032424e0  HandleCount: 5252.
Image: lsm.exe

Handle table at fffff8a0032424e0 with 5252 entries in use

0f3c: Object: fffffa804fa81f60  GrantedAccess: 00100003 Entry: fffff8a02cd3ecf0
Object: fffffa804fa81f60  Type: (fffffa8048fc8790) Semaphore
ObjectHeader: fffffa804fa81f30 (new version)
HandleCount: 1  PointerCount: 1

0: kd> !thread -t 00000000`000021a0 3f
THREAD fffffa804d5d51b0  Cid 03c0.21a0  Teb: 000007fffff9c000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) UserMode Non-Alertable
fffffa804d5d5578  Semaphore Limit 0×1
Waiting for reply to ALPC Message fffff8a02c9a9500 : queued at port fffffa804ac4e7d0 : owned by process fffffa804adc8730
Not impersonating
DeviceMap                 fffff8a0000088c0
Owning Process            fffffa804a16b260       Image:         lsm.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      73337319       Ticks: 1466453 (0:06:21:53.328)
Context Switch Count      69             IdealProcessor: 1
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address ntdll!TppWorkerThread (0×000000007735fbf0)
Stack Init fffff8800aa1fdb0 Current fffff8800aa1f600
Base fffff8800aa20000 Limit fffff8800aa1a000 Call 0
Priority 9 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.

Child-SP          RetAddr           Call Site
fffff880`0aa1f640 fffff800`01c76972 nt!KiSwapContext+0x7a
fffff880`0aa1f780 fffff800`01c87d8f nt!KiCommitThreadWait+0x1d2
fffff880`0aa1f810 fffff800`01ca25af nt!KeWaitForSingleObject+0x19f
fffff880`0aa1f8b0 fffff800`01f968b6 nt!AlpcpSignalAndWait+0x8f
fffff880`0aa1f960 fffff800`01f95fb0 nt!AlpcpReceiveSynchronousReply+0x46
fffff880`0aa1f9c0 fffff800`01f93dab nt!AlpcpProcessSynchronousRequest+0x33d
fffff880`0aa1fb00 fffff800`01c801d3 nt!NtAlpcSendWaitReceivePort+0x1ab
fffff880`0aa1fbb0 00000000`77391b0a nt!KiSystemServiceCopyEnd+0x13 (TrapFrame @ fffff880`0aa1fc20)
00000000`01dddb48 000007fe`fd8c8306 ntdll!ZwAlpcSendWaitReceivePort+0xa
00000000`01dddb50 000007fe`fd8c2a02 RPCRT4!LRPC_CCALL::SendReceive+0x156
00000000`01dddc10 000007fe`ff5b28c0 RPCRT4!I_RpcSendReceive+0x42
00000000`01dddc40 000007fe`ff5b282f ole32!ThreadSendReceive+0x40 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 5003]
00000000`01dddc90 000007fe`ff5b265b ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0xa3 [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 4454]
00000000`01dddd30 000007fe`ff46daaa ole32!CRpcChannelBuffer::SendReceive2+0x11b [d:\w7rtm\com\ole32\com\dcomrem\channelb.cxx @ 4074]
00000000`01dddef0 000007fe`ff46da0c ole32!CAptRpcChnl::SendReceive+0x52 [d:\w7rtm\com\ole32\com\dcomrem\callctrl.cxx @ 603]
00000000`01dddfc0 000007fe`ff5b205d ole32!CCtxComChnl::SendReceive+0x68 [d:\w7rtm\com\ole32\com\dcomrem\ctxchnl.cxx @ 734]
00000000`01dde070 000007fe`fd96b949 ole32!NdrExtpProxySendReceive+0x45 [d:\w7rtm\com\rpc\ndrole\proxy.cxx @ 1932]
00000000`01dde0a0 000007fe`ff5b21d0 RPCRT4!NdrpClientCall3+0x2e2
00000000`01dde360 000007fe`ff46d8a2 ole32!ObjectStublessClient+0x11d [d:\w7rtm\com\rpc\ndrole\amd64\stblsclt.cxx @ 621]
00000000`01dde6f0 00000000`ff417d26 ole32!ObjectStubless+0x42 [d:\w7rtm\com\rpc\ndrole\amd64\stubless.asm @ 117]
00000000`01dde740 00000000`ff4186ba lsm!CTSSession::Disconnect+0x3a5
00000000`01dde810 000007fe`fd8bff85 lsm!RpcDisconnect+0x15e
00000000`01dde850 000007fe`fd96b68e RPCRT4!Invoke+0x65
00000000`01dde8a0 000007fe`fd8a92e0 RPCRT4!Ndr64StubWorker+0x61b
00000000`01ddee60 000007fe`fd8b3254 RPCRT4!NdrServerCallAll+0x40
00000000`01ddeeb0 000007fe`fd8b33b6 RPCRT4!DispatchToStubInCNoAvrf+0x14
00000000`01ddeee0 000007fe`fd8b3aa9 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x146
00000000`01ddf000 000007fe`fd8b375d RPCRT4!LRPC_SCALL::DispatchRequest+0x149
00000000`01ddf0e0 000007fe`fd8d09ff RPCRT4!LRPC_SCALL::HandleRequest+0x20d
00000000`01ddf210 000007fe`fd8d05b5 RPCRT4!LRPC_ADDRESS::ProcessIO+0x3bf
00000000`01ddf350 00000000`7735b6bb RPCRT4!LrpcIoComplete+0xa5
00000000`01ddf3e0 00000000`7735ff2f ntdll!TppAlpcpExecuteCallback+0x26b
00000000`01ddf470 00000000`7713652d ntdll!TppWorkerThread+0x3f8
00000000`01ddf770 00000000`7736c541 kernel32!BaseThreadInitThunk+0xd
00000000`01ddf7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

We see the wait chain continues with waiting for an ALPC request.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

The availability of direct dump modification raises the possibility of Tampered Dumps. These are memory dumps specifically modified to alter structural and behavioural diagnostic patterns, for example, to suppress certain module involvement or introduce fictitious past objects and interaction traces such as Execution Residue and Module Hints. There can be 2 types of such artefacts: strong tampering with new or altered information completely integrated into memory fabric and weak tampering to confuse inexperienced software support engineers and memory forensics analysts.

For example, in one such experimental process memory dump we see Exception Stack Trace pointing to a problem in calc module:

0:003> k
Child-SP RetAddr Call Site
00000000`0244e858 000007fe`fd061430 ntdll!NtWaitForMultipleObjects+0xa
00000000`0244e860 00000000`76ec1723 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`0244e960 00000000`76f3b5e5 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`0244e9f0 00000000`76f3b767 kernel32!WerpReportFaultInternal+0x215
00000000`0244ea90 00000000`76f3b7bf kernel32!WerpReportFault+0x77
00000000`0244eac0 00000000`76f3b9dc kernel32!BasepReportFault+0x1f
00000000`0244eaf0 00000000`77153398 kernel32!UnhandledExceptionFilter+0x1fc
00000000`0244ebd0 00000000`770d85c8 ntdll! ?? ::FNODOBFM::`string'+0x2365
00000000`0244ec00 00000000`770e9d2d ntdll!_C_specific_handler+0x8c
00000000`0244ec70 00000000`770d91cf ntdll!RtlpExecuteHandlerForException+0xd
00000000`0244eca0 00000000`77111248 ntdll!RtlDispatchException+0x45a
00000000`0244f380 00000000`ffdbdb27 ntdll!KiUserExceptionDispatch+0×2e
00000000`0244fab0 00000000`76eb59ed calc!CTimedCalc::WatchDogThread+0xb2
00000000`0244faf0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd
00000000`0244fb20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The default analysis command (!analyse -v) diagnoses “stack corruption”:

FAULTING_IP:
kernel32!UnhandledExceptionFilter+1fc
00000000`76f3b9dc 448bf0 mov r14d,eax

EXCEPTION_RECORD: ffffffffffffffff -- (.exr 0xffffffffffffffff)
ExceptionAddress: 0000000076f3b9dc (kernel32!UnhandledExceptionFilter+0x00000000000001fc)
ExceptionCode: 0244e9f0
ExceptionFlags: 00000000
NumberParameters: 0

DEFAULT_BUCKET_ID: STACK_CORRUPTION

PRIMARY_PROBLEM_CLASS: STACK_CORRUPTION

BUGCHECK_STR: APPLICATION_FAULT_STACK_CORRUPTION

IP_ON_HEAP: 8d483674c33bfffa
The fault address in not in any loaded module, please check your build's rebase
log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
contain the address if it were loaded.

UNALIGNED_STACK_POINTER: 0000000076f3b767

STACK_TEXT:
00000000`00000000 00000000`00000000 calc!CTimedCalc::WatchDogThread+0x0

FOLLOWUP_IP:
calc!CTimedCalc::WatchDogThread+0
00000000`ffd92254 48895c2408 mov qword ptr [rsp+8],rbx

Stored Exception resembles signs of Local Buffer Overflow (segment register values and CPU flags have suspiciously invalid values, possibly Lateral Damage):

0:003> .ecxr
rax=0000000000000000 rbx=0000000000000001 rcx=000000000244ec30
rdx=000000000244ec30 rsi=0100000000000080 rdi=0000000000000158
rip=0000000076f3b9dc rsp=0000000076f3b767 rbp=0000000000000000
r8=0000000000000000 r9=ffffffffffffffff r10=0000000076f3b7bf
r11=000000000244ec30 r12=0000000000000001 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up di pl nz na pe nc
cs=0000 ss=0000 ds=0266 es=0000 fs=0000 gs=0154 efl=00000000
kernel32!UnhandledExceptionFilter+0×1fc:
00000000`76f3b9dc 448bf0 mov r14d,eax

0:003> k
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr Call Site
00000000`76f3b767 8d483674`c33bfffa kernel32!UnhandledExceptionFilter+0x1fc
00000000`76f3b847 5aa3e800`05bfac0d 0x8d483674`c33bfffa
00000000`76f3b84f ebffcf83`48ccfff9 0x5aa3e800`05bfac0d
00000000`76f3b857 8348c000`0409ba27 0xebffcf83`48ccfff9
00000000`76f3b85f 54dfe8cf`8b48ffcf 0x8348c000`0409ba27
00000000`76f3b867 4c02778d`db33fff9 0x54dfe8cf`8b48ffcf
00000000`76f3b86f 4c000000`e024a48b 0x4c02778d`db33fff9
00000000`76f3b877 ffcf8348`04ebeb8b 0x4c000000`e024a48b
00000000`76f3b87f fffc59e9`e8cc8b49 0xffcf8348`04ebeb8b
00000000`76f3b887 42e9c78b`0775c73b 0xfffc59e9`e8cc8b49
00000000`76f3b88f fffa6fa9`e8000003 0x42e9c78b`0775c73b
00000000`76f3b897 32e9c033`0774c33b 0xfffa6fa9`e8000003
00000000`76f3b89f fa7f3d8d`4c000003 0x32e9c033`0774c33b
00000000`76f3b8a7 de15ffcf`8b490006 0xfa7f3d8d`4c000003
00000000`76f3b8af f9370d8b`4800000e 0xde15ffcf`8b490006
00000000`76f3b8b7 000014a1`15ff0006 0xf9370d8b`4800000e
00000000`76f3b8bf 840fc33b`48f08b4c 0x000014a1`15ff0006
00000000`76f3b8c7 f6158b48`00000099 0x840fc33b`48f08b4c
00000000`76f3b8cf 0238c281`480006f3 0xf6158b48`00000099
00000000`76f3b8d7 48cfe8c8`8b480000 0x0238c281`480006f3
00000000`76f3b8df 8b4c7f74`c33bfff9 0x48cfe8c8`8b480000
00000000`76f3b8e7 888b4900`06f3dc05 0x8b4c7f74`c33bfff9
00000000`76f3b8ef 75083949`00000238 0x888b4900`06f3dc05
00000000`76f3b8f7 00000240`808b496c 0x75083949`00000238
00000000`76f3b8ff 8b415f75`08403949 0x00000240`808b496c
00000000`76f3b907 00024880`3b411040 0x8b415f75`08403949
00000000`76f3b90f 01040000`a9527500 0x00024880`3b411040
00000000`76f3b917 00025090`8d491874 0x01040000`a9527500
00000000`76f3b91f c68a4418`488d4900 0x00025090`8d491874
00000000`76f3b927 c33a0000`117315ff 0xc68a4418`488d4900
00000000`76f3b92f 4e15ffcf`8b493374 0xc33a0000`117315ff
00000000`76f3b937 ff41cc8b`4900000e 0x4e15ffcf`8b493374
00000000`76f3b93f 00028c84`0fc63bd6 0xff41cc8b`4900000e
00000000`76f3b947 00028484`0fc73b00 0x00028c84`0fc63bd6
00000000`76f3b94f 6ee7e819`75c33b00 0x00028484`0fc73b00
00000000`76f3b957 c0331074`c33bfffa 0x6ee7e819`75c33b00
00000000`76f3b95f cf8b4900`000270e9 0xc0331074`c33bfffa
00000000`76f3b967 8b490000`0e1b15ff 0xcf8b4900`000270e9
00000000`76f3b96f 3b000013`e215ffcc 0x8b490000`0e1b15ff
00000000`76f3b977 0253e9c7`8b0775c7 0x3b000013`e215ffcc
00000000`76f3b97f 41fff959`4ae80000 0x0253e9c7`8b0775c7
00000000`76f3b987 c6844100`000002be 0x41fff959`4ae80000
00000000`76f3b98f 15ff0000`023d850f 0xc6844100`000002be
00000000`76f3b997 850f20a8`00000f65 0x15ff0000`023d850f
00000000`76f3b99f 245c8948`0000022f 0x850f20a8`00000f65
00000000`76f3b9a7 448d4c3e`4e8d4520 0x245c8948`0000022f
00000000`76f3b9af ffc933d6`8b416024 0x448d4c3e`4e8d4520
00000000`76f3b9b7 7cc33b00`0009f415 0xffc933d6`8b416024
00000000`76f3b9bf 730a7024`64ba0f0f 0x7cc33b00`0009f415
00000000`76f3b9c7 00000205`e9c68b07 0x730a7024`64ba0f0f
00000000`76f3b9cf cc8b49d6`8bfb8b44 0x00000205`e9c68b07
00000000`76f3b9d7 f08b44ff`fffdc4e8 0xcc8b49d6`8bfb8b44
00000000`76f3b9df e9c03307`7508f883 0xf08b44ff`fffdc4e8
00000000`76f3b9e7 7506f883`000001e9 0xe9c03307`7508f883
00000000`76f3b9ef c33bfffa`6e4be810 0x7506f883`000001e9
00000000`76f3b9f7 0001d4e9`c0330774 0xc33bfffa`6e4be810
00000000`76f3b9ff 86850f04`fe834100 0x0001d4e9`c0330774
00000000`76f3ba07 0000024a`ba000001 0x86850f04`fe834100
00000000`76f3ba0f 00b841ce`8b45c933 0x0000024a`ba000001
00000000`76f3ba17 fff7a249`e8000010 0x00b841ce`8b45c933
00000000`76f3ba1f 0775c33b`48e88b4c 0xfff7a249`e8000010
00000000`76f3ba27 48000001`a6e9c033 0x0775c33b`48e88b4c
00000000`76f3ba2f 24448948`3024448d 0x48000001`a6e9c033
00000000`76f3ba37 0000f024`8c8d4c20 0x24448948`3024448d
00000000`76f3ba3f 49000001`25b84100 0x0000f024`8c8d4c20
00000000`76f3ba47 8a0fe8cf`8b48d58b 0x49000001`25b84100
00000000`76f3ba4f 4166097c`c33bfffe 0x8a0fe8cf`8b48d58b
00000000`76f3ba57 39fe450f`44005d39 0x4166097c`c33bfffe
00000000`76f3ba5f 850f0000`00f0249c 0x39fe450f`44005d39
00000000`76f3ba67 240c8b49`000000bc 0x850f0000`00f0249c
00000000`76f3ba6f 40244489`48016348 0x240c8b49`000000bc
00000000`76f3ba77 24448948`10418b48 0x40244489`48016348
00000000`76f3ba7f 75c00000`06398148 0x24448948`10418b48
00000000`76f3ba87 480b7203`18798318 0x75c00000`06398148
00000000`76f3ba8f 50244489`4830418b 0x480b7203`18798318
00000000`76f3ba97 eb50245c`89481ceb 0x50244489`4830418b
00000000`76f3ba9f 8b480b72`18713915 0xeb50245c`89481ceb
00000000`76f3baa7 eb502444`89482041 0x8b480b72`18713915
00000000`76f3baaf 02ba5024`5c894805 0xeb502444`89482041
00000000`76f3bab7 0b721851`39000000 0x02ba5024`5c894805
00000000`76f3babf 24448948`28418b48 0x0b721851`39000000
00000000`76f3bac7 58245c89`4805eb58 0x24448948`28418b48
00000000`76f3bacf ba1d3808`74fb3b44 0x58245c89`4805eb58
00000000`76f3bad7 48d68b02`740006fd 0xba1d3808`74fb3b44
00000000`76f3badf 48000000`e824848d 0x48d68b02`740006fd
00000000`76f3bae7 20245489`28244489 0x48000000`e824848d
00000000`76f3baef c0334540`244c8d4c 0x20245489`28244489
00000000`76f3baf7 000144b9`04508d41 0xc0334540`244c8d4c
00000000`76f3baff ba00000d`7215ffd0 0x000144b9`04508d41
00000000`76f3bb07 8c8bc223`c0000000 0xba00000d`7215ffd0
00000000`76f3bb0f b8c23b00`0000e824 0x8c8bc223`c0000000
00000000`76f3bb17 89c8440f`00000006 0xb8c23b00`0000e824
00000000`76f3bb1f 07eb0000`00e8248c 0x89c8440f`00000006
00000000`76f3bb27 44000000`e8248c8b 0x07eb0000`00e8248c
00000000`76f3bb2f 7403f983`5d74fb3b 0x44000000`e8248c8b
00000000`76f3bb37 000000f0`249c3909 0x7403f983`5d74fb3b
00000000`76f3bb3f 0006fd4d`058a4f74 0x000000f0`249c3909
00000000`76f3bb47 f85f5ce8`4b75c33a 0x0006fd4d`058a4f74
00000000`76f3bb4f 448b3b75`5c5838ff 0xf85f5ce8`4b75c33a
00000000`76f3bb57 894c2824`44893024 0x448b3b75`5c5838ff
00000000`76f3bb5f 08244c8b`4d20246c 0x894c2824`44893024
00000000`76f3bb67 fec2c748`24048b4d 0x08244c8b`4d20246c
00000000`76f3bb6f b6e8cf8b`48ffffff 0xfec2c748`24048b4d
00000000`76f3bb77 fd130db6`0fffffea 0xb6e8cf8b`48ffffff
00000000`76f3bb7f 88ce4c0f`c33b0006 0xfd130db6`0fffffea
00000000`76f3bb87 ebfb8b00`06fd080d 0x88ce4c0f`c33b0006
00000000`76f3bb8f 3a0006fc`fe058a29 0xebfb8b00`06fd080d
00000000`76f3bb97 8b240c8b`491874c3 0x3a0006fc`fe058a29
00000000`76f3bb9f 060f15ff`cf8b4811 0x8b240c8b`491874c3
00000000`76f3bba7 0000f824`bc8b0000 0x060f15ff`cf8b4811
00000000`76f3bbaf 00f824bc`8b07eb00 0x0000f824`bc8b0000
00000000`76f3bbb7 331074eb`3b4c0000 0x00f824bc`8b07eb00
00000000`76f3bbbf 49000080`00b841d2 0x331074eb`3b4c0000
00000000`76f3bbc7 8bfff74b`5ae8cd8b 0x49000080`00b841d2
00000000`76f3bbcf c48148c6`8b02ebc7 0x8bfff74b`5ae8cd8b
00000000`76f3bbd7 5e415f41`000000a0 0xc48148c6`8b02ebc7
00000000`76f3bbdf c35b5e5f`5c415d41 0x5e415f41`000000a0
00000000`76f3bbe7 158ead00`00000090 0xc35b5e5f`5c415d41
00000000`76f3bbef 00000200`00000053 0x158ead00`00000090
00000000`76f3bbf7 09bc2400`00002500 0x00000200`00000053
00000000`76f3bbff 00000000`09b42400 0x09bc2400`00002500
00000000`76f3bc07 7e023553`158ead00 0x9b42400
00000000`76f3bc0f 00000400`00000a19 0x7e023553`158ead00
00000000`76f3bc17 09b42000`09bc2000 0x00000400`00000a19
00000000`76f3bc1f 445352bb`03197e00 0x09b42000`09bc2000
00000000`76f3bc27 4c886225`48e28953 0x445352bb`03197e00
00000000`76f3bc2f 4fb29af4`dfbb8344 0x4c886225`48e28953
00000000`76f3bc37 72656b00`0000020e 0x4fb29af4`dfbb8344
00000000`76f3bc3f 64702e32`336c656e 0x72656b00`0000020e
00000000`76f3bc47 00000000`00000062 0x64702e32`336c656e

We check for any Hidden Exceptions and find it was NULL Data Pointer:

0:003> .cxr
Resetting default scope

0:003> k
Child-SP RetAddr Call Site
00000000`0244e858 000007fe`fd061430 ntdll!NtWaitForMultipleObjects+0xa
00000000`0244e860 00000000`76ec1723 KERNELBASE!WaitForMultipleObjectsEx+0xe8
00000000`0244e960 00000000`76f3b5e5 kernel32!WaitForMultipleObjectsExImplementation+0xb3
00000000`0244e9f0 00000000`76f3b767 kernel32!WerpReportFaultInternal+0x215
00000000`0244ea90 00000000`76f3b7bf kernel32!WerpReportFault+0x77
00000000`0244eac0 00000000`76f3b9dc kernel32!BasepReportFault+0x1f
00000000`0244eaf0 00000000`77153398 kernel32!UnhandledExceptionFilter+0x1fc
00000000`0244ebd0 00000000`770d85c8 ntdll! ?? ::FNODOBFM::`string'+0x2365
00000000`0244ec00 00000000`770e9d2d ntdll!_C_specific_handler+0x8c
00000000`0244ec70 00000000`770d91cf ntdll!RtlpExecuteHandlerForException+0xd
00000000`0244eca0 00000000`77111248 ntdll!RtlDispatchException+0×45a
00000000`0244f380 00000000`ffdbdb27 ntdll!KiUserExceptionDispatch+0×2e
00000000`0244fab0 00000000`76eb59ed calc!CTimedCalc::WatchDogThread+0xb2
00000000`0244faf0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd
00000000`0244fb20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:003> dps 00000000`0244eca0 00000000`0244fab0
00000000`0244eca0 00000000`02450000
00000000`0244eca8 00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0)
00000000`0244ecb0 00000000`00012f00
00000000`0244ecb8 00000000`7711920a ntdll!RtlDosApplyFileIsolationRedirection_Ustr+0x3da
00000000`0244ecc0 00000000`00000005
00000000`0244ecc8 00000000`00000000
00000000`0244ecd0 00000000`00000000
00000000`0244ecd8 00000000`00000000
00000000`0244ece0 00000000`0244fb20
00000000`0244ece8 00000000`00000000
00000000`0244ecf0 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0)
00000000`0244ecf8 00000000`00000000
00000000`0244ed00 00000000`00000000
00000000`0244ed08 00000000`02450000
00000000`0244ed10 00000000`771e8180 ntdll!`string'+0xc040
00000000`0244ed18 00000000`0244b000
00000000`0244ed20 00000000`0244f250
00000000`0244ed28 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`0244ed30 00000000`770ec541 ntdll!RtlUserThreadStart+0x1d
00000000`0244ed38 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`0244ed40 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0)
00000000`0244ed48 00000000`0244fb20
00000000`0244ed50 00000000`771d7718 ntdll!LdrpDefaultExtension
00000000`0244ed58 00000000`0244ed80
00000000`0244ed60 00000000`770d852c ntdll!_C_specific_handler
00000000`0244ed68 00000000`771e8180 ntdll!`string'+0xc040
00000000`0244ed70 00000000`0244f250
00000000`0244ed78 00000000`00000000
00000000`0244ed80 00000000`00000000
00000000`0244ed88 00000000`00000000
00000000`0244ed90 00000000`00000000
00000000`0244ed98 00000000`00000000
00000000`0244eda0 00000000`00000000
00000000`0244eda8 00000000`00000000
00000000`0244edb0 00001f80`00000000
00000000`0244edb8 00000000`00000033
00000000`0244edc0 00010246`002b0000
00000000`0244edc8 00000000`00000000
00000000`0244edd0 00000000`00000000
00000000`0244edd8 00000000`00000000
00000000`0244ede0 00000000`00000000
00000000`0244ede8 000007fe`ff3625c0 msctf!s_szCompClassName
00000000`0244edf0 00000000`00200000
00000000`0244edf8 00000000`0244ee40
00000000`0244ee00 00000000`0244ee40
00000000`0244ee08 00000000`0244ee40
00000000`0244ee10 00000000`00000000
00000000`0244ee18 00000000`0244fb70
00000000`0244ee20 00000000`00000000
00000000`0244ee28 00000000`00000000
00000000`0244ee30 00000000`00000000
00000000`0244ee38 000007fe`fd602790 ole32!`string'
00000000`0244ee40 00000000`00292170
00000000`0244ee48 00000000`770e7a33 ntdll!LdrpFindOrMapDll+0x138
00000000`0244ee50 00000000`0244ef68
00000000`0244ee58 00000000`00000000
00000000`0244ee60 00000000`00000000
00000000`0244ee68 00000000`00000000
00000000`0244ee70 00000000`00000000
00000000`0244ee78 00000000`00000000
00000000`0244ee80 00000000`0000027f
00000000`0244ee88 00000000`00000000
00000000`0244ee90 00000000`00000000
00000000`0244ee98 0000ffff`00001f80
00000000`0244eea0 00000000`00000000
00000000`0244eea8 00000000`00000000
00000000`0244eeb0 00000000`00000000
00000000`0244eeb8 00000000`00000000
00000000`0244eec0 00000000`00000000
00000000`0244eec8 00000000`00000000
00000000`0244eed0 00000000`00000000
00000000`0244eed8 00000000`00000000
00000000`0244eee0 00000000`00000000
00000000`0244eee8 00000000`00000000
00000000`0244eef0 00000000`00000000
00000000`0244eef8 00000000`00000000
00000000`0244ef00 00000000`00000000
00000000`0244ef08 00000000`00000000
00000000`0244ef10 00000000`00000000
00000000`0244ef18 00000000`00000000
00000000`0244ef20 00000000`00000000
00000000`0244ef28 00000000`771192a8 ntdll!LdrpApplyFileNameRedirection+0x2d3
00000000`0244ef30 00000000`00000000
00000000`0244ef38 00000000`00000000
00000000`0244ef40 00000000`00000000
00000000`0244ef48 00000000`02080000
00000000`0244ef50 00000000`0244f028
00000000`0244ef58 00000000`0244f020
00000000`0244ef60 00000000`00000000
00000000`0244ef68 00000000`00000000
00000000`0244ef70 00000000`00000000
00000000`0244ef78 000007fe`fd602848 ole32!`string'
00000000`0244ef80 00000000`00000000
00000000`0244ef88 00000000`00000000
00000000`0244ef90 00000000`00000000
00000000`0244ef98 00000000`00000000
00000000`0244efa0 00000000`00000000
00000000`0244efa8 00000000`00000000
00000000`0244efb0 00000000`00000000
00000000`0244efb8 00000000`00000000
00000000`0244efc0 00000000`00000000
00000000`0244efc8 00000000`00000000
00000000`0244efd0 00000000`00000000
00000000`0244efd8 00000000`00000000
00000000`0244efe0 00000000`00000000
00000000`0244efe8 00000000`00000000
00000000`0244eff0 00000000`00000000
00000000`0244eff8 00000000`00000000
00000000`0244f000 00000000`00000000
00000000`0244f008 00000000`00000000
00000000`0244f010 00000000`00000000
00000000`0244f018 00000000`00000000
00000000`0244f020 00000000`0244f038
00000000`0244f028 00000000`0000011b
00000000`0244f030 00000000`024d0000
00000000`0244f038 00000080`001a024d
00000000`0244f040 00000000`01c0c8a0
00000000`0244f048 00000000`002f0101
00000000`0244f050 00000000`00000000
00000000`0244f058 00000000`00000022
00000000`0244f060 00000000`002f9b00
00000000`0244f068 00000000`01bd5390
00000000`0244f070 00000000`002f7c00
00000000`0244f078 00000000`01bd5580
00000000`0244f080 00000000`01bd57b0
00000000`0244f088 00000000`002f9b00
00000000`0244f090 00000000`00000000
00000000`0244f098 00000024`00000003
00000000`0244f0a0 00000000`002e91b0
00000000`0244f0a8 00000000`00000022
00000000`0244f0b0 00000000`771d5430 ntdll!RtlpInterceptorRoutines
00000000`0244f0b8 00000000`00000000
00000000`0244f0c0 00000000`00000010
00000000`0244f0c8 00000000`01bd0000
00000000`0244f0d0 00000000`00000008
00000000`0244f0d8 00000000`00000001
00000000`0244f0e0 00000000`01bd0288
00000000`0244f0e8 00000000`77113448 ntdll!RtlAllocateHeap+0xe4
00000000`0244f0f0 00000000`00000000
00000000`0244f0f8 00000000`00000001
00000000`0244f100 000002b2`000f002f
00000000`0244f108 00000000`01bd5780
00000000`0244f110 00000000`00250230
00000000`0244f118 00000000`000000df
00000000`0244f120 00000000`002551a0
00000000`0244f128 00000000`00255210
00000000`0244f130 00000000`002f9b00
00000000`0244f138 00000000`002551a0
00000000`0244f140 00000000`000000df
00000000`0244f148 00000000`10000010
00000000`0244f150 00000000`00250230
00000000`0244f158 00000000`00000000
00000000`0244f160 00000000`00250498
00000000`0244f168 00000000`0025026c
00000000`0244f170 00000000`002f9b00
00000000`0244f178 00000000`002551a0
00000000`0244f180 00000000`00000022
00000000`0244f188 00000000`76fd88b8 user32!GetPropW+0x4d
00000000`0244f190 00000000`00002974
00000000`0244f198 00000000`76fd88b8 user32!GetPropW+0x4d
00000000`0244f1a0 00000000`00250230
00000000`0244f1a8 00000000`76fd7931 user32!IsWindow+0x9
00000000`0244f1b0 00000000`002ed6d0
00000000`0244f1b8 00000000`76fd7931 user32!IsWindow+0x9
00000000`0244f1c0 00000000`00000000
00000000`0244f1c8 00000000`01c0c8d0
00000000`0244f1d0 00000000`01c0c8a0
00000000`0244f1d8 00000000`00000000
00000000`0244f1e0 00000000`00000008
00000000`0244f1e8 00000000`01bd0000
00000000`0244f1f0 00000000`00000000
00000000`0244f1f8 00000000`770f41c8 ntdll!RtlpReAllocateHeap+0x178
00000000`0244f200 00000000`00000002
00000000`0244f208 00000000`00000002
00000000`0244f210 00000000`00000000
00000000`0244f218 000007fe`4f00024d
00000000`0244f220 00000000`00000000
00000000`0244f228 000007fe`fb601381 uxtheme!CThemeWnd::_PreDefWindowProc+0x31
00000000`0244f230 00000000`00000082
00000000`0244f238 00000000`00000000
00000000`0244f240 00000000`7a337100
00000000`0244f248 00000000`01c0c8c0
00000000`0244f250 00000000`00000003
00000000`0244f258 00000000`76eb59e0 kernel32!BaseThreadInitThunk
00000000`0244f260 00000000`ffdbdb32 calc!CTimedCalc::Start+0xa9
00000000`0244f268 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0x0)
00000000`0244f270 00000000`ffe0ac64 calc!_dyn_tls_init_callback <PERF> (calc+0x7ac64)
00000000`0244f278 00000000`76ea0000 kernel32!TestResourceDataMatchEntry <PERF> (kernel32+0x0)
00000000`0244f280 00000000`76fadda0 kernel32!__PchSym_ <PERF> (kernel32+0x10dda0)
00000000`0244f288 00000000`770c0000 ntdll!RtlDeactivateActivationContext <PERF> (ntdll+0x0)
00000000`0244f290 00000000`77202dd0 ntdll!CsrPortMemoryRemoteDelta <PERF> (ntdll+0x142dd0)
00000000`0244f298 00000000`76fd760e user32!RealDefWindowProcW+0x5a
00000000`0244f2a0 00000000`00000001
00000000`0244f2a8 000007fe`fb600037 uxtheme!operator delete <PERF> (uxtheme+0x37)
00000000`0244f2b0 00000000`01bd0158
00000000`0244f2b8 00000000`00000082
00000000`0244f2c0 00000000`00000000
00000000`0244f2c8 00000000`00000003
00000000`0244f2d0 00000000`000111f2
00000000`0244f2d8 00000000`00000054
00000000`0244f2e0 00000000`00000000
00000000`0244f2e8 00000000`00000000
00000000`0244f2f0 00000000`00000001
00000000`0244f2f8 00000000`01c11c60
00000000`0244f300 00000000`0244f462
00000000`0244f308 00000000`01bd0230
00000000`0244f310 00000000`00000000
00000000`0244f318 00000000`00000000
00000000`0244f320 00000000`00000000
00000000`0244f328 00000000`14010015
00000000`0244f330 00000000`01c11570
00000000`0244f338 00000000`00000000
00000000`0244f340 00000000`00000000
00000000`0244f348 00000000`00000000
00000000`0244f350 00000000`00009c40
00000000`0244f358 00000000`00000000
00000000`0244f360 00000000`00000000
00000000`0244f368 00000000`00000000
00000000`0244f370 00000000`00002710
00000000`0244f378 00000000`77111248 ntdll!KiUserExceptionDispatch+0×2e
00000000`0244f380 00000000`0244f870
00000000`0244f388 00000000`0244f380
00000000`0244f390 00000000`00000000
00000000`0244f398 00000000`00000000
00000000`0244f3a0 000007fe`fb63fb40 uxtheme!$$VProc_ImageExportDirectory
00000000`0244f3a8 00000000`00000ad5
00000000`0244f3b0 00001f80`0010005f
00000000`0244f3b8 0053002b`002b0033
00000000`0244f3c0 00010246`002b002b
00000000`0244f3c8 00000000`00000000
00000000`0244f3d0 00000000`00000000
00000000`0244f3d8 00000000`00000000
00000000`0244f3e0 00000000`00000000
00000000`0244f3e8 00000000`00000000
00000000`0244f3f0 00000000`00000000
00000000`0244f3f8 00000000`0012c770
00000000`0244f400 00000000`00000000
00000000`0244f408 00000000`00000000
00000000`0244f410 00000000`00002710
00000000`0244f418 00000000`0244fab0
00000000`0244f420 00000000`00000000
00000000`0244f428 00000000`00000000
00000000`0244f430 00000000`00000000
00000000`0244f438 00000000`0244f938
00000000`0244f440 00000000`00962210
00000000`0244f448 00000000`00000000
00000000`0244f450 00000000`0244f9a0
00000000`0244f458 00000000`00009c40
00000000`0244f460 00000000`00000000
00000000`0244f468 00000000`00000000
00000000`0244f470 00000000`00000000
00000000`0244f478 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2
00000000`0244f480 00000000`0000027f
00000000`0244f488 00000000`00000000
00000000`0244f490 00000000`00000000
00000000`0244f498 0000ffff`00001f80
00000000`0244f4a0 00000000`00000000
00000000`0244f4a8 00000000`00000000
00000000`0244f4b0 00000000`00000000
00000000`0244f4b8 00000000`00000000
00000000`0244f4c0 00000000`00000000
00000000`0244f4c8 00000000`00000000
00000000`0244f4d0 00000000`00000000
00000000`0244f4d8 00000000`00000000
00000000`0244f4e0 00000000`00000000
00000000`0244f4e8 00000000`00000000
00000000`0244f4f0 00000000`00000000
00000000`0244f4f8 00000000`00000000
00000000`0244f500 00000000`00000000
00000000`0244f508 00000000`00000000
00000000`0244f510 00000000`00000000
00000000`0244f518 00000000`00000000
00000000`0244f520 00000000`00000000
00000000`0244f528 00000000`00000000
00000000`0244f530 00000000`00000000
00000000`0244f538 00000000`00000000
00000000`0244f540 00000000`00000000
00000000`0244f548 00000000`00000000
00000000`0244f550 00000000`00000000
00000000`0244f558 00000000`00000000
00000000`0244f560 00000000`00000000
00000000`0244f568 00000000`00000000
00000000`0244f570 00000000`00000000
00000000`0244f578 00000000`00000000
00000000`0244f580 00000000`00000000
00000000`0244f588 00000000`00000000
00000000`0244f590 00000000`00000000
00000000`0244f598 00000000`00000000
00000000`0244f5a0 00000000`00000000
00000000`0244f5a8 00000000`00000000
00000000`0244f5b0 00000000`00000000
00000000`0244f5b8 00000000`00000000
00000000`0244f5c0 00000000`00000000
00000000`0244f5c8 00000000`00000000
00000000`0244f5d0 00000000`00000000
00000000`0244f5d8 00000000`00000000
00000000`0244f5e0 00000000`00000000
00000000`0244f5e8 00000000`00000000
00000000`0244f5f0 00000000`00000000
00000000`0244f5f8 00000000`00000000
00000000`0244f600 00000000`00000000
00000000`0244f608 00000000`00000000
00000000`0244f610 00000000`00000000
00000000`0244f618 00000000`00000000
00000000`0244f620 00000000`00000000
00000000`0244f628 00000000`00000000
00000000`0244f630 00000000`00000000
00000000`0244f638 00000000`00000000
00000000`0244f640 00000000`00000000
00000000`0244f648 00000000`00000000
00000000`0244f650 00000000`00000000
00000000`0244f658 00000000`00000000
00000000`0244f660 00000000`00000000
00000000`0244f668 fffff800`032d5e53
00000000`0244f670 00000000`00000002
00000000`0244f678 00000000`00000000
00000000`0244f680 00000000`01c11580
00000000`0244f688 00000000`00000082
00000000`0244f690 00000000`00000082
00000000`0244f698 00000000`000111e4
00000000`0244f6a0 00000000`00000002
00000000`0244f6a8 00000000`0244f6f0
00000000`0244f6b0 00000000`00000002
00000000`0244f6b8 00000000`00000000
00000000`0244f6c0 00000000`000111e4
00000000`0244f6c8 00000000`00000000
00000000`0244f6d0 00000000`00000082
00000000`0244f6d8 00000000`00000000
00000000`0244f6e0 00000000`00000000
00000000`0244f6e8 00000000`76fe76c2 user32!DefDlgProcW+0×36
00000000`0244f6f0 00000000`00000000
00000000`0244f6f8 00000000`00000000
00000000`0244f700 00000000`000111e4
00000000`0244f708 00000000`00000000
00000000`0244f710 00000000`00000082
00000000`0244f718 00000000`00000000
00000000`0244f720 00000000`0244f908
00000000`0244f728 00000000`76fd9bef user32!UserCallWinProcCheckWow+0×1cb
00000000`0244f730 00000000`00962210
00000000`0244f738 00000000`00000001
00000000`0244f740 00000000`00000000
00000000`0244f748 00000000`00000000
00000000`0244f750 00000000`0244f768
00000000`0244f758 00000000`0244f778
00000000`0244f760 00000000`00000001
00000000`0244f768 00000000`00000000
00000000`0244f770 00000000`00000000
00000000`0244f778 00000000`00000000
00000000`0244f780 00000000`00000048
00000000`0244f788 00000000`00000001
00000000`0244f790 00000000`00000000
00000000`0244f798 00000000`00000000
00000000`0244f7a0 00000000`00000070
00000000`0244f7a8 ffffffff`ffffffff
00000000`0244f7b0 ffffffff`ffffffff
00000000`0244f7b8 00000000`76fd9b43 user32!UserCallWinProcCheckWow+0×99
00000000`0244f7c0 00000000`76fd9bef user32!UserCallWinProcCheckWow+0×1cb
00000000`0244f7c8 00000000`00000000
00000000`0244f7d0 00000000`00000000
00000000`0244f7d8 00000000`00000000
00000000`0244f7e0 00000000`00000000
00000000`0244f7e8 00000000`76fd72cb user32!DispatchClientMessage+0xc3
00000000`0244f7f0 00000000`00000000
00000000`0244f7f8 00000000`770e46b4 ntdll!NtdllDialogWndProc_W
00000000`0244f800 00000000`00000000
00000000`0244f808 00000000`00000000
00000000`0244f810 00000000`00000000
00000000`0244f818 00000000`00000000
00000000`0244f820 00000000`00962238
00000000`0244f828 00000000`00000001
00000000`0244f830 00000000`00000000
00000000`0244f838 00000000`00000000
00000000`0244f840 00000000`00000000
00000000`0244f848 00000000`00000000
00000000`0244f850 00000730`fffffb30
00000000`0244f858 000004d0`fffffb30
00000000`0244f860 00000170`000000f0
00000000`0244f868 0000002c`00000001
00000000`0244f870 00000000`c0000005
00000000`0244f878 00000000`00000000
00000000`0244f880 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2
00000000`0244f888 00000000`00000002
00000000`0244f890 00000000`00000000
00000000`0244f898 00000000`00000000
00000000`0244f8a0 00000000`00000000
00000000`0244f8a8 00000000`00000000
00000000`0244f8b0 00000000`00000000
00000000`0244f8b8 00000000`00000000
00000000`0244f8c0 00000000`00000000
00000000`0244f8c8 00000000`00000000
00000000`0244f8d0 00000000`00000000
00000000`0244f8d8 00000000`00000000
00000000`0244f8e0 00000000`00000000
00000000`0244f8e8 00000000`00000000
00000000`0244f8f0 00000000`00000000
00000000`0244f8f8 00000000`00000000
00000000`0244f900 00000000`00000000
00000000`0244f908 00000000`00962210
00000000`0244f910 00000000`ffdbdb27 calc!CTimedCalc::WatchDogThread+0xb2
00000000`0244f918 00000000`00000000
00000000`0244f920 00000000`00000000
00000000`0244f928 00000000`0244fab0
00000000`0244f930 00000000`77101530 ntdll!NtdllDispatchMessage_W
00000000`0244f938 00000000`76fe505b user32!DialogBox2+0×2ec
00000000`0244f940 00000000`00000000
00000000`0244f948 00000000`00000000
00000000`0244f950 00000000`00000000
00000000`0244f958 00000000`00000000
00000000`0244f960 00000000`00000000
00000000`0244f968 00000000`00000000
00000000`0244f970 00000000`00000000
00000000`0244f978 00000000`00000000
00000000`0244f980 00000000`00000002
00000000`0244f988 00000000`000111f0
00000000`0244f990 00000271`0f689359
00000000`0244f998 00000000`00000030
00000000`0244f9a0 00000000`00000000
00000000`0244f9a8 00000000`00000000
00000000`0244f9b0 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0×0)
00000000`0244f9b8 00000000`001a17e0
00000000`0244f9c0 00000000`00000000
00000000`0244f9c8 00000000`76fe4edd user32!InternalDialogBox+0×135
00000000`0244f9d0 00000000`00000000
00000000`0244f9d8 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`0244f9e0 00000000`00000000
00000000`0244f9e8 00000000`00000000
00000000`0244f9f0 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`0244f9f8 00000000`00000000
00000000`0244fa00 00000000`00000001
00000000`0244fa08 00000000`00000000
00000000`0244fa10 00000000`00000000
00000000`0244fa18 00000000`00009c40
00000000`0244fa20 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0×0)
00000000`0244fa28 00000000`76fe4f52 user32!DialogBoxIndirectParamAorW+0×58
00000000`0244fa30 00000000`001a17e0
00000000`0244fa38 00000000`00000000
00000000`0244fa40 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`0244fa48 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`0244fa50 00000000`00000000
00000000`0244fa58 00000000`00000001
00000000`0244fa60 00000000`ffd90000 calc!CCalculatorController::CCalculatorController <PERF> (calc+0×0)
00000000`0244fa68 00000000`76fdd476 user32!DialogBoxParamW+0×66
00000000`0244fa70 ffffffff`ffffffff
00000000`0244fa78 00000000`00000000
00000000`0244fa80 00000000`ffdcedb0 calc!CTimedCalc::TimeOutDlgProc
00000000`0244fa88 00000000`00000000
00000000`0244fa90 00000000`00000000
00000000`0244fa98 00000000`00000000
00000000`0244faa0 00000000`00000000
00000000`0244faa8 00000000`ffdbdafa calc!CTimedCalc::WatchDogThread+0×72
00000000`0244fab0 00000000`00002710

Segment registers and flags look normal now:

0:003> .cxr 00000000`0244f380
rax=000000000012c770 rbx=0000000000002710 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=0000000000000000
rip=00000000ffdbdb27 rsp=000000000244fab0 rbp=0000000000000000
r8=000000000244f938 r9=0000000000962210 r10=0000000000000000
r11=000000000244f9a0 r12=0000000000009c40 r13=0000000000000000
r14=0000000000000000 r15=0000000000000000
iopl=0 nv up ei pl zr na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010246
calc!CTimedCalc::WatchDogThread+0xb2:
00000000`ffdbdb27 488b01 mov rax,qword ptr [rcx] ds:00000000`00000000=????????????????

0:003> k
*** Stack trace for last set context - .thread/.cxr resets it
Child-SP RetAddr Call Site
00000000`0244fab0 00000000`76eb59ed calc!CTimedCalc::WatchDogThread+0xb2
00000000`0244faf0 00000000`770ec541 kernel32!BaseThreadInitThunk+0xd
00000000`0244fb20 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we suspect a particular thread doing I/O but IRP is missing in the output of !thread WinDbg command the best way is to examine the list of IRPs and associated threads from the output of !irpfind command. Here is a synthesized example from a few Virtualized Young System crash dumps:

0: kd> !thread fffffa8004e2d280

THREAD fffffa8004e2d280 Cid 0004.0020 Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (Executive) KernelMode Non-Alertable
fffff880009ec440 NotificationEvent
Not impersonating
[…]

0: kd> !irpfind

Irp [ Thread ] irpStack: (Mj,Mn) DevObj [Driver] MDL Process
[...]
fffffa800424e4e0 [fffffa8004e2d280] irpStack: (3, 0) fffffa8004ed6d40 [ \Driver\DriverA]
[…]

Now we can inspect the found IRP (!irp command) and device object (for example, by using !devobj and !devstack commands). Sometimes we can see the same IRP address as Execution Residue among “Args to Child” values in the output of !thread command or kv (if the thread is current). We call such pattern Hidden IRP.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When calculating effective addresses such as [r10+10h] or [rax+rcx*12h+40h] to show their value in the output of some commands such as .trap or .cxr a debugger uses CPU register values from a saved trap frame or context structure. If such information is invalid the reported effective address doesn’t correspond to the real one during code execution. So we call this analysis pattern False Effective Address similar to False Function Parameters. Therefore, if a fault address is saved during bugcheck or exception processing it may not correspond to the output of some commands where such calculation is necessary. For example, in a bugcheck parameter we have this referenced memory address:

Arg1: fffffadda17d001d, memory referenced

but the output of .trap command shows a NULL pointer address:

NOTE: The trap frame does not contain all registers.
Some register values may be zeroed or incorrect.
rax=0000000000000000 rbx=0000000000000000 rcx=0000000000000000
[...]
movzx eax,word ptr [rax+10h] 0010=????

Usually we are lucky and an effective address is correct despite the warning such as here and here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a variation of a general Wait Chain pattern related to CLR threads. When looking at Stack Trace Collection from a complete memory dump we may find threads using a monitor synchronization mechanism:

[... 32-bit ...]
09d2e908 6ba4d409 clr!CLREvent::WaitEx+0x106
09d2e91c 6bb90160 clr!CLREvent::Wait+0x19
09d2e9ac 6bb90256 clr!AwareLock::EnterEpilogHelper+0xa8
09d2e9ec 6bb9029b clr!AwareLock::EnterEpilog+0x42
09d2ea0c 6ba90f78 clr!AwareLock::Enter+0x5f
09d2eaa8 05952499 clr!JIT_MonEnterWorker_Portable+0xf8
[…]

or

[... 64-bit ...]
00000000`2094e230 000007fe`eedc3e3a clr!CLREvent::WaitEx+0xc1
00000000`2094e2d0 000007fe`eedc3d43 clr!AwareLock::EnterEpilogHelper+0xca
00000000`2094e3a0 000007fe`eee3e613 clr!AwareLock::EnterEpilog+0x63
00000000`2094e400 000007ff`007f4c38 clr!JIT_MonEnterWorker_Portable+0×14f
[…]

When seeing such threads we may ask for a process memory dump to perform .NET memory dump analysis using SOS or other WinDbg extensions such as in Deadlock pattern example for CLR 2 (mscorwks).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we look at a stack trace collection or it’s predicate subset and recognize that one of parameters is actually the same structure address or handle. We call this pattern Shared Structure. In x64 case we may possibly see it from the return address backwards disassembly (ub WinDbg command) but in x86 case most of the time we can spot that directly from the verbose stack trace, like in the snippet below (unless a parameter memory slot was reused):

THREAD 830f9990 Cid 0428.0e94 Teb: 7ffdf000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
[...]
ChildEBP RetAddr  Args to Child
0031f74c 7784b071 00000000 00000000 7ffdb000 ntdll!RtlpWaitOnCriticalSection+0x154
0031f774 00a91150 00a9b7a8 00000000 00a91452 ntdll!RtlEnterCriticalSection+0×152
WARNING: Stack unwind information not available. Following frames may be wrong.
0031f7c8 76113833 7ffdb000 0031f814 7784a9bd Application+0×1150
0031f7d4 7784a9bd 7ffdb000 003114bf 00000000 kernel32!BaseThreadInitThunk+0xe
0031f814 00000000 00a914a9 7ffdb000 00000000 ntdll!_RtlUserThreadStart+0×23

THREAD 886ee030 Cid 0428.0ef4 Teb: 7ffde000 Win32Thread: 00000000 WAIT: (UserRequest) UserMode Non-Alertable
[...]
ChildEBP RetAddr  Args to Child
0098fcb8 77f881b1 00000000 00000000 001614a0 ntdll!RtlpUnWaitCriticalSection+0x1b
0098fce0 00a9102e 00a9b7a8 00000000 00000000 ntdll!RtlEnterCriticalSection+0×152
WARNING: Stack unwind information not available. Following frames may be wrong.
0098fd28 00a91275 0098fd3c 76113833 001614a0 Application+0×102e
0098fd30 76113833 001614a0 0098fd7c 7784a9bd Application+0×1275
0098fd3c 7784a9bd 001614a0 009811d7 00000000 kernel32!BaseThreadInitThunk+0xe
0098fd7c 00000000 00a911ff 001614a0 00000000 ntdll!_RtlUserThreadStart+0×23

In case of multiple exceptions or even a single exception on one thread involving invalid access to a structure field the reference to the same structure on a different thread may point to possible synchronization problems.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we see the so called Small Values in memory (such as on raw stack) or in CPU registers which can be an ASCII or UNICODE value, some ID or even a handle. When in aggregates they can form a certain Semantic Structure such as a PID.TID example or Regular Data pattern. Here we illustrate a handle example (also an example of a Wait Chain analysis in user space):

0:000> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0016de78 000007fe`fcf010dc : 00000000`02c79fa0 00000000`08c3faf0 00000000`021551f0 00000000`08c3fb00 : ntdll!NtWaitForSingleObject+0xa
00000000`0016de80 000007fe`f90e6d7f : 00000000`10b40010 00000000`10b40010 00000000`00000000 00000000`000007e0 : KERNELBASE!WaitForSingleObjectEx+0×79
[…]

0:000> !handle 00000000`000007e0 ff
Handle 00000000000007d0
  Type          Thread
  Attributes    0
  GrantedAccess 0x1fffff:
         Delete,ReadControl,WriteDac,WriteOwner,Synch
         Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken,
Impersonate,DirectImpersonate
  HandleCount   5
  PointerCount  9
  Name          <none>
  Object specific information
    Thread Id   278c.a58
    Priority    13
    Base Priority 0

0:000> ~~[a58]s
ntdll!NtWaitForMultipleObjects+0xa:
00000000`770c186a c3              ret

0:002> kv
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000000`0f6af758 000007fe`fcf01430 : 00000000`00000025 00000000`00000000 00000000`00000000 000007fe`e35a1fb0 : ntdll!NtWaitForMultipleObjects+0xa
00000000`0f6af760 00000000`76e61220 : 00000000`0f6af8a8 00000000`0f6af890 00000000`00000000 00000000`00000000 : KERNELBASE!WaitForMultipleObjectsEx+0xe8
[...]

0:026> dp 00000000`0f6af890 L4
00000000`0f6af890  00000000`00000dbc 00000000`000007c0
00000000`0f6af8a0  00000000`00000000 00000000`00000000

0:002> !handle dbc ff
Handle 0000000000000dbc
  Type          Thread
  Attributes    0
  GrantedAccess 0x1fffff:
         Delete,ReadControl,WriteDac,WriteOwner,Synch
         Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken,
Impersonate,DirectImpersonate
  HandleCount   2
  PointerCount  4
  Name          <none>
  Object specific information
    Thread Id   278c.24ac
    Priority    14
    Base Priority 0

0:002> !handle 7c0 ff
Handle 00000000000007c0
  Type          Thread
  Attributes    0
  GrantedAccess 0x1fffff:
         Delete,ReadControl,WriteDac,WriteOwner,Synch
         Terminate,Suspend,Alert,GetContext,SetContext,SetInfo,QueryInfo,SetToken,
Impersonate,DirectImpersonate
  HandleCount   2
  PointerCount  4
  Name          <none>
  Object specific information
    Thread Id   278c.628
    Priority    14
    Base Priority 0

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes there are similar crashes in multiplatform products where only some potion of Crash Signature is similar. We call such a pattern Crash Signature Invariant, for example:

x86: cmp dword ptr [eax], 1
x64: cmp dword ptr [r10]. 1

One crash dump had the following condensed stack trace: 

0: kd> kc
DriverA
win32k!DrvSetMonitorPowerState
win32k!xxxSysCommand
win32k!xxxRealDefWindowProc
win32k!NtUserfnNCDESTROY
win32k!NtUserMessageCall
nt!KiSystemServiceCopyEnd

with the following faulting instruction:

DriverA+0x1234:
cmp     dword ptr [r11],1 ds:002b:00000000`00000000=????????

A search for DriverA led to this x86 crash analysed some time ago:

0: kd> kc
DriverA
nt!IopfCallDriver
win32k!GreDeviceIoControl
win32k!DrvSetMonitorPowerState
win32k!xxxSysCommand
win32k!xxxRealDefWindowProc
win32k!xxxWrapRealDefWindowProc
win32k!NtUserfnNCDESTROY
win32k!NtUserMessageCall
nt!KiSystemServicePostCall

0: kd> r
DtiverA+0x1423:
cmp     dword ptr [ecx],1    ds:0023:00000000=????????

We see common function names on both stack traces and overall flow is the same (only 3 functions are omitted in x64 trace); we see the same NULL pointer dereference for the same comparison instruction with the same comparison operand, #1.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a kernel space counterpart of Ubiquitous Component pattern. Such a component especially when it is Top Module can be a sign of Wait Chain(s) and Blocking Module and if it is present in the same process names - a sign of Distributed Wait Chain.

0: kd> !stacks 0 ModuleA
Proc.Thread  .Thread  Ticks   ThreadState Blocker

                            [fffffa800e673b30 svchost.exe]
 534.006240  fffffa801388f5f0 fffd41d9 Blocked    ModuleA+0x12468

                            [fffffa800e705b30 svchost.exe]
 630.000e14  fffffa800edacb50 fffdcf7a Blocked    ModuleA+0x12468
 630.000f04  fffffa8012c2fb50 fffdcf49 Blocked    ModuleA+0x12468
 630.006610  fffffa80134f5b50 fffdcf46 Blocked    ModuleA+0x12468
 630.001cfc  fffffa800f55a2d0 fffdcf44 Blocked    ModuleA+0x12468
 630.003db8  fffffa80121f1540 fffdcf43 Blocked    ModuleA+0x12468
 630.000b9c  fffffa80133d1780 fffdcf3c Blocked    ModuleA+0x12468
 630.0041c4  fffffa8013c77b50 fffdcf43 Blocked    ModuleA+0x12468
 630.00641c  fffffa8012476b50 fffdcf43 Blocked    ModuleA+0x12468
 630.006424  fffffa8013207b50 fffdcf40 Blocked    ModuleA+0x12468
 630.002fcc  fffffa80128f9060 fffdcf3e Blocked    ModuleA+0x12468
 630.003de8  fffffa80139edb50 fffdcf3d Blocked    ModuleA+0x12468
 630.0062c4  fffffa800f5ff2d0 fffdcf3c Blocked    ModuleA+0x12468
 630.0065e8  fffffa80139dcb50 fffdcf3b Blocked    ModuleA+0x12468
 630.004524  fffffa8011e51b50 fffdcf3a Blocked    ModuleA+0x12468
 630.004570  fffffa801346b060 fffdcf39 Blocked    ModuleA+0x12468
 630.00173c  fffffa8010b99b50 fffdcf39 Blocked    ModuleA+0x12468

                            [fffffa800f63db30 iexplore.exe]
24c4.0024c8  fffffa800fe854e0 fffcb6cf Blocked    ModuleA+0x12468

                            [fffffa8010b9ab30 explorer.exe]
2b64.0043d0  fffffa8012e8ab00 fffd9095 Blocked    ModuleA+0x12468

                            [fffffa800fe55060 explorer.exe]
2c80.002e58  fffffa8012e75060 fffba7af Blocked    ModuleA+0x12468

                            [fffffa8010c54b30 iexplore.exe]
2e3c.002e98  fffffa8010c75620 fffcbb7f Blocked    ModuleA+0x12468

                            [fffffa80111c3720 iexplore.exe]
32d8.003230  fffffa80111b1b00 fffd41d9 Blocked    ModuleA+0x12468

                            [fffffa80110cb690 iexplore.exe]
2e74.002854  fffffa8011121b00 fffbe8a4 Blocked    ModuleA+0x12468

                            [fffffa801146cb30 OUTLOOK.EXE]
35cc.0035e8  fffffa8013831b00 fffaf33a Blocked    ModuleA+0x12468

                            [fffffa80105a5640 OUTLOOK.EXE]
3858.00385c  fffffa801133ab00 fffd3691 Blocked    ModuleA+0x12468

                            [fffffa8011998060 explorer.exe]
3d70.004a0c  fffffa80139ddb00 fffd0482 Blocked    ModuleA+0x12468

                            [fffffa8010ff5850 OUTLOOK.EXE]
3540.000458  fffffa8011052b00 fffbd007 Blocked    ModuleA+0x12468

                            [fffffa8011d3d060 OUTLOOK.EXE]
49f8.0049fc  fffffa8011c78060 fffdbbf9 Blocked    ModuleA+0x12468

                            [fffffa801241b060 OUTLOOK.EXE]
4888.005af0  fffffa8012e8eab0 fffae442 Blocked    ModuleA+0x12468
4888.003d24  fffffa800eca7b00 fffae443 Blocked    ModuleA+0x12468

                            [fffffa8012687b30 explorer.exe]
5048.0051fc  fffffa801129cb00 fffca8bf Blocked    ModuleA+0x12468

                            [fffffa8011c1e060 OUTLOOK.EXE]
52c4.00117c  fffffa80130f8710 fffaa157 Blocked    ModuleA+0x12468
52c4.0045fc  fffffa801374f060 fffaa15e Blocked    ModuleA+0x12468

                            [fffffa8011c42b30 explorer.exe]
5898.0001ec  fffffa80137a1b00 fffd8da0 Blocked    ModuleA+0x12468

                            [fffffa8012e04b30 OUTLOOK.EXE]
5a74.004954  fffffa8012e05060 fffa9ff8 Blocked    ModuleA+0x12468

                            [fffffa8010908b30 spoolsv.exe]
2724.004190  fffffa8011ea1060 fffdcafb Blocked    ModuleA+0x12468

                            [fffffa801206eb30 WerFault.exe]
3e50.005424  fffffa8013c5eb00 fffdcf39 Blocked    ModuleA+0x12468

                            [fffffa800f8cf2a0 WerFault.exe]
 9f4.00570c  fffffa8013c8ab00 fffdca9f Blocked    ModuleA+0x12468

                            [fffffa8013af1060 WerFault.exe]
3c74.002b80  fffffa8013c5c060 fffd9dc8 Blocked    ModuleA+0x12468

                            [fffffa800f8053a0 WINWORD.EXE]
3dd0.0066a8  fffffa800ce618c0 fffd7c02 Blocked    ModuleA+0x12468

                            [fffffa8010b66b30 WINWORD.EXE]
62a4.001934  fffffa801368c430 fffd7ce7 Blocked    ModuleA+0x12468

                            [fffffa80141dc060 WerFault.exe]
17d0.0052e4  fffffa801347a060 fffd57b8 Blocked    ModuleA+0x12468

                            [fffffa8012629760 WerFault.exe]
621c.005b64  fffffa8011e395d0 fffc8dc2 Blocked    ModuleA+0x12468

                            [fffffa80131a75d0 explorer.exe]
4884.002b34  fffffa8013dc3b00 fffd67bc Blocked    ModuleA+0x12468

[...]

Threads Processed: 5948

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Processes with one thread like Notepad are rare. Such a process is always suspicious especially if it is a service or belongs to a complex product. We call such a pattern One-Thread Process. Usually this happens when all other threads terminated and the remaining thread is blocked in some wait chain. For example, this process has a thread which is blocked in an ALPC request to itself (the same process):

0: kd> !process fffffa8013ed9b30 ff
PROCESS fffffa8013ed9b30
    SessionId: 0  Cid: 44b4    Peb: 7fffffd8000  ParentCid: 0114
    DirBase: 2da448000  ObjectTable: fffff8a01948c670  HandleCount: 660.
    Image: ServiceA.exe
    VadRoot fffffa801356dd10 Vads 398 Clone 0 Private 5795. Modified 204253. Locked 0.
    DeviceMap fffff8a000008340
    Token                             fffff8a01b546060
    ElapsedTime                       01:32:37.622
    UserTime                          00:00:01.421
    KernelTime                        00:00:01.578
    QuotaPoolUsage[PagedPool]         0
    QuotaPoolUsage[NonPagedPool]      0
    Working Set Sizes (now,min,max)  (1525, 50, 345) (6100KB, 200KB, 1380KB)
    PeakWorkingSetSize                7607
    VirtualSize                       178 Mb
    PeakVirtualSize                   182 Mb
    PageFaultCount                    752709
    MemoryPriority                    BACKGROUND
    BasePriority                      8
    CommitCharge                      8043

        THREAD fffffa8012caab50  Cid 44b4.4f70  Teb: 000007fffff5a000 Win32Thread: 0000000000000000 WAIT: (WrLpcReply) KernelMode Non-Alertable
            fffffa8012caaf18  Semaphore Limit 0x1
        Waiting for reply to ALPC Message fffff8a0194d4780 : queued at port fffffa8012911c80 : owned by process fffffa8013ed9b30
        IRP List:
            fffffa8013923300: (0006,0118) Flags: 00060000  Mdl: 00000000
        Not impersonating
        DeviceMap                 fffff8a000008340
        Owning Process            fffffa8013ed9b30       Image:         ServiceA.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      139828         Ticks: 347372 (0:01:30:27.687)
        Context Switch Count      7380            
        UserTime                  00:00:00.031
        KernelTime                00:00:04.890
        Win32 Start Address ServiceA (0×00000001401156e0)
        Stack Init fffff88014c9ddb0 Current fffff88014c9c6b0
        Base fffff88014c9e000 Limit fffff88014c98000 Call 0
        Priority 8 BasePriority 8 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
        Child-SP          RetAddr           Call Site
        fffff880`14c9c6f0 fffff800`01873652 nt!KiSwapContext+0×7a
        fffff880`14c9c830 fffff800`01884a9f nt!KiCommitThreadWait+0×1d2
        fffff880`14c9c8c0 fffff800`0189f04f nt!KeWaitForSingleObject+0×19f
        fffff880`14c9c960 fffff800`01b919f6 nt!AlpcpSignalAndWait+0×8f
        fffff880`14c9ca10 fffff800`01b910f0 nt!AlpcpReceiveSynchronousReply+0×46
        fffff880`14c9ca70 fffff800`01b9519d nt!AlpcpProcessSynchronousRequest+0×33d
        fffff880`14c9cbb0 fffff800`01b95276 nt!LpcpRequestWaitReplyPort+0×9c
        fffff880`14c9cc10 fffff800`0187ced3 nt!NtRequestWaitReplyPort+0×76
        fffff880`14c9cc60 fffff800`01879490 nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`14c9cc60)
        fffff880`14c9cdf8 fffff880`05c31050 nt!KiServiceLinkage
        fffff880`14c9ce70 fffff880`045ce005 ModuleA+0×12468
        […]
        fffff880`14c9da10 fffff800`01b9d3b6 nt!IopXxxControlFile+0×607
        fffff880`14c9db40 fffff800`0187ced3 nt!NtDeviceIoControlFile+0×56
        fffff880`14c9dbb0 00000000`76d8138a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`14c9dc20)
        00000000`082af028 000007fe`fd366cf6 ntdll!NtDeviceIoControlFile+0xa
        00000000`082af030 00000000`76c2683f KERNELBASE!TlsGetValue+0×1a36
        00000000`082af0a0 00000001`4019d38c kernel32!DeviceIoControlImplementation+0×7f
        […]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

All previous wait chain patterns were about single wait chains. However, it is often a case when there are many different wait chains in a memory dump especially in terminal services environments. There can be ALPC and critical section wait chains at the same time. The can be related or completely disjoint. Here we call a special case of several wait chains having the same structure (and possibly pointing to one direction) Distributed Wait Chain. One such example we put below. In a stack trace collection from a complete memory dump from a hanging system we found several explorer.exe processes with critical section wait chains having the same structure and endpoint of a top and blocking ModuleA:

THREAD fffffa80137cf060  Cid 4884.4f9c  Teb: 000007fffffaa000 Win32Thread: fffff900c0fb98b0 WAIT: (UserRequest) UserMode Non-Alertable
    fffffa8013570dc0  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a014e21d90
Owning Process            fffffa80131a75d0       Image:         explorer.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      274752         Ticks: 212448 (0:00:55:19.500)
Context Switch Count      9889                 LargeStack
UserTime                  00:00:00.093
KernelTime                00:00:00.171
Win32 Start Address SHLWAPI!WrapperThreadProc (0×000007fefdafc608)
Stack Init fffff88013c25db0 Current fffff88013c25900
Base fffff88013c26000 Limit fffff88013c1b000 Call 0
Priority 11 BasePriority 9 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Kernel stack not resident.
Child-SP          RetAddr           Call Site
fffff880`13c25940 fffff800`01873652 nt!KiSwapContext+0×7a
fffff880`13c25a80 fffff800`01884a9f nt!KiCommitThreadWait+0×1d2
fffff880`13c25b10 fffff800`01b7768e nt!KeWaitForSingleObject+0×19f
fffff880`13c25bb0 fffff800`0187ced3 nt!NtWaitForSingleObject+0xde
fffff880`13c25c20 00000000`76d8135a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`13c25c20)
00000000`0489e518 00000000`76d7e4e8 ntdll!ZwWaitForSingleObject+0xa
00000000`0489e520 00000000`76d7e3db ntdll!RtlpWaitOnCriticalSection+0xe8
00000000`0489e5d0 000007fe`fdf8ff50 ntdll!RtlEnterCriticalSection+0xd1
00000000`0489e600 000007fe`fdf8fbd3 SHELL32!CFSFolder::GetIconOf+0×24b
00000000`0489f3a0 000007fe`fdf903d3 SHELL32!SHGetIconIndexFromPIDL+0×3f
00000000`0489f3d0 00000000`ff900328 SHELL32!SHMapIDListToSystemImageListIndexAsync+0×73
00000000`0489f470 00000000`ff8fff4b Explorer!SFTBarHost::AddImageForItem+0×9c
00000000`0489f4d0 00000000`ff8fd2f1 Explorer!SFTBarHost::_InternalRepopulateList+0×4ad
00000000`0489f5d0 00000000`ff8fd0b4 Explorer!SFTBarHost::_RepopulateList+0×1f3
00000000`0489f600 00000000`ff8fcccd Explorer!SFTBarHost::_OnBackgroundEnumDone+0xc1
00000000`0489f630 00000000`ff8fc9e2 Explorer!SFTBarHost::_WndProc+0×451
00000000`0489f680 00000000`76669bd1 Explorer!SFTBarHost::_WndProc_ProgramsMFU+0×1b
00000000`0489f6b0 00000000`766698da USER32!UserCallWinProcCheckWow+0×1ad
00000000`0489f770 00000000`ff8f1177 USER32!DispatchMessageWorker+0×3b5
00000000`0489f7f0 00000000`ff9130e9 Explorer!CTray::_MessageLoop+0×446
00000000`0489f880 000007fe`fdafc71e Explorer!CTray::MainThreadProc+0×8a
00000000`0489f8b0 00000000`76c2652d SHLWAPI!WrapperThreadProc+0×19b
00000000`0489f9b0 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd
00000000`0489f9e0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0: kd> .process /r /p fffffa80131a75d0
Implicit process is now fffffa80`131a75d0
Loading User Symbols

0: kd> !cs -l -o -s
-----------------------------------------
DebugInfo          = 0x0000000000499d90
Critical section   = 0x000007fefe3d5900 (SHELL32!g_csIconCache+0x0)
LOCKED
LockCount          = 0×2
WaiterWoken        = No
OwningThread       = 0×0000000000002b34
RecursionCount     = 0×1
LockSemaphore      = 0×7F8
SpinCount          = 0×0000000000000000
OwningThread       = .thread fffffa8013dc3b00

THREAD fffffa8013dc3b00  Cid 4884.2b34  Teb: 000007fffffac000 Win32Thread: fffff900c2bc1010 WAIT: (Executive) KernelMode Non-Alertable
    fffff88011c03600  SynchronizationEvent
IRP List:
    fffffa800f8fc790: (0006,0430) Flags: 00000404  Mdl: 00000000
Not impersonating
DeviceMap                 fffff8a014e21d90
Owning Process            fffffa80131a75d0       Image:         explorer.exe
Attached Process          N/A            Image:         N/A
Wait Start TickCount      170052         Ticks: 317148 (0:01:22:35.437)
Context Switch Count      2                 LargeStack
UserTime                  00:00:00.000
KernelTime                00:00:00.000
Win32 Start Address SHELL32!ShutdownThreadProc (0x000007fefe13ef54)
Stack Init fffff88011c03db0 Current fffff88011c03320
Base fffff88011c04000 Limit fffff88011bfd000 Call 0
Priority 11 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`11c03360 fffff800`01873652 nt!KiSwapContext+0x7a
fffff880`11c034a0 fffff800`01884a9f nt!KiCommitThreadWait+0x1d2
fffff880`11c03530 fffff880`05c12383 nt!KeWaitForSingleObject+0x19f
fffff880`11c035d0 fffff880`012b9288 ModuleA+0×12468
fffff880`11c03750 fffff880`012b7d1b fltmgr!FltpPerformPostCallbacks+0×368
fffff880`11c03820 fffff880`012b66df fltmgr!FltpLegacyProcessingAfterPreCallbacksCompleted+0×39b
fffff880`11c038b0 fffff880`01b895ff fltmgr!FltpDispatch+0xcf
fffff880`11c03a30 fffff800`01b783b4 nt!IopCloseFile+0×11f
fffff880`11c03ac0 fffff800`01b78171 nt!ObpDecrementHandleCount+0xb4
fffff880`11c03b40 fffff800`01b78734 nt!ObpCloseHandleTableEntry+0xb1
fffff880`11c03bd0 fffff800`0187ced3 nt!ObpCloseHandle+0×94
fffff880`11c03c20 00000000`76d8140a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`11c03c20)
00000000`0754f348 000007fe`fd341873 ntdll!NtClose+0xa
00000000`0754f350 00000000`76c32f51 KERNELBASE!CloseHandle+0×13
00000000`0754f380 000007fe`fdaf9690 kernel32!CloseHandleImplementation+0×3d
00000000`0754f490 000007fe`fe191d7f SHLWAPI!CFileStream::Release+0×84
00000000`0754f4c0 000007fe`fe13ed57 SHELL32!IconCacheSave+0×2b7
00000000`0754f780 000007fe`fe13f0c6 SHELL32!CommonRestart+0×2f
00000000`0754f7f0 00000000`76c2652d SHELL32!ShutdownThreadProc+0×172
00000000`0754f820 00000000`76d5c521 kernel32!BaseThreadInitThunk+0xd
00000000`0754f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

There is an extension shortcut to the usual WinDbg command .effmach for 64-bit memory dumps of 32-bit processes:

0:000> .load wow64exts

0:000> !sw

Switched to 32bit mode

0:000:x86> !sw

Switched to 64bit mode

Also !k command will display both thread stacks (32-bit and 64-bit):

0:000> !k
Walking 64bit Stack...
Child-SP          RetAddr           Call Site
00000000`0016e018 00000000`74f9aea8 wow64win!NtUserGetMessage+0xa
00000000`0016e020 00000000`74fecf87 wow64win!whNtUserGetMessage+0x30
00000000`0016e080 00000000`74f72776 wow64!Wow64SystemServiceEx+0xd7
00000000`0016e940 00000000`74fed07e wow64cpu!ServiceNoTurbo+0x2d
00000000`0016ea00 00000000`74fec549 wow64!RunCpuSimulation+0xa
00000000`0016ea50 00000000`77c54956 wow64!Wow64LdrpInitialize+0x429
00000000`0016efa0 00000000`77c51a17 ntdll!LdrpInitializeProcess+0x17e4
00000000`0016f490 00000000`77c3c32e ntdll! ?? ::FNODOBFM::`string'+0x29220
00000000`0016f500 00000000`00000000 ntdll!LdrInitializeThunk+0xe
Walking 32bit Stack...
ChildEBP RetAddr
002cf6a0 76ba790d user32!NtUserGetMessage+0x15
002cf6bc 0048148a user32!GetMessageW+0x33
002cf6fc 004816ec notepad!WinMain+0xe6
002cf78c 755533aa notepad!_initterm_e+0x1a1
002cf798 77e29ef2 kernel32!BaseThreadInitThunk+0xe
002cf7d8 77e29ec5 ntdll_77df0000!__RtlUserThreadStart+0x70
002cf7f0 00000000 ntdll_77df0000!_RtlUserThreadStart+0x1b

However, I don’t recommend its usage in iterative scripts because if something goes wrong at one iteration then all subsequent !sw commands will trigger the wrong machine mode but explicit .effmach will set the correct one.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This pattern (we call it String Hint) covers traces of ASCII and UNICODE strings that look suspicious such as website, password and HTTP forms or strange names that intuitively shouldn’t be present according to the purpose of a module or its container process (example is taken from Victimware presentation case study):

0:005> s-sa 00040000 L1d000
0004004d  "!This program cannot be run in D"
0004006d  "OS mode."
00040081  "3y@"
000400b8  "Rich"
000401d0  ".text"
000401f7  "`.rdata"
0004021f  "@.data"
00040248  ".reloc"
[...]
00054018  "GET /stat?uptime=%d&downlink=%d&"
00054038  "uplink=%d&id=%s&statpass=%s&comm"
00054058  "ent=%s HTTP/1.0"
000540ac  "%s%s%s"
000540d8  "ftp://%s:%s@%s:%d"
000540fc  "Accept-Encoding:"
00054118  "Accept-Encoding:"
00054130  "0123456789ABCDEF"
00054144  "://"
00054160  "POST %s HTTP/1.0"
00054172  "Host: %s"
0005417c  "User-Agent: %s"
0005418c  "Accept: text/html"
0005419f  "Connection: Close"
000541b2  "Content-Type: application/x-www-"
000541d2  "form-urlencoded"
000541e3  "Content-Length: %d"
000541fc  "id="
00054208  "POST %s HTTP/1.1"
0005421a  "Host: %s"
00054224  "User-Agent: %s"
00054234  "Accept: text/html"
00054247  "Connection: Close"
0005425a  "Content-Type: application/x-www-"
0005427a  "form-urlencoded"
0005428b  "Content-Length: %d"
000542a4  "id=%s&base="
000542b8  "id=%s&brw=%d&type=%d&data="
000542d8  "POST %s HTTP/1.1"
000542ea  "Host: %s"
000542f4  "User-Agent: %s"
00054304  "Accept: text/html"
00054317  "Connection: Close"
0005432a  "Content-Type: application/x-www-"
0005434a  "form-urlencoded"
0005435b  "Content-Length: %d"
00054378  "id=%s&os=%s&plist="
00054390  "POST %s HTTP/1.1"
000543a2  "Host: %s"
000543ac  "User-Agent: %s"
000543bc  "Accept: text/html"
000543cf  "Connection: Close"
000543e2  "Content-Type: application/x-www-"
00054402  "form-urlencoded"
00054413  "Content-Length: %d"
00054430  "id=%s&data=%s"
00054440  "POST %s HTTP/1.1"
00054452  "Host: %s"
0005445c  "User-Agent: %s"
0005446c  "Accept: text/html"
0005447f  "Connection: Close"
00054492  "Content-Type: application/x-www-"
000544b2  "form-urlencoded"
000544c3  "Content-Length: %d"
000544e0  "GET %s HTTP/1.0"
000544f1  "Host: %s"
000544fb  "User-Agent: %s"
0005450b  "Connection: close"
00054528  "POST /get/scr.html HTTP/1.0"
00054545  "Host: %s"
0005454f  "User-Agent: %s"
0005455f  "Connection: close"
00054572  "Content-Length: %d"
00054586  "Content-Type: multipart/form-dat"
000545a6  "a; boundary=--------------------"
000545c6  "-------%d"
000545d4  "-----------------------------%d"
000545f8  "%sContent-Disposition: form-data"
00054618  "; name="id""
00054630  "%sContent-Disposition: form-data"
00054650  "; name="screen"; filename="%d""
00054670  "Content-Type: application/octet-"
00054690  "stream"
000546a0  "%s(%d) : %s"
000546ac  "%s failed with error %d: %s"
000546c8  "%02X"
000546d8  "BlackwoodPRO"
000546e8  "FinamDirect"
000546f4  "GrayBox"
000546fc  "MbtPRO"
00054704  "Laser"
0005470c  "LightSpeed"
00054718  "LTGroup"
00054720  "Mbt"
00054724  "ScotTrader"
00054730  "SaxoTrader"
00054740  "Program:   %s"
0005474f  "Username:  %s"
0005475e  "Password:  %s"
0005476d  "AccountNO: %s"
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Whereas some false positives can be considered soft debugger bugs false negatives can have more severe impact on software behavior analysis especially in malware analysis. We name this pattern Debugger Omission. Typical example here is current .imgscan command which according to documentation should by default scan virtual process space for MZ/PE signatures. Unfortunately it doesn’t detect such signatures in resource pages (we haven’t checked stack regions yet):

0000000000fd0000 image base

SECTION HEADER #4
.rsrc name
6430 virtual size
4000 virtual address
6600 size of raw data
1600 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only

0:000> .imgscan /r 00000000`00fd4000 L200

0:000> s -[l2]sa 00000000`00fd4000 l200
00000000`00fd40b0  "MZ"
00000000`00fd40fd  "!This program cannot be run in D"
00000000`00fd411d  "OS mode."
00000000`00fd4188  "Rich"
00000000`00fd4198  "PE"

0:000> !dh 00000000`00fd40b0

File Type: DLL
FILE HEADER VALUES
14C machine (i386)
3 number of sections
time date stamp Fri Jan 18 21:27:25 2013

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2102 characteristics
Executable
32 bit word machine
DLL
[...]

Another other analysis scenarios found will be added to this pattern. Milder version of it includes !analyze -v that shows us a breakpoint instead of an exception violation from a parallel thread.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have a Broken Link for some reason, either from memory corruption, Lateral Damage or Truncated Dump. For example, an active process list enumeration stops after showing some processes (!for_each_thread and !vm also don’t work):

0: kd> !process 0 ff

[...]

TYPE mismatch for process object at fffffa80041da5c0

0: kd> !validatelist nt!PsActiveProcessHead
Blink at address fffffa80041da748 does not point back to previous at fffffa8005bc8cb8

Here we can either try to repair or navigate links manually or use other means such as dumping pool allocations for process structures with Proc pool tag:

0: kd> !poolfind Proc

Searching NonPaged pool (fffffa80032fc000 : ffffffe000000000) for Tag: Proc

*fffffa80033879a0 size:  510 previous size:   a0  (Allocated) Proc (Protected)
*fffffa80033ffad0 size:  530 previous size:  280  (Allocated) Proc (Protected)
*fffffa80041a2af0 size:  510 previous size:   90  (Allocated) Proc (Protected)
*fffffa800439c5c0 size:  530 previous size:   80  (Allocated) Proc (Protected)
[...]
*fffffa8007475ad0 size:  530 previous size:   30  (Allocated) Proc (Protected)
*fffffa80074e8490 size:  530 previous size:  100  (Allocated) Proc (Protected)
*fffffa80075ee0b0 size:  530 previous size:   b0  (Free)      Pro.
*fffffa800761d000 size:  530 previous size:    0  (Free)      Pro.
*fffffa8007645ad0 size:  530 previous size:   b0  (Allocated) Proc (Protected)

0: kd> dc fffffa8007645ad0
fffffa80`07645ad0  0253000b e36f7250 07644030 fffffa80  ..S.Pro.0.d.....
fffffa80`07645ae0  00001000 00000528 00000068 fffff800  ....(...h.......
fffffa80`07645af0  01a1a940 fffff800 00080090 00490024  @...........$.I.
fffffa80`07645b00  000000c4 00000000 00000008 00000000  ................
fffffa80`07645b10  00000000 00000000 00080007 00300033  ............3.0.
fffffa80`07645b20  01a1a940 fffff800 013cfeae fffff8a0  @.........<.....
fffffa80`07645b30  00580003 00000000 05ba19a0 fffffa80  ..X………….
fffffa80`07645b40  05ba19a0 fffffa80 07645b48 fffffa80  ……..H[d…..

0: kd> !process fffffa80`07645b30 ff
PROCESS fffffa8007645b30
SessionId: 0  Cid: 14c4    Peb: 7fffffd4000  ParentCid: 02c4
DirBase: 7233e000  ObjectTable: fffff8a0014d4220  HandleCount: 399.
Image: AppA.exe
VadRoot fffffa80072bc5b0 Vads 239 Clone 0 Private 24675. Modified 23838. Locked 0.
DeviceMap fffff8a0000088f0
Token                             fffff8a000f28060
ElapsedTime                       00:00:53.066
UserTime                          00:00:00.000
KernelTime                        00:00:00.000
QuotaPoolUsage[PagedPool]         0
QuotaPoolUsage[NonPagedPool]      0
Working Set Sizes (now,min,max)  (11960, 50, 345) (47840KB, 200KB, 1380KB)
PeakWorkingSetSize                74346
VirtualSize                       331 Mb
PeakVirtualSize                   478 Mb
PageFaultCount                    92214
MemoryPriority                    BACKGROUND
BasePriority                      8
CommitCharge                      25905

[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Here we provide examples of threads waiting for pushlocks as they are not normally seen in crash dumps:

THREAD fffffa80033b5b50  Cid 0004.0030  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable
fffff880021d9750  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a0000088f0
Owning Process            fffffa80033879e0       Image:         System
Attached Process          fffffa800439c620       Image:         AppA.exe
Wait Start TickCount      30819          Ticks: 14746574 (2:15:54:08.028)
Context Switch Count      2800
UserTime                  00:00:00.000
KernelTime                00:00:00.374
Win32 Start Address nt!ExpWorkerThread (0xfffff8000189e530)
Stack Init fffff880021d9db0 Current fffff880021d9470
Base fffff880021da000 Limit fffff880021d4000 Call 0
Priority 12 BasePriority 12 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`021d94b0 fffff800`0188aa32 nt!KiSwapContext+0×7a
fffff880`021d95f0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2
fffff880`021d9680 fffff800`018c4bf8 nt!KeWaitForSingleObject+0×19f
fffff880`021d9720 fffff800`01c2915d nt!ExfAcquirePushLockShared+0×138
fffff880`021d97a0 fffff800`01c6da31 nt!MmEnumerateAndReferenceImages+0×6d
[…]
fffff880`021d9cb0 fffff800`01b2be5a nt!ExpWorkerThread+0×111
fffff880`021d9d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a
fffff880`021d9d80 00000000`00000000 nt!KxStartSystemThread+0×16

THREAD fffffa8003c9d600  Cid 0004.00ac  Teb: 0000000000000000 Win32Thread: 0000000000000000 WAIT: (WrPushLock) KernelMode Non-Alertable
fffff880023d1b30  SynchronizationEvent
Not impersonating
DeviceMap                 fffff8a0000088f0
Owning Process            fffffa80033879e0       Image:         System
Attached Process          N/A            Image:         N/A
Wait Start TickCount      177686         Ticks: 14599707 (2:15:15:56.888)
Context Switch Count      1590
UserTime                  00:00:00.000
KernelTime                00:00:00.124
Win32 Start Address 0xfffff80001bac754
Stack Init fffff880023d1db0 Current fffff880023d1850
Base fffff880023d2000 Limit fffff880023cc000 Call 0
Priority 15 BasePriority 15 UnusualBoost 0 ForegroundBoost 0 IoPriority 2 PagePriority 5
Child-SP          RetAddr           Call Site
fffff880`023d1890 fffff800`0188aa32 nt!KiSwapContext+0×7a
fffff880`023d19d0 fffff800`0189bd8f nt!KiCommitThreadWait+0×1d2
fffff880`023d1a60 fffff800`01886183 nt!KeWaitForSingleObject+0×19f
fffff880`023d1b00 fffff800`01cd9982 nt!ExfAcquirePushLockExclusive+0×188
[…]
fffff880`023d1d40 fffff800`01885d26 nt!PspSystemThreadStartup+0×5a
fffff880`023d1d80 00000000`00000000 nt!KxStartSystemThread+0×16

Instead of explaining what a pushlock is we provide a link to ntdebugging blog article.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -