Software Diagnostics Library

Manual analysis of Execution Residue in stack regions can be quite daunting so some basic statistics on the distribution of address, value, and symbolic information can be useful. To automate this process we use Pandas python library and interpret preprocessed WinDbg output of dps and dpp commands as DataFrame object:

import pandas as pd
import pandas_profiling

df = pd.read_csv("stack.csv")
html_file = open("stack.html", "w")
html_file.write (pandas_profiling.ProfileReport(df).to_html())
html_file.close()

We get a rudimentary profile report: stack.html for stack.csv file. The same was also done for Address, Address/Value, Value, Symbol output of dpp command: stack4columns.html for stack4columns.csv file.

We call this analysis pattern Region Profile since any Memory Region can be used. This analysis pattern is not limited to Python/Pandas and different libraries/scripts/script languages can also be used for statistical and correlational profiling. We plan to provide more examples here in the future.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we have complex plugins or subsystems consisting from several modules that are loaded in the same process address space (and this is also possible in kernel space as well). Usually subsystem components reside in the same persistent folder (including its possible subfolders):

0:000> lmf
start             end                 module name
...
00007fff`46ee0000 00007fff`4cde8000 libcef C:\Program Files\Adobe\Adobe Photoshop CC 2019\Required\Plug-ins\Spaces\libcef.dll
…
00007fff`7fa40000 00007fff`7fbc2000 Spaces C:\Program Files\Adobe\Adobe Photoshop CC 2019\Required\Plug-ins\Spaces\Spaces.8li
…
00007fff`8ba50000 00007fff`8bae3000 chrome_elf C:\Program Files\Adobe\Adobe Photoshop CC 2019\Required\Plug-ins\Spaces\chrome_elf.dll
…

When we know product architecture we can group modules according to the known design not only by their physical locations.

This analysis pattern, that we call Subsystem Modules, is useful for the analysis of possibles relationships of Stack Traces from Stack Trace Collections and other Historical Information during the root cause analysis of various crash and hang issues. This can also be applicable to .NET managed space analysis that includes various multi-file assemblies.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Stack Overflow (user mode) analysis pattern relied on prominent stack trace signs. However, in cases of low memory condition there may not be an opportunity to create a new guard page for the expanding stack region and the exception itself may be Hidden Exception:

0:003> kc
# Call Site
00 KERNELBASE!GetModuleHandleExW
01 EdgeContent!wil::details::GetModuleInformation
02 EdgeContent!wil::details::GetCurrentModuleName
03 EdgeContent!wil::details::LogFailure
04 EdgeContent!wil::details::ReportFailure
05 EdgeContent!wil::details::ReportFailure_Hr
06 EdgeContent!wil::details::in1diag3::FailFast_Hr
07 EdgeContent!`anonymous namespace'::MemoryLimitWatchdogThreadProc
08 kernel32!BaseThreadInitThunk
09 ntdll!RtlUserThreadStart

0:003> .exr -1
ExceptionAddress: 00007ff992f6391b (KERNELBASE!GetModuleHandleExW+0x000000000000007b)
ExceptionCode: c00000fd (Stack overflow)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 0000000000000001
Parameter[1]: 000000fe709fdfc8

0:003> r
rax=0000000000000001 rbx=0000000000000006 rcx=00007ff945e3fa80
rdx=000000fe709fe020 rsi=00007ff945e3fa80 rdi=000000fe709fe030
rip=00007ff992f6391b rsp=000000fe709fdfd0 rbp=0000000000000040
r8=000000fe709fe030  r9=0000000000000040 r10=00000fff28bc7dba
r11=0440000001000010 r12=00007ff94606be60 r13=000000000000013f
r14=0000000000000003 r15=0000000000000000
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010200
KERNELBASE!GetModuleHandleExW+0x7b:
00007ff9`92f6391b ff1597f31200    call    qword ptr [KERNELBASE!_imp_RtlPcToFileHeader (00007ff9`93092cb8)] ds:00007ff9`93092cb8={ntdll!RtlPcToFileHeader (00007ff9`960d0f50)}

The stack trace is short and the exception seem happened during call so we examine the stack pointer:

0:003> !address 000000fe709fdfd0
Usage:                  Stack
Base Address:           000000fe`709fb000
End Address:            000000fe`70a00000
Region Size:            00000000`00005000 (  20.000 kB)
State:                  00001000          MEM_COMMIT
Protect:                00000004          PAGE_READWRITE
Type:                   00020000          MEM_PRIVATE
Allocation Base:        000000fe`70900000
Allocation Protect:     00000004          PAGE_READWRITE

The committed stack region is very small so we expect its surrounding from the output of !address WinDbg command (without any address parameters):

Opened log file 'c:\MemoryDumps\stack.txt'
0:003> !address 000000fe709fdfd0
[...]
+ fe`70600000 fe`706fb000 0`000fb000 MEM_PRIVATE MEM_RESERVE Stack [~1; 1f50.21d4]
fe`706fb000 fe`706fe000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~1; 1f50.21d4]
fe`706fe000 fe`70700000 0`00002000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~1; 1f50.21d4]
+ fe`70700000 fe`70800000 0`00100000 MEM_FREE PAGE_NOACCESS Free
+ fe`70800000 fe`708f9000 0`000f9000 MEM_PRIVATE MEM_RESERVE Stack [~2; 1f50.460c]
fe`708f9000 fe`708fc000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~2; 1f50.460c]
fe`708fc000 fe`70900000 0`00004000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~2; 1f50.460c]
+ fe`70900000 fe`709fb000 0`000fb000 MEM_PRIVATE MEM_RESERVE Stack [~3; 1f50.4f80]
fe`709fb000 fe`70a00000 0`00005000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~3; 1f50.4f80]
+ fe`70a00000 fe`70afc000 0`000fc000 MEM_PRIVATE MEM_RESERVE Stack [~4; 1f50.4ff4]
fe`70afc000 fe`70aff000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~4; 1f50.4ff4]
fe`70aff000 fe`70b00000 0`00001000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~4; 1f50.4ff4]
+ fe`70b00000 fe`70bf9000 0`000f9000 MEM_PRIVATE MEM_RESERVE Stack [~5; 1f50.5ea0]
fe`70bf9000 fe`70bfc000 0`00003000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE|PAGE_GUARD Stack [~5; 1f50.5ea0]
fe`70bfc000 fe`70c00000 0`00004000 MEM_PRIVATE MEM_COMMIT PAGE_READWRITE Stack [~5; 1f50.5ea0]
[…]

We see that there is no guard page for the stack ~3 compared to other stacks and there is plenty of reserved space before that to create a new guard page in the normal circumstances. We also recall that the stack overflow exception is about the inability to create a guard page:

0:003> !error c00000fd
Error code: (NTSTATUS) 0xc00000fd (3221225725) - A new guard page for the stack cannot be created.

Looking at Stack Trace Collection we find the thread ~11 that was doing System Call:

0:003> ~*kc
[...]

11  Id: 1f50.2424 Suspend: 1 Teb: 000000fe`6fec8000 Unfrozen
# Call Site
00 ntdll!NtAllocateVirtualMemory
01 KERNELBASE!VirtualAlloc
02 eShims!NS_ACGLockdownTelemetry::APIHook_VirtualAlloc
03 Chakra!Memory::VirtualAllocWrapper::AllocPages
[…]

For this thread the last error value show that there was low memory condition, perhaps the page file was full to create new committed pages:

0:003> ~11s
ntdll!NtAllocateVirtualMemory+0x14:
00007ff9`9615aca4 c3              ret

0:011> !teb
TEB at 000000fe6fec8000
ExceptionList:        0000000000000000
StackBase:            000000fe71e00000
StackLimit:           000000fe7140a000
SubSystemTib:         0000000000000000
FiberData:            0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self:                 000000fe6fec8000
EnvironmentPointer:   0000000000000000
ClientId:             0000000000001f50 . 0000000000002424
RpcHandle:            0000000000000000
Tls Storage:          0000023c84be2c70
PEB Address:          000000fe6fe97000
LastErrorValue:       1455
LastStatusValue:      c000012d
Count Owned Locks:    0
HardErrorMode:        0

0:011> !error c000012d
Error code: (NTSTATUS) 0xc000012d (3221225773) - {Out of Virtual Memory}  Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.

0:011> !error 0n1455
Error code: (Win32) 0x5af (1455) - The paging file is too small for this operation to complete.

Because the original analysis you see above was done on the memory dump from the working computer we are not able to provide that memory dump for security reasons. So, to model this behavior, we created an application that has two threads: one leaks committed virtual memory and when the next allocation fails it creates a separate thread that attempts to do stack overflow. We then launched 20 copies of that process to limit the size of the individual crash dump. The example memory dump and the application PDB file can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When we have Active Thread:

0:001> kL
# Child-SP RetAddr Call Site
00 (Inline Function) --------`-------- Windows_Web!JsonParser::ParseString+0x96
01 0000007f`56efdb00 00007fff`700198c5 Windows_Web!JsonParser::ParseValue+0x3b7
02 0000007f`56efdb80 00007fff`70018ab3 Windows_Web!JsonParser::ParseArray+0xf5
03 0000007f`56efdbd0 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0x663
04 0000007f`56efdc50 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0x422
05 0000007f`56efdda0 00007fff`700198c5 Windows_Web!JsonParser::ParseValue+0x2ab
06 0000007f`56efde20 00007fff`70018ab3 Windows_Web!JsonParser::ParseArray+0xf5
07 0000007f`56efde70 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0x663
08 0000007f`56efdef0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0x422
09 0000007f`56efe040 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0x2ab
0a 0000007f`56efe0c0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0x422
0b 0000007f`56efe210 00007fff`70011a69 Windows_Web!JsonParser::ParseValue+0x2ab
0c 0000007f`56efe290 00007fff`700119d2 Windows_Web!JsonValueImpl::ParseInternal+0x75
0d 0000007f`56efe2d0 00007fff`700118e8 Windows_Web!JsonValueImpl::InitializeFromString+0x16
0e 0000007f`56efe300 00007fff`70011876 Windows_Web!JsonValueFactory::TryParseInternal+0x50
0f 0000007f`56efe330 00007fff`3a6fe48e Windows_Web!JsonValueFactory::TryParse+0x36
10 0000007f`56efe360 00007fff`3a78f95c eModel!JsonUtils::GetJsonObject+0x82
[...]

which is also Spiking Thread:

0:001> !runaway f
User Mode Time
Thread Time
1:1a2c 0 days 0:03:37.562
8:52a8 0 days 0:00:36.890
[…]

Kernel Mode Time
Thread Time
8:52a8 0 days 0:00:10.625
1:1a2c 0 days 0:00:18.078
[...]

we can look at the relative proportion of User Mode / Kernel Mode times to infer Active Space. This can be further confirmed if we have Step Dumps, for example, in our case we had an earlier memory dump with the same Active Thread and Constant Subtrace (with Variable Subtraces in the same Active Space):

0:001> kL
# Child-SP RetAddr Call Site
00 0000007f`56efd8d0 00007fff`a1109453 ntdll!RtlpHpLfhSlotAllocate+0×126
01 0000007f`56efd9c0 00007fff`a110a8db ntdll!RtlpAllocateHeapInternal+0×313
02 0000007f`56efdaa0 00007fff`9ea09960 ntdll!RtlpHpAllocWithExceptionProtection+0×3b
03 0000007f`56efdb70 00007fff`7001ba1b msvcrt!malloc+0×70
04 0000007f`56efdba0 00007fff`700187ca Windows_Web!operator new+0×23
05 (Inline Function) ——–`——– Windows_Web!StringBuilder::EnsureBufferSpace+0×18
06 (Inline Function) ——–`——– Windows_Web!JsonParser::ParseString+0×59
07 0000007f`56efdbd0 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0×37a
08 0000007f`56efdc50 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0×422
09 0000007f`56efdda0 00007fff`700198c5 Windows_Web!JsonParser::ParseValue+0×2ab
0a 0000007f`56efde20 00007fff`70018ab3 Windows_Web!JsonParser::ParseArray+0xf5
0b 0000007f`56efde70 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0×663
0c 0000007f`56efdef0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0×422
0d 0000007f`56efe040 00007fff`70018f72 Windows_Web!JsonParser::ParseValue+0×2ab
0e 0000007f`56efe0c0 00007fff`700186fb Windows_Web!JsonParser::ParseObject+0×422
0f 0000007f`56efe210 00007fff`70011a69 Windows_Web!JsonParser::ParseValue+0×2ab
10 0000007f`56efe290 00007fff`700119d2 Windows_Web!JsonValueImpl::ParseInternal+0×75
11 0000007f`56efe2d0 00007fff`700118e8 Windows_Web!JsonValueImpl::InitializeFromString+0×16
12 0000007f`56efe300 00007fff`70011876 Windows_Web!JsonValueFactory::TryParseInternal+0×50
13 0000007f`56efe330 00007fff`3a6fe48e Windows_Web!JsonValueFactory::TryParse+0×36
14 0000007f`56efe360 00007fff`3a78f95c eModel!JsonUtils::GetJsonObject+0×82
[…]

Even in the absence of Active Threads in process memory dumps it is useful to look at time spent in kernel and user modes to see any anomalies, especially when threads are blocked in System Calls. Then the high value of kernel mode time should suggest to inspect kernel Activity Space (for example, via a kernel dump). If we have an indication of high values of user mode time in a kernel memory dump we may need to obtain process memory Step Dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

In addition to Exception Thread analysis pattern (or several threads when we have Multiple Exceptions) that we named Exception Stack Trace there may be a separated thread that reports such exceptions (Exception Reporting Thread):

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(4a0c.2ab4): Application hang - code cfffffff (first/second chance not available)
win32u!NtUserCreateWindowEx+0x14:

0:002> kc
# Call Site
00 win32u!NtUserCreateWindowEx
01 user32!VerNtUserCreateWindowEx
02 user32!CreateWindowInternal
03 user32!CreateWindowExW
04 CoreMessaging!Microsoft::CoreUI::Dispatch::UserAdapter:: InitializeWindow
05 CoreMessaging!Microsoft::CoreUI::Dispatch::UserAdapter::Initialize
06 CoreMessaging!Microsoft::CoreUI::Dispatch::EventLoop::Callback_Run
07 CoreMessaging!Microsoft::CoreUI::Messaging::MessageSession$R:: Microsoft__CoreUI__IExportMessageSession_Impl::Run
08 CoreMessaging!Microsoft::CoreUI:: IExportMessageSession$X__ExportAdapter::Run
09 CoreMessaging!Windows::System::DispatcherQueue::RunLoop
0a CoreMessaging!Windows::System::DispatcherQueueController:: DispatcherQueueThreadProc
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

0:002> ~42kc
# Call Site
00 ntdll!NtWaitForMultipleObjects
01 ntdll!WerpWaitForCrashReporting
02 ntdll!RtlReportExceptionEx
03 eModel!s_ReportAsyncBoundaryHangOnSuspendedThread
04 kernel32!BaseThreadInitThunk
05 ntdll!RtlUserThreadStart

We can spot such a thread in Stack Trace Collection.

Although the example above is for application hang (Blocked Thread) and also illustrates Self-Diagnosis, similar threads may report Handled Exceptions. Usually, Exception Threads are also Exception Reporting Threads as shown in Invalid Parameter analysis pattern example.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Insufficient stack memory may not result in Stack Overflow if there are internal thread checks or external watchdog threads. In some way this is a specialization of a more general Insufficient Memory (region) analysis pattern.

Consider this thread stack trace we see when we open a crash dump in WinDbg:

0:003> kc
# Call Site
00 EdgeContent!wil::details::ReportFailure
01 EdgeContent!wil::details::ReportFailure_Hr
02 EdgeContent!wil::details::in1diag3::FailFast_Hr
03 EdgeContent!`anonymous namespace'::MemoryLimitWatchdogThreadProc
04 kernel32!BaseThreadInitThunk
05 ntdll!RtlUserThreadStart

The reported error is related to general security checks and doesn’t reveal much:

0:003> .lastevent
Last event: 10e0.25fc: Security check failure or stack buffer overrun - code c0000409 (first/second chance not available)
debugger time: Sat Sep 29 21:30:31.531 2018 (UTC + 1:00)

However, from Stack Trace Motif we may infer that the failure was related to some memory limit.

When looking at Stack Trace Collection we notice the thread #11 which has the long sequence of frames:

11 Id: 10e0.d68 Suspend: 2 Teb: 000000ca`e96e3000 Unfrozen
# Call Site
00 ntdll!NtWaitForSingleObject
01 KERNELBASE!WaitForSingleObjectEx
02 Chakra!Memory::Recycler::WaitForConcurrentThread
03 Chakra!Memory::Recycler::FinishConcurrentCollect
04 Chakra!ThreadContext::ExecuteRecyclerCollectionFunction
05 Chakra!Memory::Recycler::FinishConcurrentCollectWrapped
06 Chakra!Memory::Recycler::LargeAlloc<0>
07 Chakra!Memory::Recycler::AllocLeaf
08 Chakra!Js::CompoundString::GetSz
09 Chakra!Js::JSONStringifier::ReadProperty
0a Chakra!Js::JSONStringifier::ReadObject
0b Chakra!Js::JSONStringifier::ReadProperty
0c Chakra!Js::JSONStringifier::Stringify
0d Chakra!JSON::Stringify
0e Chakra!amd64_CallFunction
0f Chakra!Js::InterpreterStackFrame::OP_CallCommon<Js::OpLayoutDynamicProfile<
Js::OpLayoutT_CallIWithICIndex<Js::LayoutSizePolicy<0> > > >
10 Chakra!Js::InterpreterStackFrame::ProcessUnprofiled
11 Chakra!Js::InterpreterStackFrame::Process
12 Chakra!Js::InterpreterStackFrame::InterpreterHelper
13 Chakra!Js::InterpreterStackFrame::InterpreterThunk
[...]
fa Chakra!ThreadContext::ExecuteImplicitCall<<
lambda_5a46706206cf607f01fe0fb33b6e8acf> >
fb Chakra!Js::DynamicObject::CallToPrimitiveFunction
fc Chakra!Js::DynamicObject::ToPrimitiveImpl<409>
fd Chakra!Js::DynamicObject::ToPrimitive
fe Chakra!Js::JavascriptConversion::OrdinaryToPrimitive
ff Chakra!Js::JavascriptConversion::MethodCallToPrimitive

It looks like the thread stack is really long that we suspect the possibility of overflow:

0:003> ~11kc 0xffff
[...]
c473 Chakra!amd64_CallFunction
c474 Chakra!Js::JavascriptFunction::CallFunction<1>
c475 Chakra!Js::JavascriptFunction::CallRootFunctionInternal
c476 Chakra!Js::JavascriptFunction::CallRootFunction
c477 Chakra!ScriptSite::CallRootFunction
c478 Chakra!ScriptSite::Execute
c479 Chakra!ScriptEngineBase::Execute
c47a edgehtml!CMutationObserver::PerformMicrotaskCheckpoint
c47b edgehtml!CObserverManager::InvokeObserversForCheckpoint
c47c edgehtml!CObserverManager::MicroTaskExecutionCallback
c47d edgehtml!HTML5TaskScheduler::RunReadiedTask
c47e edgehtml!HTML5TaskScheduler::PerformMicrotaskCheckpoint
c47f edgehtml!CJScript9Holder::ExecuteCallback
c480 edgehtml!CScriptTimer::ExecuteTimer
c481 edgehtml!CScriptTimerManager::OnTick
c482 edgehtml!CPaintAlignedTimerManager<CPaintBeat>::ProcessTimers
c483 edgehtml!CPaintBeat::OnProcessTimersTask
c484 edgehtml!GWndAsyncTask::Run
c485 edgehtml!HTML5TaskScheduler::RunReadiedTask
c486 edgehtml!TaskSchedulerBase::RunReadiedTasksInTaskQueueWithCallback
c487 edgehtml!HTML5TaskScheduler::RunReadiedTasks
c488 edgehtml!HTML5EventLoopDriver::DriveLowPriorityTaskExecution
c489 edgehtml!GlobalWndOnPaintPriorityMethodCall
c48a edgehtml!GlobalWndProc
c48b user32!FilteredProcessRedirectingWndProcW
c48c user32!UserCallWinProcCheckWow
c48d user32!DispatchClientMessage
c48e user32!_fnDWORD
c48f ntdll!KiUserCallbackDispatcherContinue
c490 win32u!NtUserDispatchMessage
c491 user32!DispatchMessageWorker
c492 EdgeContent!CBrowserTab::_TabWindowThreadProc
c493 EdgeContent!LCIETab_ThreadProc
c494 edgeIso!_IsoThreadProc_WrapperToReleaseScope
c495 kernel32!BaseThreadInitThunk
c496 ntdll!RtlUserThreadStart

However, the current stack pointer is the way above the stack region limit:

0:003> ~11s
ntdll!NtWaitForSingleObject+0x14:
00007ffa`b0d99f74 c3              ret

0:011> r rsp
rsp=000000caeb033348

0:011> !teb
TEB at 000000cae96e3000
ExceptionList:        0000000000000000
StackBase:            000000caeba00000
StackLimit:           000000caeb00b000
SubSystemTib:         0000000000000000
FiberData:            0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self:                 000000cae96e3000
EnvironmentPointer:   0000000000000000
ClientId:             00000000000010e0 . 0000000000000d68
RpcHandle:            0000000000000000
Tls Storage:          000001e915185590
PEB Address:          000000cae96a8000
LastErrorValue:       1455
LastStatusValue:      c000012d
Count Owned Locks:    0
HardErrorMode:        0

The last error code shows possible Insufficient Memory (committed memory) pattern but there were enough space in the page file and the same crashes were observed in a much more spacious memory environments:

0:011> !error c000012d
Error code: (NTSTATUS) 0xc000012d (3221225773) - {Out of Virtual Memory}  Your system is low on virtual memory. To ensure that Windows runs properly, increase the size of your virtual memory paging file. For more information, see Help.

0:011> !error 0n1455
Error code: (Win32) 0x5af (1455) - The paging file is too small for this operation to complete.

When we look at a raw stack trace region though, we see it was used to the fullest:

0:011> dps 000000caeb00b000 L30
000000ca`eb00b000 00000000`00000000
000000ca`eb00b008 00000000`00000000
000000ca`eb00b010 00000000`00000000
000000ca`eb00b018 00000000`00000000
000000ca`eb00b020 00000000`00000000
000000ca`eb00b028 00000000`00000000
000000ca`eb00b030 00000000`00000000
000000ca`eb00b038 00000000`00000000
000000ca`eb00b040 00000000`00000000
000000ca`eb00b048 00000000`00000000
000000ca`eb00b050 00000000`00000000
000000ca`eb00b058 00000000`00000000
000000ca`eb00b060 00000000`00000000
000000ca`eb00b068 00000000`00000000
000000ca`eb00b070 00000000`00000000
000000ca`eb00b078 00000000`00000000
000000ca`eb00b080 00000000`00000000
000000ca`eb00b088 00000000`00000000
000000ca`eb00b090 00000000`00000000
000000ca`eb00b098 000001e9`258805c0
000000ca`eb00b0a0 00000000`00000000
000000ca`eb00b0a8 00000000`00000000
000000ca`eb00b0b0 00000000`00000131
000000ca`eb00b0b8 00007ffa`8de137a9 Chakra!Js::DictionaryTypeHandlerBase<unsigned>::GetProperty+0×119
000000ca`eb00b0c0 00000000`00000006
000000ca`eb00b0c8 000001e9`258805c0
000000ca`eb00b0d0 000001e9`258805c0
000000ca`eb00b0d8 000001e9`283d4e70
000000ca`eb00b0e0 000000ca`eb00b210
000000ca`eb00b0e8 00000000`00000000
000000ca`eb00b0f0 00000000`00000131
000000ca`eb00b0f8 000001f1`71863c70
000000ca`eb00b100 00007ffa`8e3cd468 Chakra!Js::BuiltInPropertyRecords::number
000000ca`eb00b108 00000000`00000000
000000ca`eb00b110 000001f1`71863c70
000000ca`eb00b118 000000ca`eb00b210
000000ca`eb00b120 000001e9`258805c0
000000ca`eb00b128 00007ffa`8de103d3 Chakra!Js::DynamicObject::GetPropertyQuery+0×53
000000ca`eb00b130 000001e9`24cc65b0
000000ca`eb00b138 000001e9`258805c0
000000ca`eb00b140 000001e9`258805c0
000000ca`eb00b148 00000000`00000131
000000ca`eb00b150 000000ca`eb00b210
000000ca`eb00b158 00000000`00000000
000000ca`eb00b160 000001f1`71863c70
000000ca`eb00b168 00000000`00000000
000000ca`eb00b170 00000000`00000000
000000ca`eb00b178 00007ffa`8df79864 Chakra!Js::JavascriptOperators::GetProperty_InternalSimple+0×64

We also see Execution Residue possibly indicating catch exception processing:

0:011> dpS 000000caeb00b000 L100
00007ffa`8de137a9 Chakra!Js::DictionaryTypeHandlerBase<unsigned short>::GetProperty+0x119
00007ffa`8e3cd468 Chakra!Js::BuiltInPropertyRecords::number
00007ffa`8de103d3 Chakra!Js::DynamicObject::GetPropertyQuery+0x53
00007ffa`8df79864 Chakra!Js::JavascriptOperators::GetProperty_InternalSimple+0x64
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b048a5da msvcrt!getptd_noexit+0x6e
00007ffa`8dfb36d6 Chakra!`Js::JavascriptExceptionOperators::StackTraceAccessor'::`1'::catch$8
+0x50
00007ffa`b048a553 msvcrt!getptd+0xb
00007ffa`b045aad6 msvcrt!_DestructExceptionObject+0x46
00007ffa`b0463798 msvcrt!_CxxCallCatchBlock+0x1e8
00007ffa`8ddb6d54 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor+0x2a4
00007ffa`8e324798 Chakra!ValueType::PrimitiveOrObject+0xa80
00007ffa`8dfb3686 Chakra!`Js::JavascriptExceptionOperators::StackTraceAccessor'::`1'::catch$8
00007ffa`8dd20000 Chakra!_jscriptinfo_IID_Lookup <PERF> (Chakra+0x0)
00007ffa`b04635b0 msvcrt!_CxxCallCatchBlock
00007ffa`b0d9e2b3 ntdll!RcConsolidateFrames+0x3
00007ffa`8ddb6d54 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor+0x2a4
00007ffa`8de137a9 Chakra!Js::DictionaryTypeHandlerBase<unsigned short>::GetProperty+0x119
00007ffa`8e3cd468 Chakra!Js::BuiltInPropertyRecords::number
00007ffa`8de103d3 Chakra!Js::DynamicObject::GetPropertyQuery+0x53
00007ffa`8df79864 Chakra!Js::JavascriptOperators::GetProperty_InternalSimple+0x64
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b0d470d0 ntdll!RtlSetLastWin32Error+0x40
00007ffa`b048a5da msvcrt!getptd_noexit+0x6e
00007ffa`8dfb36d6 Chakra!`Js::JavascriptExceptionOperators::StackTraceAccessor'::`1'::catch$8
+0x50
00007ffa`b048a553 msvcrt!getptd+0xb
00007ffa`b045aad6 msvcrt!_DestructExceptionObject+0x46
00007ffa`b0463798 msvcrt!_CxxCallCatchBlock+0x1e8
00007ffa`8ddb6d54 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor+0x2a4

Another similar crash on a different machine with much more memory available had Active Thread Exception Stack Trace:

0:013> kc
# Call Site
00 ntdll!NtQueryWnfStateNameInformation
01 ntdll!SignalStartWerSvc
02 ntdll!SendMessageToWERService
03 ntdll!ReportExceptionInternal
04 kernel32!WerpReportFaultInternal
05 kernel32!WerpReportFault
06 KERNELBASE!UnhandledExceptionFilter
07 Chakra!FatalExceptionFilter
08 Chakra!ReportFatalException$filt$0
09 msvcrt!_C_specific_handler
0a ntdll!RtlpExecuteHandlerForException
0b ntdll!RtlDispatchException
0c ntdll!RtlRaiseException
0d KERNELBASE!RaiseException
0e Chakra!ReportFatalException
0f Chakra!OutOfMemory_fatal_error
10 Chakra!Js::Exception::RaiseIfScriptActive
11 Chakra!Js::Throw::OutOfMemory
12 Chakra!Memory::Recycler::LargeAlloc<0>
13 Chakra!Memory::Recycler::AllocLeaf
14 Chakra!Js::LazyJSONString::GetSz
15 Chakra!Js::CompoundString::AppendGeneric<Js::CompoundString>
16 Chakra!Js::JavascriptExceptionOperators::StackTraceAccessor
17 Chakra!Js::InlineCache::TryGetProperty<1,1,1,1,0>
18 Chakra!Js::CacheOperators::TryGetProperty<1,1,1,1,1,1,1,0,0>
19 Chakra!Js::JavascriptOperators::PatchGetValue<1,Js::InlineCache>
1a js!d.toString
[...]

which actually showed the Javascript URL:

0:013> .frame 1a
1a 00000037`4671e5e0 00007ffe`1753f1a1 js!d.toString [https://yastatic.net/awaps-ad-sdk-js-bundles/1.0-889/bundles/inpage.bundle.js @ 8,36454]

The stack region was also almost full with execution residue including exception catch processing. But this time it show “stack space” hint:

0:013> !teb
TEB at 00000037456dc000
ExceptionList:        0000000000000000
StackBase:            0000003747100000
StackLimit:           000000374670a000
SubSystemTib:         0000000000000000
FiberData:            0000000000001e00
ArbitraryUserPointer: 0000000000000000
Self:                 00000037456dc000
EnvironmentPointer:   0000000000000000
ClientId:             0000000000005914 . 0000000000001150
RpcHandle:            0000000000000000
Tls Storage:          000001a5bcfd88d0
PEB Address:          00000037456bd000
LastErrorValue:       0
LastStatusValue:      c000012d
Count Owned Locks:    0
HardErrorMode:        0

0:013> s-su 000000374670a000 00000037`46720000
[…]
00000037`4670d210 “tack space”
[…]

Note: Regardless of the actual root cause and mechanism for these crash dumps, such stack region monitoring can also be used for software prognostics purposes.

The example dump can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Backwards disassembling used in memory analysis patterns such as Coincidental Symbolic Information may be ambiguous and can show Wild Code output. This may also be debugger disassembling algorithm dependent. For example, default 8-instruction backwards disassembly shows this code:

0:011> ub 00007ff8`cdc9b4bf
00007ff8`cdc9b4ab 855948 test dword ptr [rcx+48h],ebx
00007ff8`cdc9b4ae b988bf03a8 mov ecx,0A803BF88h
00007ff8`cdc9b4b3 f4 hlt
00007ff8`cdc9b4b4 0100 add dword ptr [rax],eax
00007ff8`cdc9b4b6 00488b add byte ptr [rax-75h],cl
00007ff8`cdc9b4b9 09e8 or eax,ebp
00007ff8`cdc9b4bb 117236 adc dword ptr [rdx+36h],esi
00007ff8`cdc9b4be 5f pop rdi

However, if we specify the number of instructions to disassemble except 7 and 8 we get a different result (which is more correct from the forward code execution view since we disassembled the saved return address from the stack region):

0:011> ub 00007ff8`cdc9b4bf L1
00007ff8`cdc9b4ba e81172365f call clr!JIT_MonEnter (00007ff9`2d0026d0)

0:011> ub 00007ff8`cdc9b4bf L2
00007ff8`cdc9b4b7 488b09 mov rcx,qword ptr [rcx]
00007ff8`cdc9b4ba e81172365f call clr!JIT_MonEnter (00007ff9`2d0026d0)

0:011> k L10
# Child-SP RetAddr Call Site
00 0000002a`fc23e308 00007ff9`53d06099 ntdll!NtWaitForMultipleObjects+0x14
01 0000002a`fc23e310 00007ff9`2d1a96be KERNELBASE!WaitForMultipleObjectsEx+0xf9
02 0000002a`fc23e610 00007ff9`2d1a951c clr!WaitForMultipleObjectsEx_SO_TOLERANT+0x62
03 0000002a`fc23e670 00007ff9`2d1a9315 clr!Thread::DoAppropriateWaitWorker+0x1e4
04 0000002a`fc23e770 00007ff9`2d0c2b7f clr!Thread::DoAppropriateWait+0x7d
05 0000002a`fc23e7f0 00007ff9`2d1aa491 clr!CLREventBase::WaitEx+0xc4
06 0000002a`fc23e880 00007ff9`2d1aa39e clr!AwareLock::EnterEpilogHelper+0xc2
07 0000002a`fc23e940 00007ff9`2d1c1a92 clr!AwareLock::EnterEpilog+0x62
08 0000002a`fc23e9a0 00007ff8`cdc9b4bf clr!JITutil_MonEnterWorker+0xe2
09 0000002a`fc23eb40 00007ff9`275231d3 0×00007ff8`cdc9b4bf
0a 0000002a`fc23eb80 00007ff9`27523064 mscorlib_ni+0×5031d3
0b 0000002a`fc23ec50 00007ff9`27523032 mscorlib_ni+0×503064
0c 0000002a`fc23ec80 00007ff9`2751c812 mscorlib_ni+0×503032
0d 0000002a`fc23ecd0 00007ff9`2d006bb3 mscorlib_ni+0×4fc812
0e 0000002a`fc23ed10 00007ff9`2d006a70 clr!CallDescrWorkerInternal+0×83
0f 0000002a`fc23ed50 00007ff9`2d00735d clr!CallDescrWorkerWithHandler+0×4e

We call this analysis pattern Disassembly Ambiguity. The example dump can be downloaded from here.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

A virtual memory may contain regions that are memories of some other processes or systems. We do not consider the ordinary case of memory-mapped regions here but the case of type 2 hypervisor. In such a case, memory regions may be “physical memories” of Virtualized Systems. For example, we discovered such a region in crashed vmware-vmx.exe process memory dump:

0:007> !address -summary

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 231 7ffe`d009b000 ( 127.995 TB) 100.00%
<unknown> 518 1`2508e000 ( 4.579 GB) 96.41% 0.00%
Image 547 0`07056000 ( 112.336 MB) 2.31% 0.00%
Heap 73 0`0216a000 ( 33.414 MB) 0.69% 0.00%
Stack 81 0`01b00000 ( 27.000 MB) 0.56% 0.00%
Other 11 0`001d0000 ( 1.813 MB) 0.04% 0.00%
TEB 27 0`00036000 ( 216.000 kB) 0.00% 0.00%
PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_MAPPED 88 1`0e25a000 ( 4.221 GB) 88.88% 0.00%
MEM_PRIVATE 623 0`1aca5000 ( 428.645 MB) 8.81% 0.00%
MEM_IMAGE 547 0`07056000 ( 112.336 MB) 2.31% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 231 7ffe`d009b000 ( 127.995 TB) 100.00%
MEM_COMMIT 1185 1`27657000 ( 4.616 GB) 97.18% 0.00%
MEM_RESERVE 73 0`088fe000 ( 136.992 MB) 2.82% 0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_READWRITE 473 1`1f38b000 ( 4.488 GB) 94.49% 0.00%
PAGE_READONLY 400 0`04a05000 ( 74.020 MB) 1.52% 0.00%
PAGE_EXECUTE_READ 196 0`0367a000 ( 54.477 MB) 1.12% 0.00%
PAGE_WRITECOPY 59 0`001de000 ( 1.867 MB) 0.04% 0.00%
PAGE_READWRITE|PAGE_GUARD 27 0`00051000 ( 324.000 kB) 0.01% 0.00%
PAGE_NOACCESS 27 0`0001b000 ( 108.000 kB) 0.00% 0.00%
PAGE_EXECUTE_READWRITE 3 0`00003000 ( 12.000 kB) 0.00% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 290`ffe50000 7d66`9b210000 ( 125.401 TB)
<unknown> 28f`f8f90000 1`00000000 ( 4.000 GB)
Image 7ffa`9969f000 0`00e47000 ( 14.277 MB)
Heap 28f`95c7b000 0`00ae4000 ( 10.891 MB)
Stack b8`f7b00000 0`000fc000 (1008.000 kB)
Other 28f`f2050000 0`00181000 ( 1.504 MB)
TEB b8`f7147000 0`00002000 ( 8.000 kB)
PEB b8`f7146000 0`00001000 ( 4.000 kB)

The size of the region is 4 GB which coincides with the size of Windows VM:

We assume that the whole VM physical space was placed there and we had an instance of a physical memory dump inside a process memory dump. Whatever is such a physical memory dump internal organization, most likely the pages correspond to 4 Kb memory chunks inside. We can employ WinDbg commands that allow the address parameter. For example, we can look for Hidden Modules:

0:007> .imgscan /r 28f`f8f90000 L?1`00000000
[…]
MZ at 00000290`f5867000 - size 7f000
Name: HAL.dll
[…]
MZ at 00000290`a089b000 - size 3000
Name: TDI.SYS
[…]

0:007> !dh 00000290`a089b000

File Type: DLL
FILE HEADER VALUES
14C machine (i386)
2 number of sections
592AD310 time date stamp Sun May 28 06:39:28 2017

0 file pointer to symbol table
0 number of symbols
E0 size of optional header
2122 characteristics
Executable
App can handle >2gb addresses
32 bit word machine
DLL

OPTIONAL HEADER VALUES
10B magic #
9.00 linker version
A00 size of code
400 size of initialized data
0 size of uninitialized data
0 address of entry point
1000 base of code
----- new -----
ffffffff8a7d0000 image base
1000 section alignment
200 file alignment
3 subsystem (Windows CUI)
10.00 operating system version
10.00 image version
5.01 subsystem version
3000 size of image
400 size of headers
10F33 checksum
0000000000040000 size of stack reserve
0000000000001000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit
540 DLL characteristics
Dynamic base
NX compatible
No structured exception handler
1140 [ 73A] address [size] of Export Directory
0 [ 0] address [size] of Import Directory
2000 [ 3E8] address [size] of Resource Directory
0 [ 0] address [size] of Exception Directory
0 [ 0] address [size] of Security Directory
0 [ 0] address [size] of Base Relocation Directory
1000 [ 1C] address [size] of Debug Directory
0 [ 0] address [size] of Description Directory
0 [ 0] address [size] of Special Directory
0 [ 0] address [size] of Thread Storage Directory
0 [ 0] address [size] of Load Configuration Directory
0 [ 0] address [size] of Bound Import Directory
0 [ 0] address [size] of Import Address Table Directory
0 [ 0] address [size] of Delay Import Directory
0 [ 0] address [size] of COR20 Header Directory
0 [ 0] address [size] of Reserved Directory

SECTION HEADER #1
.text name
87A virtual size
1000 virtual address
A00 size of raw data
400 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
60000020 flags
Code
(no align specified)
Execute Read

Debug Directories(1)
Type Size Address Pointer
cv 20 101c 41c Format: RSDS, guid, 1, tdi.pdb

SECTION HEADER #2
.rsrc name
3E8 virtual size
2000 virtual address
400 size of raw data
E00 file pointer to raw data
0 file pointer to relocation table
0 file pointer to line numbers
0 number of relocations
0 number of line numbers
40000040 flags
Initialized Data
(no align specified)
Read Only
[...]

We call such pattern Hyperdump:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Sometimes we can diagnose potential data races by looking at Stack Trace Collection (from all processes and threads or from CPUs). For example, if we have one Exception Stack Trace and a very similar stack trace, for example, with Constant Subtrace and the same Stack Trace Motif, it may point to such a situation with potential diagnosis of related components. Here’s an example from a kernel memory dump where it was suggested as a workaround to close ProcessA before hibernating a system (the problem repeatedly happened on wake up):

0: kd> !analyze -v

[...]

KERNEL_SECURITY_CHECK_FAILURE (139)
A kernel component has corrupted a critical data structure. The corruption
could potentially allow a malicious user to gain control of this machine.
Arguments:
Arg1: 0000000000000003, A LIST_ENTRY has been corrupted (i.e. double remove).
Arg2: ffff8900ea8fae40, Address of the trap frame for the exception that caused the bugcheck
Arg3: ffff8900ea8fad98, Address of the exception record for the exception that caused the bugcheck
Arg4: 0000000000000000, Reserved

[...]

EXCEPTION_RECORD: ffff8900ea8fad98 -- (.exr 0xffff8900ea8fad98)
ExceptionAddress: fffff8034a9e2e93 (nt!CmpFreeKeyControlBlock+0x0000000000106a73)
ExceptionCode: c0000409 (Security check failure or stack buffer overrun)
ExceptionFlags: 00000001
NumberParameters: 1
Parameter[0]: 0000000000000003
Subcode: 0x3 FAST_FAIL_CORRUPT_LIST_ENTRY

[...]

0: kd> k
# Child-SP RetAddr Call Site
00 ffff8900`ea8fab18 fffff803`4a576ba9 nt!KeBugCheckEx
01 ffff8900`ea8fab20 fffff803`4a576f50 nt!KiBugCheckDispatch+0x69
02 ffff8900`ea8fac60 fffff803`4a575286 nt!KiFastFailDispatch+0xd0
03 ffff8900`ea8fae40 fffff803`4a9e2e93 nt!KiRaiseSecurityCheckFailure+0x3c6
04 ffff8900`ea8fafd0 fffff803`4a8c444b nt!CmpFreeKeyControlBlock+0×106a73
05 ffff8900`ea8fb000 fffff803`4a8e1392 nt!CmpDoParseKey+0×2adb
06 ffff8900`ea8fb3d0 fffff803`4a8bdd61 nt!CmpParseKey+0×302
07 ffff8900`ea8fb570 fffff803`4a8d3a2d nt!ObpLookupObjectName+0xb71
08 ffff8900`ea8fb740 fffff803`4a8d370d nt!ObOpenObjectByNameEx+0×1dd
09 ffff8900`ea8fb880 fffff803`4a8cfc5f nt!CmOpenKey+0×29d
0a ffff8900`ea8fba40 fffff803`4a576683 nt!NtOpenKeyEx+0xf
0b ffff8900`ea8fba80 00007ffc`ac3b82b4 nt!KiSystemServiceCopyEnd+0×13
0c 00000001`2c0feb28 00000000`00000000 0×00007ffc`ac3b82b4

0: kd> !stacks 2 nt!Cmp

[...]

[ffffaa0a35ccb2c0 winlogon.exe]
13dc.002668 ffffaa0a37e02080 fff4e8f4 RUNNING nt!KeBugCheckEx
nt!KiBugCheckDispatch+0×69
nt!KiFastFailDispatch+0xd0
nt!KiRaiseSecurityCheckFailure+0×3c6
nt!CmpFreeKeyControlBlock+0×106a73
nt!CmpDoParseKey+0×2adb
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf
nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

[ffffaa0a35bda780 ProcessA.exe]
1794.0017e4 ffffaa0a35c06080 fff4e8f4 RUNNING nt!CmpLockKcbStackShared+0×2
nt!CmpWalkOneLevel+0×93
nt!CmpDoParseKey+0×1c36
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf
nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

[ffffaa0a35bb8780 svchost.exe]
1754.001fec ffffaa0a3636a7c0 fff4e8f4 READY nt!KxDispatchInterrupt+0x122
nt!KiDpcInterrupt+0x3a6
nt!KeAbPreAcquire+0xd7
nt!ExfAcquirePushLockExclusiveEx+0x10b
nt!CmpDoParseKey+0×294e
nt!CmpParseKey+0×302
nt!ObpLookupObjectName+0xb71
nt!ObOpenObjectByNameEx+0×1dd
nt!CmOpenKey+0×29d
nt!NtOpenKeyEx+0xf
nt!KiSystemServiceCopyEnd+0×13
+0×7ffcac3b82b4

[...]

We call this pattern Stack Trace Race.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is a variant of Data Correlation (function parameters) analysis pattern where we look at correlations across memory structures. Simple arithmetical ratios may link such structures and correlate corresponding behavioral processes. Here we look at a recent instance of calc.exe consuming a lot of CPU. Upon the discovery of that process we were curious and saved its full process memory dump via Task Manager. In the dump we discovered 4 Spiking Threads:

0:000> !runaway f
User Mode Time
Thread       Time
13:1b68      0 days 1:51:39.906
10:23a8      0 days 1:51:37.796
11:1b98      0 days 0:00:09.890
14:88c       0 days 0:00:09.828
1:2eb4      0 days 0:00:00.390
18:2a44      0 days 0:00:00.015
19:28f0      0 days 0:00:00.000
17:22c0      0 days 0:00:00.000
16:232c      0 days 0:00:00.000
15:2008      0 days 0:00:00.000
12:2880      0 days 0:00:00.000
9:2f38      0 days 0:00:00.000
8:1a98      0 days 0:00:00.000
7:1dcc      0 days 0:00:00.000
6:c58       0 days 0:00:00.000
5:1550      0 days 0:00:00.000
4:2938      0 days 0:00:00.000
3:2b64      0 days 0:00:00.000
2:2f90      0 days 0:00:00.000
0:dc4       0 days 0:00:00.000
[…]

We see that 10/13 approx. equals 1 and #11/#14 too, or #10/#11 approx. equals #13/#14 in user mode CPU consumption. If we look at kernel times we see the same ratios:

[...]
Kernel Mode Time
Thread Time
10:23a8 0 days 0:10:36.718
13:1b68 0 days 0:10:32.968
14:88c 0 days 0:00:23.859
11:1b98 0 days 0:00:23.812
1:2eb4 0 days 0:00:00.218
2:2f90 0 days 0:00:00.015
0:dc4 0 days 0:00:00.015
19:28f0 0 days 0:00:00.000
18:2a44 0 days 0:00:00.000
17:22c0 0 days 0:00:00.000
16:232c 0 days 0:00:00.000
15:2008 0 days 0:00:00.000
12:2880 0 days 0:00:00.000
9:2f38 0 days 0:00:00.000
8:1a98 0 days 0:00:00.000
7:1dcc 0 days 0:00:00.000
6:c58 0 days 0:00:00.000
5:1550 0 days 0:00:00.000
4:2938 0 days 0:00:00.000
3:2b64 0 days 0:00:00.000
[…]

Elapsed times are also correlated and we see that correlated threads were created in pairs {#10, #11} and {#13, #14}:

[...]
Elapsed Time
Thread Time
0:dc4 0 days 18:20:55.778
1:2eb4 0 days 18:20:55.731
2:2f90 0 days 18:20:55.725
3:2b64 0 days 18:20:55.721
4:2938 0 days 18:20:55.715
5:1550 0 days 18:20:55.582
6:c58 0 days 18:20:55.522
7:1dcc 0 days 18:20:55.522
8:1a98 0 days 18:20:55.522
9:2f38 0 days 18:20:55.522
10:23a8 0 days 16:12:52.330
11:1b98 0 days 16:12:52.329
12:2880 0 days 16:12:52.195
13:1b68 0 days 16:11:44.822
14:88c 0 days 16:11:44.821
15:2008 0 days 16:11:44.693
16:232c 0 days 2:09:35.021
17:22c0 0 days 2:05:13.038
18:2a44 0 days 0:23:38.000
19:28f0 0 days 0:00:24.261

This suggests that the threads are related. We call such analysis pattern variant Data Correlation (CPU times). It may also help in finding weak Coupled Processes.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Usually threads are Blocked waiting for synchronization objects or Active running threads. There is a different category of threads that appear blocked in user space but in fact may be doing a lot of invisible work (not visible from the dump type) in kernel space. We call such an analysis pattern System Call. Compare these thread Stack Traces from Stack Trace Collection:

# Call Site
00 ntdll!NtWaitForMultipleObjects
01 KERNELBASE!WaitForMultipleObjectsEx
02 user32!MsgWaitForMultipleObjectsEx
03 combase!ASTAWaitContext::KernelWait
04 combase!ASTAWaitContext::Wait
05 combase!CoMsgWaitInProcessEvents
06 Windows_UI!Windows::UI::Core::CDispatcher::WaitAndProcessMessages
07 Windows_UI!Windows::UI::Core::CDispatcher::ProcessEvents
08 Windows_UI_Xaml!CJupiterWindow::RunCoreWindowMessageLoop
09 Windows_UI_Xaml!CJupiterControl::RunMessageLoop
0a Windows_UI_Xaml!DirectUI::DXamlCore::RunMessageLoop
0b twinapi_appcore!Windows::ApplicationModel::Core::CoreApplicationView::Run
[...]
0e kernel32!BaseThreadInitThunk
0f ntdll!RtlUserThreadStart

# Call Site
00 user32!NtUserCallNoParam
01 user32!MsgWaitForMultipleObjectsEx
02 combase!ASTAWaitContext::KernelWait
03 combase!ASTAWaitContext::Wait
04 combase!CoMsgWaitInProcessEvents
05 Windows_UI!Windows::UI::Core::CDispatcher::WaitAndProcessMessages
06 Windows_UI!Windows::UI::Core::CDispatcher::ProcessEvents
07 Windows_UI_Xaml!CJupiterWindow::RunCoreWindowMessageLoop
08 Windows_UI_Xaml!CJupiterControl::RunMessageLoop
09 Windows_UI_Xaml!DirectUI::DXamlCore::RunMessageLoop
0a twinapi_appcore!Windows::ApplicationModel::Core::CoreApplicationView::Run
[…]
0d kernel32!BaseThreadInitThunk
0e ntdll!RtlUserThreadStart

The first is traditionally waiting, but the other was waked up to process an input message and may be other blocked in kernel or doing some work there. In case of a doubt we can double check the last executed instruction (from the stack trace perspective):

# Child-SP          RetAddr           Call Site
00 000000b9`358bf068 00007ffd`3de0a3be user32!NtUserCallNoParam+0xa
01 000000b9`358bf070 00007ffd`3dbdc106 user32!MsgWaitForMultipleObjectsEx+0×15e
[…]

0:000> ub user32!NtUserCallNoParam+0xa
user32!NtUserInvalidateRect:
00007ffd`3de1fc90 4c8bd1          mov     r10,rcx
00007ffd`3de1fc93 b807100000      mov     eax,1007h
00007ffd`3de1fc98 0f05            syscall
00007ffd`3de1fc9a c3              ret
00007ffd`3de1fc9b 0f1f440000      nop     dword ptr [rax+rax]
user32!NtUserCallNoParam:
00007ffd`3de1fca0 4c8bd1          mov     r10,rcx
00007ffd`3de1fca3 b808100000      mov     eax,1008h
00007ffd`3de1fca8 0f05            syscall

System Calls (especially discovered in unmanaged Execution Residue data and from Past Stack Traces reconstruction from raw stack regions and Historical Information) may aid in memory forensics (broadly speaking as reconstruction of past behaviour) in addition to System Objects.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

When the same API function is executed it may leave the same data structure in its Execution Residue inside raw stack regions of different threads. We call this analysis pattern Frame Regularity. It may be used to find and correctly interpret stack frame values from thread stack traces or regions based on values found in some other thread stack trace or region where we already know their meaning.

Consider the critical section Wait Chain below. We want to know what critical section the thread F34 is waiting for. The backwards disassembly of 00007ffd`41894e14 return address doesn’t show the critical section address explicitly. However, 00007ffd`43827e40 return address disassembly from a different critical section thread owner D7C shows critical section address 00007ffd`4391d8a8 which we can find as a 3rd “argument” to ntdll!RtlpWaitOnCriticalSection (which is just a value picked up from the stack region and not the real function argument passed by stack). Using Frame Regularity we try to see if the 3rd value for the same function from the thread F34 under question is also a critical section address 00000071`3f08c488. Indeed, we find that it is the address of the critical section owned by the thread B18.

0:000> !cs -l -o -s
-----------------------------------------
DebugInfo          = 0x00007ffd4391dbe8
Critical section   = 0x00007ffd4391d8a8 (ntdll!LdrpLoaderLock+0x0)
LOCKED
LockCount          = 0x13
WaiterWoken        = No
OwningThread       = 0×0000000000000f34
RecursionCount     = 0×1
LockSemaphore      = 0×8B0
SpinCount          = 0×0000000004000000
OwningThread DbgId = ~14s
OwningThread Stack =
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000071`40d4c618 00007ffd`43829a85 : 00000071`3f08c488 00000000`00000002 00000000`bd000200 00000000`00000000 : ntdll!NtWaitForSingleObject+0xa
00000071`40d4c620 00007ffd`43827f44 : 00000000`a90002ab 00007ffd`40d52eba 00000071`3f08c488 00000000`00000000 : ntdll!RtlpWaitOnCriticalSection+0xe1
00000071`40d4c6f0 00007ffd`41894e14 : 00000071`3f08c488 00007ffd`4229b3a8 00000000`00000000 00000071`3f08c470 : ntdll!RtlpEnterCriticalSectionContended+0xa4
00000071`40d4c730 00007ffd`41895d8d : 00000071`40c52ab0 00000000`00000000 00230022`00210020 00270026`00250024 : shell32!kfapi::CFolderPathResolver::GetPath+0×168
00000071`40d4dd50 00007ffd`41892d50 : 00000000`00000080 00000000`000000ce 00000000`00000104 00000000`000003ff : shell32!kfapi::CFolderCache::GetPath+0×772
00000071`40d4e0c0 00007ffd`41893540 : 00000071`40d4e258 00000000`20000020 00000000`000000ce 00000071`42141080 : shell32!kfapi::CKFFacade::GetFolderPath+0×120
00000071`40d4e220 00007ffd`4189368c : 00000000`00008023 00000000`000000ce 00000000`5307cc38 00000000`000000ce : shell32!SHGetFolderPathEx+0×80
00000071`40d4e290 00007ffd`40d61939 : 00000000`00000000 00000000`00000000 00000000`00000037 00000000`00008023 : shell32!SHGetFolderPathWWorker+0xf9
00000071`40d4e2f0 00007ffd`41a80fba : 00000000`00000000 00000000`00000000 00000071`40d4e400 00000071`42010000 : KERNELBASE!SHGetFolderPathW+0×49
00000071`40d4e330 00007ffd`40d7e60f : 00000071`00000001 00000000`00000020 00000000`00000035 00000071`42141080 : shell32!SHGetSpecialFolderPathWWorker+0×23
00000071`40d4e370 00000000`5317704d : 00000071`40d4e7d0 00000071`40d4ee68 00000000`00000000 00000000`53062b30 : KERNELBASE!SHGetSpecialFolderPathW+0×3f
00000071`40d4e3a0 00000000`530e72f1 : 00000000`5333fa10 00000000`00000001 00000000`5333fa10 00000000`530ab470 : ModuleA+0xccded
[…]
00000071`40d4eba0 00007ffd`4380e82e : 00000071`40cc6780 00000000`53060000 00000071`00000001 00007ffd`40892421 : ntdll!LdrpCallInitRoutine+0×4c
00000071`40d4ec00 00007ffd`4380e624 : 00000071`40cb2790 00007ffd`4384ee00 00000071`40cb2790 00000000`53224dec : ntdll!LdrpInitializeNode+0×176
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
—————————————–
DebugInfo          = 0×0000007140c52a70
Critical section   = 0×00007ffd31f41a88 (localspl!g_csBidiAccess+0×8)
LOCKED
LockCount          = 0×4
WaiterWoken        = No
OwningThread       = 0×0000000000000d7c
RecursionCount     = 0×1
LockSemaphore      = 0×1410
SpinCount          = 0×00000000020007d0
OwningThread DbgId = ~7s
OwningThread Stack =
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000071`4029e708 00007ffd`43829a85 : 0000a498`9a4b3f83 00000071`4029ec00 00000071`4029e880 00000071`4029e858 : ntdll!NtWaitForSingleObject+0xa
00000071`4029e710 00007ffd`43827f44 : 00000000`00000002 00007ffd`00000000 00007ffd`4391d8a8 00000071`4029e9d0 : ntdll!RtlpWaitOnCriticalSection+0xe1
00000071`4029e7e0 00007ffd`43827e40 : 00007ffd`41613750 00000071`3efa2a10 00007ffd`416b4d58 00007ffd`41686000 : ntdll!RtlpEnterCriticalSectionContended+0xa4
00000071`4029e820 00007ffd`4380ad5a : 00007ffd`41686000 00000071`4029e8e0 00007ffd`41613750 00000071`4029eeb8 : ntdll!LdrpAcquireLoaderLock+0×2c
00000071`4029e860 00007ffd`4386785f : 00000000`00000000 00007ffd`41685c64 00000000`00000000 00000071`4029e980 : ntdll!LdrpGetDelayloadExportDll+0×7e
00000071`4029e920 00007ffd`43826c8e : 00000000`00000000 00007ffd`41685c64 00000000`00000000 00007ffd`41613750 : ntdll!LdrpHandleProtectedDelayload+0×3f
00000071`4029ede0 00007ffd`4153763b : 00000071`4029ef50 00000000`00000000 00007ffd`2f4d3be0 00000000`00000003 : ntdll!LdrResolveDelayLoadedAPI+0×8e
00000071`4029ee40 00007ffd`415376df : 00000071`3ffe3020 00007ffd`2f4d2482 00000071`3ffe3020 00000071`3ffcb078 : combase!__delayLoadHelper2+0×2b
00000071`4029ee80 00007ffd`415239b6 : a4e9da62`d9130000 00000071`4029eed0 00000000`00000000 00000000`0000006c : combase!_tailMerge_OLEAUT32_dll+0×3f
00000071`4029eef0 00007ffd`41523883 : 00000071`40c22230 00000071`4029f029 00000000`80070015 00000071`3ffcf598 : combase!wCoUninitialize+0xb6
00000071`4029ef20 00007ffd`31e74e1a : 00000071`40c22230 00000000`00000000 00000000`80070015 00007ffd`2f4d3be0 : combase!CoUninitialize+0xb3
00000071`4029ef50 00007ffd`31e7459c : 00000000`00000000 00000071`4029f1d8 00007ffd`31f416d8 00000000`00000001 : localspl!GetBidiStrings+0×3be
00000071`4029f090 00007ffd`31e72150 : 00000000`00000000 00000071`4029f240 00007ffd`31f416d8 00000000`00000000 : localspl!GetBidiPropertiesForDevnode+0×9c
00000071`4029f140 00007ffd`31e7b03b : 00000000`00000008 00000071`403588e0 00000071`40a83130 00000000`00000001 : localspl!GetPropertiesFromPrinter+0×1f4
00000071`4029f750 00007ffd`31e7ae90 : 00000071`40aa6e40 00000000`00000000 00000000`00000001 00007ffd`ffff4b5e : localspl!CreatePrintQueueDevnodeWorker+0xab
00000071`4029f8a0 00007ffd`31eb475e : 00000000`00000000 00000000`00000000 00000071`3f05f470 00000000`00000000 : localspl!CreatePrintQueueDevnode+0xb4
00000071`4029f920 00007ffd`31eb46eb : 00000071`3ffe35c0 00000071`3f05f470 00007ffd`31eb4710 00000000`00000000 : localspl!DevnodeCreationWorkItem::Run+0×4e
00000071`4029f990 00007ffd`4386247e : 00007ffd`31eb46b0 00000071`3f05f3c0 00000071`4029fb78 00000071`3f05f470 : localspl!NThreadingLibrary::TWorkCrew::tpSimpleCallback+0×3b
00000071`4029f9c0 00007ffd`43824a5d : 00000071`3f05f470 00000071`3efd23f0 00000000`00000000 00000071`3f01e3e8 : ntdll!TppSimplepExecuteCallback+0×7e
00000071`4029fa00 00007ffd`410313d2 : 00000000`00000000 00007ffd`438245e0 00000071`3f01bf10 00000000`00000000 : ntdll!TppWorkerThread+0×47d
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
—————————————–
DebugInfo          = 0×0000007140c52970
Critical section   = 0×0000000001017600 (ModuleB+0×38F80)
LOCKED
LockCount          = 0×1
WaiterWoken        = No
OwningThread       = 0×0000000000000fe8
RecursionCount     = 0×1
LockSemaphore      = 0×14A0
SpinCount          = 0×00000000020007d0
OwningThread DbgId = ~22s
OwningThread Stack =
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000071`4102e6a8 00007ffd`43829a85 : 00000000`00000050 00007ffd`4381c937 00000071`3f110000 00000000`00000000 : ntdll!NtWaitForSingleObject+0xa
00000071`4102e6b0 00007ffd`43827f44 : 00000000`00000000 00000000`00000000 00007ffd`4391d8a8 00000000`00000000 : ntdll!RtlpWaitOnCriticalSection+0xe1
00000071`4102e780 00007ffd`43827e40 : 00000071`4102ebe8 00000000`00000000 00000071`4102ebb8 00000071`40cc5a00 : ntdll!RtlpEnterCriticalSectionContended+0xa4
00000071`4102e7c0 00007ffd`43818998 : 00007ff7`acadf000 00000071`4102e830 00000071`4102e830 00000071`4102e830 : ntdll!LdrpAcquireLoaderLock+0×2c
00000071`4102e800 00007ffd`4380b229 : 00000000`00000000 00000071`4102ee40 00007ffd`437f4d38 00000000`00000000 : ntdll!LdrpFindOrMapDll+0×4f8
00000071`4102eb50 00007ffd`4380af29 : 00000000`00000000 00000000`00000000 00000071`4102ee40 00000071`4102ee40 : ntdll!LdrpLoadDll+0×295
00000071`4102ed80 00007ffd`40d58e4a : 00000000`00000000 00000071`4102ee50 00000000`010154a0 00000000`00000002 : ntdll!LdrLoadDll+0×99
00000071`4102ee00 00000000`00fd36ec : 00000000`00000000 00000000`010020a0 00000000`010154a0 00000000`00000000 : KERNELBASE!LoadLibraryExW+0xca
00000071`4102ee70 00000000`00fd3aa6 : 00000071`41147420 00000000`028424c0 00000000`00000001 00000071`411473dc : ModuleB+0×1f1c
[…]
00000071`4102f5b0 00007ffd`3a9b5804 : 00000071`40c33ea8 00007ff7`00000008 00000000`00000008 00000071`3ffff190 : winspool!SpoolerPrinterEventNative+0xb1
00000071`4102f620 00007ffd`31a5566d : 00000071`40c33ea8 00000071`4102f719 00000000`00000008 00000000`00000000 : winspool!SpoolerPrinterEvent+0×54
00000071`4102f660 00007ffd`31e76682 : 00007ffd`31a55480 00000071`40c33ea8 00000071`4102f7c0 00000071`40aabcc0 : PrintIsolationProxy!sandbox::PrintSandboxObject::SandboxDriverEvent+0×1ed
00000071`4102f760 00007ffd`31e7659a : 00000071`3ffaf750 00007ffd`31e76610 00000071`40aabcc0 00000000`00000008 : localspl!sandbox::SandboxObserver::SandboxDriverEvent+0×72
00000071`4102f7b0 00007ffd`31e7621b : 00000071`40c33ea8 00000000`00000000 00000071`40c33ea8 00000000`00000000 : localspl!sandbox::SandboxPrinterDriverEvent+0xda
00000071`4102f820 00007ffd`31e76169 : 00000071`40aa6e40 00000000`00000001 00000000`00000000 00007ffd`31f40001 : localspl!SplDriverEvent+0×5b
00000071`4102f890 00007ffd`31f11c4b : 00000000`00000000 00000071`4102f940 00000071`40aa6e40 00007ffd`31f400c8 : localspl!PrinterDriverEvent+0xbd
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.
—————————————–
DebugInfo          = 0×0000007140c52630
Critical section   = 0×000000713f08c488 (+0×713F08C488)
LOCKED
LockCount          = 0×1
WaiterWoken        = No
OwningThread       = 0×0000000000000b18
RecursionCount     = 0×1
LockSemaphore      = 0×14AC
SpinCount          = 0×00000000020007d0
OwningThread DbgId = ~1s
OwningThread Stack =
Child-SP          RetAddr           : Args to Child                                                           : Call Site
00000071`3fe4b5e8 00007ffd`43829a85 : 00000001`00000000 00000000`00000000 00000071`41927fd0 00007ffd`43821067 : ntdll!NtWaitForSingleObject+0xa
00000071`3fe4b5f0 00007ffd`43827f44 : 00000000`00000000 00000000`0000017b 00007ffd`4391d8a8 00000000`00000000 : ntdll!RtlpWaitOnCriticalSection+0xe1
00000071`3fe4b6c0 00007ffd`43827e40 : 00007ffd`41c57850 00000071`3efe3700 00007ffd`4229bf90 00007ffd`42203380 : ntdll!RtlpEnterCriticalSectionContended+0xa4
00000071`3fe4b700 00007ffd`4380ad5a : 00007ffd`42203380 00000071`3fe4b7c0 00007ffd`41c57850 00007ffd`4381609c : ntdll!LdrpAcquireLoaderLock+0×2c
00000071`3fe4b740 00007ffd`4386785f : 00000000`00000000 00007ffd`422026f4 00000071`3ef01790 00000071`3fe4b860 : ntdll!LdrpGetDelayloadExportDll+0×7e
00000071`3fe4b800 00007ffd`43826c8e : 00000000`00000000 00007ffd`422026f4 00000000`00000000 00007ffd`41c57850 : ntdll!LdrpHandleProtectedDelayload+0×3f
00000071`3fe4bcc0 00007ffd`41a818eb : 00000071`3fe4c910 00000071`3fe4c8a0 00000000`00000000 ffffffff`ffffffff : ntdll!LdrResolveDelayLoadedAPI+0×8e
00000071`3fe4bd20 00007ffd`41896d9f : 00000071`40c52670 ffffffff`ffffffff 00000000`00000000 00000071`3fe4e201 : shell32!_delayLoadHelper2+0×2b
00000071`3fe4bd60 00007ffd`41894374 : 00000071`3fe4c8a0 00000071`3fe4bf10 00000000`00000027 00000071`3fe4bed0 : shell32!_tailMerge_api_ms_win_core_com_l1_1_1_dll+0×3f
00000071`3fe4bdd0 00007ffd`4189409a : 00000000`00000000 00000000`00000000 00000071`3fe4c910 00000000`00000000 : shell32!kfapi::CFolderDefinitionStorage::_LoadRegistry+0xb4
00000071`3fe4c440 00007ffd`41896943 : 00000000`00000031 00000000`00000000 00000071`3fe4c698 00000071`40c52670 : shell32!kfapi::CFolderDefinitionStorage::Load+0×66
00000071`3fe4c6a0 00007ffd`4189796a : 00000000`00000000 00000071`40c539a0 00000071`40c37700 00007ffd`41a80f90 : shell32!kfapi::CFolderPathResolver::GetPath+0×242
00000071`3fe4dcc0 00007ffd`41892d50 : 00000071`3fe4e050 00000071`3fe4e038 00000071`3fe4e038 00000000`00000200 : shell32!kfapi::CFolderCache::GetPath+0xb74
00000071`3fe4e030 00007ffd`41893540 : 00000071`3fe4e1c8 00000071`40cb16f0 00000071`40cb16f0 00000000`00000000 : shell32!kfapi::CKFFacade::GetFolderPath+0×120
00000071`3fe4e190 00007ffd`4189368c : 00000000`00008023 00007ffd`4382c0ae 00000000`00000000 00000071`3ef92bd0 : shell32!SHGetFolderPathEx+0×80
00000071`3fe4e200 00007ffd`40d61939 : 00000000`00000000 00000000`00000000 00007ffd`40dc3f80 00000000`00008023 : shell32!SHGetFolderPathWWorker+0xf9
00000071`3fe4e260 00007ffd`41a80fba : 00000000`00000000 00000000`00000000 00000071`3fe4e370 00000000`00000000 : KERNELBASE!SHGetFolderPathW+0×49
00000071`3fe4e2a0 00007ffd`40d7e60f : 00000000`00000000 00000071`3fe4e370 00000000`00000023 00000000`00000001 : shell32!SHGetSpecialFolderPathWWorker+0×23
00000071`3fe4e2e0 00000000`540b036d : 00000071`3fe4e630 00000000`00000001 00000000`00000000 00000071`3fe4e6b0 : KERNELBASE!SHGetSpecialFolderPathW+0×3f
00000071`3fe4e310 00000000`54096f45 : 00000071`3fe4e708 00000000`00000001 00000071`3fe4e6d8 00000000`53fd1903 : ModuleC+0xdca2d
ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled.

0:000> ub 00007ffd`41894e14
shell32!kfapi::CFolderPathResolver::GetPath+0×145:
00007ffd`41894df1 498d5d30        lea     rbx,[r13+30h]
00007ffd`41894df5 48895c2468      mov     qword ptr [rsp+68h],rbx
00007ffd`41894dfa 4c897c2470      mov     qword ptr [rsp+70h],r15
00007ffd`41894dff 498bff          mov     rdi,r15
00007ffd`41894e02 4c897c2470      mov     qword ptr [rsp+70h],r15
00007ffd`41894e07 488975e0        mov     qword ptr [rbp-20h],rsi
00007ffd`41894e0b 488bce          mov     rcx,rsi
00007ffd`41894e0e ff159cbaa900    call    qword ptr [shell32!_imp_EnterCriticalSection (00007ffd`423308b0)]

0:000> ub 00007ffd`43827e40
ntdll!LdrpAcquireLoaderLock:
00007ffd`43827e14 4053            push    rbx
00007ffd`43827e16 4883ec30        sub     rsp,30h
00007ffd`43827e1a 488d0d4f8d0f00  lea     rcx,[ntdll!LdrpModuleEnumLock (00007ffd`43920b70)]
00007ffd`43827e21 e8ca040000      call    ntdll!RtlAcquireSRWLockShared (00007ffd`438282f0)
00007ffd`43827e26 803c258403fe7f00 cmp     byte ptr [SharedUserData+0×384 (00000000`7ffe0384)],0
00007ffd`43827e2e 0f85242c0700    jne     ntdll!LdrpAcquireLoaderLock+0×72c44 (00007ffd`4389aa58)
00007ffd`43827e34 488d0d6d5a0f00  lea     rcx,[ntdll!LdrpLoaderLock (00007ffd`4391d8a8)]
00007ffd`43827e3b e820000000      call    ntdll!RtlEnterCriticalSection (00007ffd`43827e60)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

This is High Contention pattern variant for .NET heap where we can see gc_heap functions in unmanaged Stack Trace Collection, Spiking Threads doing garbage collection, and threads waiting for object allocation:

0:000> ~*k

[...]

4  Id: ad8.1338 Suspend: 0 Teb: 00007ff5`ff774000 Unfrozen
# Child-SP          RetAddr           Call Site
00 00000030`f9dff1c8 00007ff9`f4121118 ntdll!NtWaitForSingleObject+0xa
01 00000030`f9dff1d0 00007ff9`ee3594fb KERNELBASE!WaitForSingleObjectEx+0x94
02 00000030`f9dff270 00007ff9`ee3594a7 clr!CLREventWaitHelper2+0x38
03 00000030`f9dff2b0 00007ff9`ee359430 clr!CLREventWaitHelper+0x1f
04 00000030`f9dff310 00007ff9`ee533c96 clr!CLREventBase::WaitEx+0x63
05 00000030`f9dff3a0 00007ff9`ee85849a clr!SVR::gc_heap::user_thread_wait+0x58
06 00000030`f9dff3d0 00007ff9`ee74073f clr!SVR::gc_heap::background_gc_wait+0x4a
07 00000030`f9dff400 00007ff9`ee740cfd clr!SVR::GCHeap::GarbageCollect+0x2998df
08 00000030`f9dff440 00007ff9`ee4ccdf9 clr!SVR::WaitForFinalizerEvent+0x3272ed
09 00000030`f9dff480 00007ff9`ee35c481 clr!SVR::GCHeap::FinalizerThreadWorker+0x4a
0a 00000030`f9dff4c0 00007ff9`ee35c408 clr!ManagedThreadBase_DispatchInner+0x2d
0b 00000030`f9dff500 00007ff9`ee35c379 clr!ManagedThreadBase_DispatchMiddle+0x6c
0c 00000030`f9dff600 00007ff9`ee46c477 clr!ManagedThreadBase_DispatchOuter+0x75
0d 00000030`f9dff690 00007ff9`ee417de6 clr!SVR::GCHeap::FinalizerThreadStart+0xd7
0e 00000030`f9dff730 00007ff9`f60413d2 clr!Thread::intermediateThreadProc+0x7d
0f 00000030`f9dff7f0 00007ff9`f6ea54e4 kernel32!BaseThreadInitThunk+0x22
10 00000030`f9dff820 00000000`00000000 ntdll!RtlUserThreadStart+0x34

[...]

8  Id: ad8.1960 Suspend: 0 Teb: 00007ff5`ff58c000 Unfrozen
# Child-SP          RetAddr           Call Site
00 00000030`fb90b898 00007ff9`f4121118 ntdll!NtWaitForSingleObject+0xa
01 00000030`fb90b8a0 00007ff9`ee3594fb KERNELBASE!WaitForSingleObjectEx+0x94
02 00000030`fb90b940 00007ff9`ee3594a7 clr!CLREventWaitHelper2+0x38
03 00000030`fb90b980 00007ff9`ee359430 clr!CLREventWaitHelper+0x1f
04 00000030`fb90b9e0 00007ff9`ee533c96 clr!CLREventBase::WaitEx+0x63
05 00000030`fb90ba70 00007ff9`ee85849a clr!SVR::gc_heap::user_thread_wait+0x58
06 00000030`fb90baa0 00007ff9`ee780417 clr!SVR::gc_heap::background_gc_wait+0x4a
07 00000030`fb90bad0 00007ff9`ee4c41e8 clr!SVR::gc_heap::wait_for_bgc_high_memory+0x2b8907
08 00000030`fb90bb60 00007ff9`ee355272 clr!SVR::GCHeap::Alloc+0×257
09 00000030`fb90bbc0 00007ff9`eb524a1d clr!JIT_New+0×142
0a 00000030`fb90be70 00007ff9`e8bd8ec8 System_Xml_ni+0×154a1d
0b 00000030`fb90bed0 00007ff9`e8d6ec7c System_Runtime_Serialization_ni+0xa8ec8
0c 00000030`fb90bf20 00007ff9`e9ed637b System_Runtime_Serialization_ni+0×23ec7c
0d 00000030`fb90bf80 00007ff9`ea6df472 System_ServiceModel_ni+0xdd637b
0e 00000030`fb90bfe0 00000030`fb90bfc0 System_ServiceModel_ni+0×15df472
0f 00000030`fb90bfe8 00000000`00000000 0×00000030`fb90bfc0

[...]

14  Id: ad8.11d4 Suspend: 0 Teb: 00007ff5`ff586000 Unfrozen
# Child-SP          RetAddr           Call Site
00 00000031`4a42eb10 00007ff9`ee4a3a76 clr!SVR::gc_heap::background_mark_simple1+0xcd9
01 00000031`4a42eba0 00007ff9`ee4a458b clr!SVR::gc_heap::background_mark_simple+0×99
02 00000031`4a42ebd0 00007ff9`ee52e68c clr!SVR::gc_heap::background_promote+0×15d
03 00000031`4a42ec40 00007ff9`ee538836 clr!PinObject+0×2c
04 00000031`4a42ec80 00007ff9`ee532fc8 clr!ScanConsecutiveHandlesWithoutUserData+0×62
05 00000031`4a42ecb0 00007ff9`ee5338f2 clr!BlockScanBlocksWithoutUserData+0×44
06 00000031`4a42ece0 00007ff9`ee5339fc clr!ProcessScanQNode+0×3d
07 00000031`4a42ed10 00007ff9`ee53397d clr!ProcessScanQueue+0×3d
08 00000031`4a42ed40 00007ff9`ee533bcf clr!xxxTableScanQueuedBlocksAsync+0×5d
09 00000031`4a42ed70 00007ff9`ee5353a5 clr!xxxAsyncSegmentIterator+0×2d
0a 00000031`4a42eda0 00007ff9`ee533ab6 clr!TableScanHandles+0×79
0b 00000031`4a42ee50 00007ff9`ee535607 clr!xxxTableScanHandlesAsync+0×9b
0c 00000031`4a42eff0 00007ff9`ee53795d clr!HndScanHandlesForGC+0×137
0d 00000031`4a42f090 00007ff9`ee535769 clr!Ref_TracePinningRoots+0xfd
0e 00000031`4a42f110 00007ff9`ee4a4351 clr!CNameSpace::GcScanHandles+0×4d
0f 00000031`4a42f150 00007ff9`ee55cbe9 clr!SVR::gc_heap::background_mark_phase+0×331
10 00000031`4a42f1e0 00007ff9`ee4a32ed clr!SVR::gc_heap::gc1+0xcd
11 00000031`4a42f240 00007ff9`ee417de6 clr!SVR::gc_heap::bgc_thread_function+0×177
12 00000031`4a42f280 00007ff9`f60413d2 clr!Thread::intermediateThreadProc+0×7d
13 00000031`4a42fcc0 00007ff9`f6ea54e4 kernel32!BaseThreadInitThunk+0×22
14 00000031`4a42fcf0 00000000`00000000 ntdll!RtlUserThreadStart+0×34

15  Id: ad8.f9c Suspend: 0 Teb: 00007ff5`ff648000 Unfrozen
# Child-SP          RetAddr           Call Site
00 00000030`f4eddb58 00007ff9`f4121118 ntdll!NtWaitForSingleObject+0xa
01 00000030`f4eddb60 00007ff9`ee3594fb KERNELBASE!WaitForSingleObjectEx+0x94
02 00000030`f4eddc00 00007ff9`ee3594a7 clr!CLREventWaitHelper2+0x38
03 00000030`f4eddc40 00007ff9`ee359430 clr!CLREventWaitHelper+0x1f
04 00000030`f4eddca0 00007ff9`ee533c96 clr!CLREventBase::WaitEx+0x63
05 00000030`f4eddd30 00007ff9`ee85849a clr!SVR::gc_heap::user_thread_wait+0x58
06 00000030`f4eddd60 00007ff9`ee780417 clr!SVR::gc_heap::background_gc_wait+0x4a
07 00000030`f4eddd90 00007ff9`ee4c41e8 clr!SVR::gc_heap::wait_for_bgc_high_memory+0x2b8907
08 00000030`f4edde20 00007ff9`ee355272 clr!SVR::GCHeap::Alloc+0×257
09 00000030`f4edde80 00007ff9`e9f72cfe clr!JIT_New+0×142
0a 00000030`f4ede130 00007ff9`e6e1bf11 System_ServiceModel_ni+0xe72cfe
0b 00000030`f4ede1b0 00007ff9`e6e1be90 System_ServiceModel_Internals_ni+0×4bf11
0c 00000030`f4ede210 00007ff9`ed205156 System_ServiceModel_Internals_ni+0×4be90
0d 00000030`f4ede270 00007ff9`ee35ab53 mscorlib_ni+0×545156
0e 00000030`f4ede2d0 00007ff9`ee35aa3e clr!CallDescrWorkerInternal+0×83
0f 00000030`f4ede310 00007ff9`ee395b80 clr!CallDescrWorkerWithHandler+0×4a
10 00000030`f4ede350 00007ff9`ee9160b8 clr!DispatchCallSimple+0×60
11 00000030`f4ede3e0 00007ff9`ee35c481 clr!BindIoCompletionCallBack_Worker+0xb8
12 00000030`f4ede470 00007ff9`ee35c408 clr!ManagedThreadBase_DispatchInner+0×2d
13 00000030`f4ede4b0 00007ff9`ee35c379 clr!ManagedThreadBase_DispatchMiddle+0×6c
14 00000030`f4ede5b0 00007ff9`ee35c4bb clr!ManagedThreadBase_DispatchOuter+0×75
15 00000030`f4ede640 00007ff9`ee916219 clr!ManagedThreadBase_FullTransitionWithAD+0×2f
16 00000030`f4ede6a0 00007ff9`ee916129 clr!BindIoCompletionCallbackStubEx+0xb9
17 00000030`f4ede720 00007ff9`ee4da72d clr!BindIoCompletionCallbackStub+0×9
18 00000030`f4ede750 00007ff9`ee417de6 clr!ThreadpoolMgr::CompletionPortThreadStart+0×23d
19 00000030`f4ede7f0 00007ff9`f60413d2 clr!Thread::intermediateThreadProc+0×7d
1a 00000030`f4edf7b0 00007ff9`f6ea54e4 kernel32!BaseThreadInitThunk+0×22
1b 00000030`f4edf7e0 00000000`00000000 ntdll!RtlUserThreadStart+0×34

16  Id: ad8.d50 Suspend: 0 Teb: 00007ff5`ff64a000 Unfrozen
# Child-SP          RetAddr           Call Site
00 00000030`f4cde038 00007ff9`f4121118 ntdll!NtWaitForSingleObject+0xa
01 00000030`f4cde040 00007ff9`ee3594fb KERNELBASE!WaitForSingleObjectEx+0x94
02 00000030`f4cde0e0 00007ff9`ee3594a7 clr!CLREventWaitHelper2+0x38
03 00000030`f4cde120 00007ff9`ee359430 clr!CLREventWaitHelper+0x1f
04 00000030`f4cde180 00007ff9`ee533c96 clr!CLREventBase::WaitEx+0x63
05 00000030`f4cde210 00007ff9`ee85849a clr!SVR::gc_heap::user_thread_wait+0x58
06 00000030`f4cde240 00007ff9`ee780417 clr!SVR::gc_heap::background_gc_wait+0x4a
07 00000030`f4cde270 00007ff9`ee4c41e8 clr!SVR::gc_heap::wait_for_bgc_high_memory+0x2b8907
08 00000030`f4cde300 00007ff9`ee355272 clr!SVR::GCHeap::Alloc+0×257
09 00000030`f4cde360 00007ff9`e9f72cfe clr!JIT_New+0×142
0a 00000030`f4cde610 00007ff9`e6e1bf11 System_ServiceModel_ni+0xe72cfe
0b 00000030`f4cde690 00007ff9`e6e1be90 System_ServiceModel_Internals_ni+0×4bf11
0c 00000030`f4cde6f0 00007ff9`ed205156 System_ServiceModel_Internals_ni+0×4be90
0d 00000030`f4cde750 00007ff9`ee35ab53 mscorlib_ni+0×545156
0e 00000030`f4cde7b0 00007ff9`ee35aa3e clr!CallDescrWorkerInternal+0×83
0f 00000030`f4cde7f0 00007ff9`ee395b80 clr!CallDescrWorkerWithHandler+0×4a
10 00000030`f4cde830 00007ff9`ee9160b8 clr!DispatchCallSimple+0×60
11 00000030`f4cde8c0 00007ff9`ee35c481 clr!BindIoCompletionCallBack_Worker+0xb8
12 00000030`f4cde950 00007ff9`ee35c408 clr!ManagedThreadBase_DispatchInner+0×2d
13 00000030`f4cde990 00007ff9`ee35c379 clr!ManagedThreadBase_DispatchMiddle+0×6c
14 00000030`f4cdea90 00007ff9`ee35c4bb clr!ManagedThreadBase_DispatchOuter+0×75
15 00000030`f4cdeb20 00007ff9`ee916219 clr!ManagedThreadBase_FullTransitionWithAD+0×2f
16 00000030`f4cdeb80 00007ff9`ee916129 clr!BindIoCompletionCallbackStubEx+0xb9
17 00000030`f4cdec00 00007ff9`ee4da72d clr!BindIoCompletionCallbackStub+0×9
18 00000030`f4cdec30 00007ff9`ee417de6 clr!ThreadpoolMgr::CompletionPortThreadStart+0×23d
19 00000030`f4cdecd0 00007ff9`f60413d2 clr!Thread::intermediateThreadProc+0×7d
1a 00000030`f4cdfd10 00007ff9`f6ea54e4 kernel32!BaseThreadInitThunk+0×22
1b 00000030`f4cdfd40 00000000`00000000 ntdll!RtlUserThreadStart+0×34

17  Id: ad8.1584 Suspend: 0 Teb: 00007ff5`ff644000 Unfrozen
# Child-SP          RetAddr           Call Site
00 00000030`f4dde9e8 00007ff9`f4121118 ntdll!NtWaitForSingleObject+0xa
01 00000030`f4dde9f0 00007ff9`ee3594fb KERNELBASE!WaitForSingleObjectEx+0x94
02 00000030`f4ddea90 00007ff9`ee3594a7 clr!CLREventWaitHelper2+0x38
03 00000030`f4ddead0 00007ff9`ee359430 clr!CLREventWaitHelper+0x1f
04 00000030`f4ddeb30 00007ff9`ee533c96 clr!CLREventBase::WaitEx+0x63
05 00000030`f4ddebc0 00007ff9`ee85849a clr!SVR::gc_heap::user_thread_wait+0x58
06 00000030`f4ddebf0 00007ff9`ee780417 clr!SVR::gc_heap::background_gc_wait+0x4a
07 00000030`f4ddec20 00007ff9`ee4c41e8 clr!SVR::gc_heap::wait_for_bgc_high_memory+0x2b8907
08 00000030`f4ddecb0 00007ff9`ee35579d clr!SVR::GCHeap::Alloc+0×257
09 00000030`f4dded10 00007ff9`ee355f32 clr!AllocateArrayEx+0×1a6
0a 00000030`f4ddee40 00007ff9`ed204d4f clr!JIT_NewArr1+0×252
0b 00000030`f4ddf0f0 00007ff9`e6e13574 mscorlib_ni+0×544d4f
0c 00000030`f4ddf150 00007ff9`e6e1bf11 System_ServiceModel_Internals_ni+0×43574
0d 00000030`f4ddf1c0 00007ff9`e6e1be90 System_ServiceModel_Internals_ni+0×4bf11
0e 00000030`f4ddf220 00007ff9`ed205156 System_ServiceModel_Internals_ni+0×4be90
0f 00000030`f4ddf280 00007ff9`ee35ab53 mscorlib_ni+0×545156
10 00000030`f4ddf2e0 00007ff9`ee35aa3e clr!CallDescrWorkerInternal+0×83
11 00000030`f4ddf320 00007ff9`ee395b80 clr!CallDescrWorkerWithHandler+0×4a
12 00000030`f4ddf360 00007ff9`ee9160b8 clr!DispatchCallSimple+0×60
13 00000030`f4ddf3f0 00007ff9`ee35c481 clr!BindIoCompletionCallBack_Worker+0xb8
14 00000030`f4ddf480 00007ff9`ee35c408 clr!ManagedThreadBase_DispatchInner+0×2d
15 00000030`f4ddf4c0 00007ff9`ee35c379 clr!ManagedThreadBase_DispatchMiddle+0×6c
16 00000030`f4ddf5c0 00007ff9`ee35c4bb clr!ManagedThreadBase_DispatchOuter+0×75
17 00000030`f4ddf650 00007ff9`ee916219 clr!ManagedThreadBase_FullTransitionWithAD+0×2f
18 00000030`f4ddf6b0 00007ff9`ee916129 clr!BindIoCompletionCallbackStubEx+0xb9
19 00000030`f4ddf730 00007ff9`ee4da72d clr!BindIoCompletionCallbackStub+0×9
1a 00000030`f4ddf760 00007ff9`ee417de6 clr!ThreadpoolMgr::CompletionPortThreadStart+0×23d
1b 00000030`f4ddf800 00007ff9`f60413d2 clr!Thread::intermediateThreadProc+0×7d
1c 00000030`f4ddf840 00007ff9`f6ea54e4 kernel32!BaseThreadInitThunk+0×22
1d 00000030`f4ddf870 00000000`00000000 ntdll!RtlUserThreadStart+0×34

[...]

20  Id: ad8.fcc Suspend: 0 Teb: 00007ff5`ff64c000 Unfrozen
# Child-SP          RetAddr           Call Site
00 00000030`f4fde938 00007ff9`f4121118 ntdll!NtWaitForSingleObject+0xa
01 00000030`f4fde940 00007ff9`ee3594fb KERNELBASE!WaitForSingleObjectEx+0x94
02 00000030`f4fde9e0 00007ff9`ee3594a7 clr!CLREventWaitHelper2+0x38
03 00000030`f4fdea20 00007ff9`ee359430 clr!CLREventWaitHelper+0x1f
04 00000030`f4fdea80 00007ff9`ee533c96 clr!CLREventBase::WaitEx+0x63
05 00000030`f4fdeb10 00007ff9`ee85849a clr!SVR::gc_heap::user_thread_wait+0x58
06 00000030`f4fdeb40 00007ff9`ee780417 clr!SVR::gc_heap::background_gc_wait+0x4a
07 00000030`f4fdeb70 00007ff9`ee4c41e8 clr!SVR::gc_heap::wait_for_bgc_high_memory+0x2b8907
08 00000030`f4fdec00 00007ff9`ee3555f0 clr!SVR::GCHeap::Alloc+0×257
09 00000030`f4fdec60 00007ff9`ee41963a clr!AllocateObject+0×68
0a 00000030`f4fdecf0 00007ff9`ee41983d clr!Thread::GetExposedObject+0×4e
0b 00000030`f4fded90 00007ff9`ee355a3a clr!GetCurrentThreadHelper+0×99
0c 00000030`f4fdeef0 00007ff9`ed162b87 clr!ThreadNative::GetCurrentThread+0×22
0d 00000030`f4fdef20 00007ff9`ed162ab9 mscorlib_ni+0×4a2b87
0e 00000030`f4fdf080 00007ff9`ed1e49c2 mscorlib_ni+0×4a2ab9
0f 00000030`f4fdf0b0 00007ff9`ed1e47ae mscorlib_ni+0×5249c2
10 00000030`f4fdf140 00007ff9`ed18c084 mscorlib_ni+0×5247ae
11 00000030`f4fdf1b0 00007ff9`ee35ab53 mscorlib_ni+0×4cc084
12 00000030`f4fdf250 00007ff9`ee35aa3e clr!CallDescrWorkerInternal+0×83
13 00000030`f4fdf290 00007ff9`ee35b1d2 clr!CallDescrWorkerWithHandler+0×4a
14 00000030`f4fdf2d0 00007ff9`ee40e053 clr!MethodDescCallSite::CallTargetWorker+0×251
15 00000030`f4fdf480 00007ff9`ee35c481 clr!AppDomainTimerCallback_Worker+0×23
16 00000030`f4fdf570 00007ff9`ee35c408 clr!ManagedThreadBase_DispatchInner+0×2d
17 00000030`f4fdf5b0 00007ff9`ee35c379 clr!ManagedThreadBase_DispatchMiddle+0×6c
18 00000030`f4fdf6b0 00007ff9`ee35c4bb clr!ManagedThreadBase_DispatchOuter+0×75
19 00000030`f4fdf740 00007ff9`ee40dfe5 clr!ManagedThreadBase_FullTransitionWithAD+0×2f
1a 00000030`f4fdf7a0 00007ff9`ee40df59 clr!AppDomainTimerCallback+0×66
1b 00000030`f4fdf800 00007ff9`ee40db62 clr!ThreadpoolMgr::AsyncTimerCallbackCompletion+0×36
1c 00000030`f4fdf850 00007ff9`ee540af6 clr!UnManagedPerAppDomainTPCount::DispatchWorkItem+0×122
1d 00000030`f4fdf8f0 00007ff9`ee5409ea clr!ThreadpoolMgr::ExecuteWorkRequest+0×46
1e 00000030`f4fdf920 00007ff9`ee417de6 clr!ThreadpoolMgr::WorkerThreadStart+0xf4
1f 00000030`f4fdf9b0 00007ff9`f60413d2 clr!Thread::intermediateThreadProc+0×7d
20 00000030`f4fdfbf0 00007ff9`f6ea54e4 kernel32!BaseThreadInitThunk+0×22
21 00000030`f4fdfc20 00000000`00000000 ntdll!RtlUserThreadStart+0×34

0:000> !runaway
User Mode Time
Thread       Time
8:1960      0 days 1:09:12.218
14:11d4      0 days 0:03:31.937
2:9e8       0 days 0:03:03.250
3:e34       0 days 0:02:41.562
13:f54       0 days 0:02:38.718
4:1338      0 days 0:00:07.984
0:1014      0 days 0:00:00.046
11:1534      0 days 0:00:00.015
7:1798      0 days 0:00:00.015
20:fcc       0 days 0:00:00.000
19:f48       0 days 0:00:00.000
18:1a60      0 days 0:00:00.000
17:1584      0 days 0:00:00.000
16:d50       0 days 0:00:00.000
15:f9c       0 days 0:00:00.000
12:1988      0 days 0:00:00.000
10:14fc      0 days 0:00:00.000
9:1ae0      0 days 0:00:00.000
6:14cc      0 days 0:00:00.000
5:1a50      0 days 0:00:00.000
1:1258      0 days 0:00:00.000

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Device Stack Trace analysis pattern is about the analysis of a hierarchy of devices based on the current location of IRP Stack Trace:

0: kd> !irp ffffffa80095e6c10
Irp is active with 7 stacks 6 is current (= 0xfffffa80095e6e48)
No Mdl: System buffer=fffffa8007b52380: Thread fffffa8007486060:  Irp stack trace.
cmd  flg cl Device   File     Completion-Context
[N/A(0), N/A(0)]
0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0  0 00000000 00000000 00000000-00000000

Args: 00000000 00000000 00000000 00000000
[N/A(0), N/A(0)]
0  0 00000000 00000000 00000000-00000000

Args: 00000000 fffffa8008b045f0 00000000 00000000
>[IRP_MJ_DEVICE_CONTROL(e), N/A(0)]
5  0 fffffa80095e6060 fffffa8009883b60 00000000-00000000
\Driver\usbscan
Args: 00000000 00000010 80002010 00000000
[IRP_MJ_DEVICE_CONTROL(e), N/A(0)]
5  0 fffffa8007ee4bd0 fffffa8009883b60 00000000-00000000
\Driver\ksthunk
Args: 00000000 00000010 80002010 00000000

0: kd> !devstack  fffffa80095e6060
!DevObj           !DrvObj            !DevExt           ObjectName
fffffa8007ee4bd0  \Driver\ksthunk    fffffa8007ee4d20  00000098
> fffffa80095e6060  \Driver\usbscan    fffffa80095e61b0  Usbscan0
fffffa80073a3060 \Driver\DriverA    fffffa8007e767e0  00000097
[…]

0: kd> !devobj fffffa8007ee4bd0
Device object (fffffa8007ee4bd0) is for:
00000098 \Driver\ksthunk DriverObject fffffa8006396c60
Current Irp 00000000 RefCount 0 Type 00000019 Flags 00002050
Dacl fffff9a100099e11 DevExt fffffa8007ee4d20 DevObjExt fffffa8007ee4d28
ExtensionFlags (0x00000800)  DOE_DEFAULT_SD_PRESENT
Characteristics (0000000000)
AttachedTo (Lower) fffffa80095e6060 \Driver\usbscan
Device queue is not busy.

0: kd> !devobj fffffa80095e6060
Device object (fffffa80095e6060) is for:
Usbscan0 \Driver\usbscan DriverObject fffffa8007943570
Current Irp 00000000 RefCount 3 Type 00000019 Flags 00002050
Dacl fffff9a100099e11 DevExt fffffa80095e61b0 DevObjExt fffffa80095e6530
ExtensionFlags (0x00000800)  DOE_DEFAULT_SD_PRESENT
Characteristics (0000000000)
AttachedDevice (Upper) fffffa8007ee4bd0 \Driver\ksthunk
AttachedTo (Lower) fffffa80073a3060 \Driver\DriverA
Device queue is not busy.

0: kd> !devobj fffffa80073a3060
Device object (fffffa80073a3060) is for:
00000097 \Driver\DriverA DriverObject fffffa80063bd540
Current Irp 00000000 RefCount 1 Type 00000022 Flags 00003044
Dacl fffff9a100099e11 DevExt fffffa8007e767e0 DevObjExt fffffa80073a31d8 DevNode fffffa80086e1cb0
ExtensionFlags (0x00000800)  DOE_DEFAULT_SD_PRESENT
Characteristics (0x00000080)  FILE_AUTOGENERATED_DEVICE_NAME
AttachedDevice (Upper) fffffa80095e6060 \Driver\usbscan
Device queue is not busy.

The direction of device Stack Trace is from top to bottom, opposite to I/O Request Stack Trace which is from bottom to top.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Many OS structures have interesting fields, and it is good to maintain a list of the most useful of them to check in cases of abnormal software behavior. This Value Deviation analysis pattern variant is about structure fields that have values not usually seen in normal cases. For example, we may sometimes encounter Missing Threads in ALPC Wait Chain endpoint process. For example, Main Thread may be missing. The remaining threads may look normal waiting for synchronization objects. However, inspection of _EPROCESS ProcessDelete field shows it is 0y1. This points to the possibility that one of the threads experienced exception, was terminated and triggered process termination which was blocked by the remaining threads. In such a case we may suggest to enable WER to catch such an exception and save a process crash dump (for example, via LocalDumps registry key). Such processes are not easy to catch as Zombies Processes since their number of active threads is non-zero and they still have private memory.

In some cases we may even proactively check structure fields for specific values. For example, we can use one of the techniques to navigate an active process list from Advanced Windows Memory Dump Analysis training and list process image file names together with their ProcessDelete field:

0: kd> dt nt!_EPROCESS ActiveProcessLinks
+0x2f0 ActiveProcessLinks : _LIST_ENTRY

0: kd> !list -t nt!_LIST_ENTRY.Flink -x "? @$extret-2f0; dt nt!_EPROCESS ImageFileName @$extret-2f0; dt nt!_EPROCESS ProcessDelete @$extret-2f0" poi(nt!PsActiveProcessHead)
Evaluate expression: -92874785243072 = ffffab87`e76ac040
+0x450 ImageFileName : [15] "System"
+0x304 ProcessDelete : 0y0

Evaluate expression: -92874719545280 = ffffab87`eb553840
+0x450 ImageFileName : [15] "smss.exe"
+0x304 ProcessDelete : 0y0

Evaluate expression: -92874717837248 = ffffab87`eb6f4840
+0x450 ImageFileName : [15] "csrss.exe"
+0x304 ProcessDelete : 0y0

[...]

Evaluate expression: -92874709187456 = ffffab87`ebf34480
+0x450 ImageFileName : [15] "ServiceA.exe"
+0×304 ProcessDelete : 0y1

[...]

0: kd> !process ffffab87ebf34480 3f
[...]
THREAD ffffab8815527080 [...]
THREAD ffffab87eb41e040 [...]
THREAD ffffab881bf3c800 [...]
[...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

If OS is not inside a virtual machine it is difficult to get consistent live snapshots of physical memory (see Inconsistent Dump analysis pattern). Mirror dump options in LiveKd can save a consistent kernel memory dump. Then we can either use Fiber Bundle technique of saving individual process memory dumps or create inconsistent complete memory dump using LiveKd or both. We call this pattern Mirror Dump Set.

We can identify mirror dump with the following current stack trace:

0: kd> version
...
64-bit Kernel bitmap dump: ...
...

0: kd> k
# Child-SP RetAddr Call Site
00 ffffd000`26121700 fffff803`cf5f5ee3 nt!IopLiveDumpEndMirroringCallback+0x7f
01 ffffd000`26121750 fffff803`cf60561b nt!MmDuplicateMemory+0x807
02 ffffd000`26121830 fffff803`cf851c60 nt!IopLiveDumpCaptureMemoryPages+0x53
03 ffffd000`26121890 fffff803`cf447443 nt!IoCaptureLiveDump+0xf8
04 ffffd000`261218e0 fffff803`cf8ceb0d nt!DbgkCaptureLiveKernelDump+0x2e7
05 ffffd000`26121970 fffff803`cf3debb3 nt!NtSystemDebugControl+0x3f5
06 ffffd000`26121a90 00007ffa`2925205a nt!KiSystemServiceCopyEnd+0x13
07 000000a3`5bcddb48 00000000`00000000 0x00007ffa`2925205a

In one analysis case, we got such a set where we analyzed ALPC Wait Chains with user space stack traces in a complete memory having the endpoint blocked in a filter driver. But the search for stack traces having filter manager in their frames failed due to inconsistency:

0: kd> version
...
64-bit Full kernel dump: ...
...

0: kd> !stacks 2 FltMgr
...
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
TYPE mismatch for thread object at ffffe001b804d638
4.------ NO ETHREAD DATA
...

So we found such kernel space stack traces from the consistent mirror dump.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We got the idea of Unified Stack Trace analysis pattern from Flame Graphs. Like the latter, we combine Stack Trace Collection into one aggregate trace, but we may use the same length for repeated frames and may use different color intensities to present multiplicities. Different frame height may also be used to unify top frames such as waiting API. Different collections may be used in addition to database-like stack traces (Unmanaged, Managed, Predicate, I/O, CPU). The collections may be composed from different varieties of stack traces, such as General, Managed, Module, Quotient, Filters).

As a very simple example, consider this Stack Trace Collection from Notepad:

0:003> ~*kc

0  Id: 984.994 Suspend: 1 Teb: 00007ff6`f411d000 Unfrozen
# Call Site
00 USER32!NtUserGetMessage
01 USER32!GetMessageW
02 notepad!WinMain
03 notepad!WinMainCRTStartup
04 KERNEL32!BaseThreadInitThunk
05 ntdll!RtlUserThreadStart

1  Id: 984.eb8 Suspend: 1 Teb: 00007ff6`f411b000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 KERNEL32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

2  Id: 984.1a8c Suspend: 1 Teb: 00007ff6`f4119000 Unfrozen
# Call Site
00 ntdll!NtWaitForWorkViaWorkerFactory
01 ntdll!TppWorkerThread
02 KERNEL32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

#  3  Id: 984.11b0 Suspend: 1 Teb: 00007ff6`f4117000 Unfrozen
# Call Site
00 ntdll!DbgBreakPoint
01 ntdll!DbgUiRemoteBreakin
02 KERNEL32!BaseThreadInitThunk
03 ntdll!RtlUserThreadStart

The collection can be represented in a more compact form with multiplicities:

USER32!NtUserGetMessage
USER32!GetMessageW
notepad!WinMain           | 2* ntdll!NtWaitForWorkViaWorkerFactory | ntdll!DbgBreakPoint
notepad!WinMainCRTStartup | 2* ntdll!TppWorkerThread               | ntdll!DbgUiRemoteBreakin
4* KERNEL32!BaseThreadInitThunk
4* ntdll!RtlUserThreadStart

It can also be illustrated in the following diagram:

Such diagrams may help to spot Ubiquitous Components quickly.

Unified Stack Trace is also a generalization of Stack Trace Set where the latter only excludes fully duplicated stack traces but the former takes into account Constant Subtraces.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

We do not write a special pattern for individual stowed exceptions (0xC000027B, an application-internal exception) because we consider them examples of Mixed Exceptions (for example, as Managed Code Exceptions enveloped by unmanaged code Software Exception, see stowed exception examples here). However, we introduce a variant of Multiple Exceptions analysis patterns for them because the default analysis WinDbg command (!analyze -v) only shows the first exception or error message stored in the array of stowed exception information structures. The following example shows 3 exceptions. When we load the dump we get this indication of the stowed exceptions:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(2784.2068): Unknown exception - code c000027b (first/second chance not available)
eax=059be538 ebx=00000000 ecx=00000000 edx=00000000 esi=059be898 edi=059be538
eip=750510c0 esp=059be81c ebp=059be940 iopl=0 nv up ei pl nz ac po nc
cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000212
combase!RoFailFastWithErrorContextInternal2+0×109 [inlined in combase!RoFailFastWithErrorContextInternal+0×240]:
750510c0 6a03 push 3

0:008> !error c000027b
Error code: (NTSTATUS) 0xc000027b (3221226107) - An application-internal exception has occurred.

We check Stored Exception:

0:008> .exr -1
ExceptionAddress: 66fc2408 (Windows_UI_Xaml!DirectUI::ErrorHelper::ProcessUnhandledError+0x00432e07)
ExceptionCode: c000027b
ExceptionFlags: 00000001
NumberParameters: 2
Parameter[0]: 08d9bfa8
Parameter[1]: 00000003

We see it has 3 stowed exceptions pointed to by array elements:

0:008> dp 08d9bfa8 L3
08d9bfa8 08d8e084 08d928a8 08d928d0

We check what kind of stowed information structures they point to:

0:008> dt -a3 08d9bfa8 _STOWED_EXCEPTION_INFORMATION_HEADER*
[0] @ 08d9bfa8
———————————————
0×08d8e084
+0×000 Size : 0×28
+0×004 Signature : 0×53453032

[1] @ 08d9bfac
---------------------------------------------
0x08d928a8
+0x000 Size : 0x28
+0×004 Signature : 0×53453032

[2] @ 08d9bfb0
---------------------------------------------
0x08d928d0
+0x000 Size : 0x28
+0×004 Signature : 0×53453032

0:008> .formats 0x53453032
Evaluate expression:
Hex: 53453032
Decimal: 1397043250
Octal: 12321230062
Binary: 01010011 01000101 00110000 00110010
Chars: SE02
Time: Wed Apr 09 12:34:10 2014
Float: low 8.46917e+011 high 0
Double: 6.90231e-315

Since these are version 2 stowed information structures we use this command:

0:008> dt -a3 08d9bfa8 _STOWED_EXCEPTION_INFORMATION_V2*
[0] @ 08d9bfa8
———————————————
0×08d8e084
+0×000 Header : _STOWED_EXCEPTION_INFORMATION_HEADER
+0×008 ResultCode : 802b000a
+0×00c ExceptionForm : 0y01
+0×00c ThreadId : 0y000000000000000000100000011010 (0×81a)
+0×010 ExceptionAddress : 0×74f562f2 Void
+0×014 StackTraceWordSize : 4
+0×018 StackTraceWords : 0×3c
+0×01c StackTrace : 0×07731c08 Void
+0×010 ErrorText : 0×74f562f2 “趍ﯰ???”
+0×020 NestedExceptionType : 0
+0×024 NestedException : (null)

[1] @ 08d9bfac
---------------------------------------------
0x08d928a8
+0x000 Header : _STOWED_EXCEPTION_INFORMATION_HEADER
+0×008 ResultCode : 80004001
+0×00c ExceptionForm : 0y01
+0×00c ThreadId : 0y000000000000000000000000000000 (0)
+0×010 ExceptionAddress : (null)
+0×014 StackTraceWordSize : 4
+0×018 StackTraceWords : 0×3c
+0×01c StackTrace : 0×077d5d4c Void
+0×010 ErrorText : (null)
+0×020 NestedExceptionType : 0
+0×024 NestedException : (null)

[2] @ 08d9bfb0
---------------------------------------------
0x08d928d0
+0x000 Header : _STOWED_EXCEPTION_INFORMATION_HEADER
+0×008 ResultCode : 80004005
+0×00c ExceptionForm : 0y01
+0×00c ThreadId : 0y000000000000000000000000000000 (0)
+0×010 ExceptionAddress : (null)
+0×014 StackTraceWordSize : 4
+0×018 StackTraceWords : 0×19
+0×01c StackTrace : 0×0772df74 Void
+0×010 ErrorText : (null)
+0×020 NestedExceptionType : 0
+0×024 NestedException : (null)

Now we check Stack Traces for individual entries and their error codes:

0:008> !error 802b000a
Error code: (HRESULT) 0×802b000a (2150301706) - <Unable to get error code text>

0:008> .lines -d
Line number information will not be loaded

0:008> dps 0×07731c08 L3c
07731c08  74fba2be combase!RoOriginateErrorW+0×3e
07731c0c  66b900d7 Windows_UI_Xaml!DirectUI::ErrorHelper::OriginateError+0×8d
07731c10  66b8fe5f Windows_UI_Xaml!CJupiterErrorServiceListener::NotifyErrorAdded+0xaf
07731c14  66b8fd60 Windows_UI_Xaml!CErrorService::AddError+0×130
07731c18  66b8f0f8 Windows_UI_Xaml!CErrorService::ReportParserError+0×88
07731c1c  66b8f000 Windows_UI_Xaml!ParserErrorService::ReportError+0xd0
07731c20  670f9654 Windows_UI_Xaml!ParserErrorReporter::SetError+0×61
07731c24  670fbe70 Windows_UI_Xaml!XamlBinaryMetadataReader2::LogError+0×6c
07731c28  670fbd37 Windows_UI_Xaml!<lambda_1d4791754290213f1bf5bc456a504cc5>::operator()+0×41
07731c2c  67032601 Windows_UI_Xaml!XamlBinaryMetadataReader2::LoadProperty+0×1f7e82
07731c30  66cab9d9 Windows_UI_Xaml!XamlBinaryMetadataReader2::GetProperty+0×5c
07731c34  66d528e0 Windows_UI_Xaml!XamlBinaryFormatSubReader2::ReadXamlProperty+0×1d0
07731c38  66d520cc Windows_UI_Xaml!XamlBinaryFormatSubReader2::ReadSetValueConstantNode+0×3c
07731c3c  66d53b55 Windows_UI_Xaml!XamlBinaryFormatSubReader2::TryRead+0×135
07731c40  66d539ec Windows_UI_Xaml!XamlBinaryFormatSubReader2::TryReadHRESULT+0×3c
07731c44  66ca329e Windows_UI_Xaml!CParser::LoadXamlCore+0×5ae
07731c48  66da83d2 Windows_UI_Xaml!CCoreServices::ParseXamlWithExistingFrameworkRoot+0×100
07731c4c  66da8131 Windows_UI_Xaml!CApplication::LoadComponent+0×261
07731c50  66da7e56 Windows_UI_Xaml!Application_LoadComponent+0xaa
07731c54  66da7cfc Windows_UI_Xaml!DirectUI::FrameworkApplication::LoadComponent+0xc2
07731c58  66dab49d Windows_UI_Xaml!DirectUI::FrameworkApplicationFactory::LoadComponentWithResourceLocationImpl+0×6a
07731c5c  66dab418 Windows_UI_Xaml!DirectUI::FrameworkApplicationFactory::LoadComponentWithResourceLocation+0×28
07731c60  6c186b95*** WARNING: Unable to verify checksum for Microsoft.Msn.Weather.dll
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for Microsoft.Msn.Weather.dll -
Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×1b2515
07731c64  6c186af1 Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×1b2471
07731c68  6c199d37 Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×1c56b7
07731c6c  6c366cd8 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xaf038
07731c70  6c366c8d Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xaefed
07731c74  6c366b4d Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xaeead
07731c78  6c365b50 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xadeb0
07731c7c  6c3cc1d7 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0×114537
07731c80  6c3cbb9a Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0×113efa
07731c84  6c3cb9c0 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0×113d20
07731c88  6c35a658 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xa29b8
07731c8c  6c0fd56e Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×128eee
07731c90  6c21e3d5 Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×249d55
07731c94  66e50d37 Windows_UI_Xaml!DirectUI::FrameworkApplicationGenerated::OnLaunchedProtected+0×47
07731c98  66e509d0 Windows_UI_Xaml!DirectUI::FrameworkView::OnActivated+0×240
07731c9c  66cf4b28 Windows_UI_Xaml!`Microsoft::WRL::Callback< Windows::Foundation::ITypedEventHandler< Windows::UI::Core::CoreWindow *,IInspectable *>,DirectUI::Page,Windows::UI::Core::ICoreWindow *,IInspectable *>’::`2′::ComObject::Invoke+0×28
07731ca0  7035361e twinapi_appcore!Microsoft::WRL::InvokeTraits<-2>::InvokeDelegates< <lambda_45dc3980e5a5ff53a9eb289d8a61b7e3>, Windows::Foundation::ITypedEventHandler< Windows::ApplicationModel::Core::CoreApplicationView *,Windows::ApplicationModel::Activation::IActivatedEventArgs *> >+0×4f
07731ca4  70353493 twinapi_appcore!Microsoft::WRL::EventSource< Windows::Foundation::ITypedEventHandler< Windows::ApplicationModel::Core::CoreApplicationView *,Windows::ApplicationModel::Activation::IActivatedEventArgs *>,Microsoft::WRL::InvokeModeOptions<-2> >::DoInvoke< <lambda_45dc3980e5a5ff53a9eb289d8a61b7e3> >+0×42
07731ca8  70342a06 twinapi_appcore!Windows::ApplicationModel::Core::CoreApplicationView::Activate+0×296
07731cac  77425ebc rpcrt4!Invoke+0×34
07731cb0  773f37e3 rpcrt4!NdrStubCall2+0×2e3
07731cb4  74efc1ce combase!CStdStubBuffer_Invoke+0xde
07731cb8  7742364c rpcrt4!CStdStubBuffer_Invoke+0×2c
07731cbc  74fb659b combase!ObjectMethodExceptionHandlingAction< <lambda_adf5d6ba83bff890864fd80ca2bbf1eb> >+0×7b
07731cc0  74f83091 combase!DefaultStubInvoke+0×211
07731cc4  74f8d59e combase!ServerCall::ContextInvoke+0×38e
07731cc8  74f8ecc5 combase!AppInvoke+0xb75
07731ccc  74f81c8c combase!ComInvokeWithLockAndIPID+0×62c
07731cd0  74f66d72 combase!CComApartment::ASTAHandleMessage+0×2c2
07731cd4  74f65bfa combase!ASTAWaitContext::Wait+0×47a
07731cd8  74fbb1b4 combase!ASTAWaitInNewContext+0×81
07731cdc  74fbb0ee combase!ASTAThreadWaitForHandles+0×4e
07731ce0  74fbaf1a combase!CoWaitForMultipleHandles+0xaa
07731ce4  7035772d twinapi_appcore!CTSimpleArray<COSTaskCompletion::TaskContext *,4294967294, CTPolicyCoTaskMem<COSTaskCompletion::TaskContext *>,CSimpleArrayStandardCompareHelper<COSTaskCompletion::TaskContext *>,CSimpleArrayStandardMergeHelper<COSTaskCompletion::TaskContext *> >::RemoveAt+0×9f
07731ce8  747d3bea SHCore!CTSimpleArray<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener>,4294967294, CTPolicyCoTaskMem<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> >,CSimpleArrayStandardCompareHelper<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> >,CSimpleArrayStandardMergeHelper<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> > >::_Add<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> const &>+0×120
07731cec  75423744 kernel32!BaseThreadInitThunk+0×24
07731cf0  776d9e54 ntdll!__RtlUserThreadStart+0×2f
07731cf4  776d9e1f ntdll!_RtlUserThreadStart+0×1b

0:008> !error 80004001
Error code: (HRESULT) 0×80004001 (2147500033) - Not implemented

0:008> dps 0×077d5d4c L3c
077d5d4c  66f42bbe Windows_UI_Xaml!DirectUI::MetadataAPI::ImportClassInfo+0×2ed89e
077d5d50  66e451ce Windows_UI_Xaml!DirectUI::MetadataAPI::ImportClassInfoFromMetadataProvider+0×86
077d5d54  66c87bf9 Windows_UI_Xaml!DirectUI::MetadataAPI::GetClassInfoByTypeName+0×219
077d5d58  66c55ea0 Windows_UI_Xaml!DirectUI::MetadataAPI::GetClassInfoByXamlType+0×50
077d5d5c  66c5365d Windows_UI_Xaml!DirectUI::MetadataAPI::ImportPropertyInfo+0×8a
077d5d60  66c542a0 Windows_UI_Xaml!DirectUI::MetadataAPI::TryImportPropertyInfoFromMetadataProvider+0×9f
077d5d64  66c53f94 Windows_UI_Xaml!DirectUI::MetadataAPI::TryGetPropertyByName+0xde
077d5d68  66c564e7 Windows_UI_Xaml!XamlManagedTypeInfoProvider::ResolvePropertyName+0×47
077d5d6c  66c930c2 Windows_UI_Xaml!XamlType::GetProperty+0×1a2
077d5d70  66e3a830 Windows_UI_Xaml!XamlBinaryMetadataReader2::LoadProperty+0xb1
077d5d74  66cab9d9 Windows_UI_Xaml!XamlBinaryMetadataReader2::GetProperty+0×5c
077d5d78  66d528e0 Windows_UI_Xaml!XamlBinaryFormatSubReader2::ReadXamlProperty+0×1d0
077d5d7c  66d520cc Windows_UI_Xaml!XamlBinaryFormatSubReader2::ReadSetValueConstantNode+0×3c
077d5d80  66d53b55 Windows_UI_Xaml!XamlBinaryFormatSubReader2::TryRead+0×135
077d5d84  66d539ec Windows_UI_Xaml!XamlBinaryFormatSubReader2::TryReadHRESULT+0×3c
077d5d88  66ca329e Windows_UI_Xaml!CParser::LoadXamlCore+0×5ae
077d5d8c  66da83d2 Windows_UI_Xaml!CCoreServices::ParseXamlWithExistingFrameworkRoot+0×100
077d5d90  66da8131 Windows_UI_Xaml!CApplication::LoadComponent+0×261
077d5d94  66da7e56 Windows_UI_Xaml!Application_LoadComponent+0xaa
077d5d98  66da7cfc Windows_UI_Xaml!DirectUI::FrameworkApplication::LoadComponent+0xc2
077d5d9c  66dab49d Windows_UI_Xaml!DirectUI::FrameworkApplicationFactory::LoadComponentWithResourceLocationImpl+0×6a
077d5da0  66dab418 Windows_UI_Xaml!DirectUI::FrameworkApplicationFactory::LoadComponentWithResourceLocation+0×28
077d5da4  6c186b95 Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×1b2515
077d5da8  6c186af1 Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×1b2471
077d5dac  6c199d37 Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×1c56b7
077d5db0  6c366cd8 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xaf038
077d5db4  6c366c8d Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xaefed
077d5db8  6c366b4d Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xaeead
077d5dbc  6c365b50 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xadeb0
077d5dc0  6c3cc1d7 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0×114537
077d5dc4  6c3cbb9a Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0×113efa
077d5dc8  6c3cb9c0 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0×113d20
077d5dcc  6c35a658 Microsoft_Msn_Weather_6bb10000!DllGetActivationFactory+0xa29b8
077d5dd0  6c0fd56e Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×128eee
077d5dd4  6c21e3d5 Microsoft_Msn_Weather_6bb10000!RHBinder__ShimExeMain+0×249d55
077d5dd8  66e50d37 Windows_UI_Xaml!DirectUI::FrameworkApplicationGenerated::OnLaunchedProtected+0×47
077d5ddc  66e509d0 Windows_UI_Xaml!DirectUI::FrameworkView::OnActivated+0×240
077d5de0  66cf4b28 Windows_UI_Xaml!`Microsoft::WRL::Callback< Windows::Foundation::ITypedEventHandler< Windows::UI::Core::CoreWindow *,IInspectable *>,DirectUI::Page,Windows::UI::Core::ICoreWindow *,IInspectable *>’::`2′::ComObject::Invoke+0×28
077d5de4  7035361e twinapi_appcore!Microsoft::WRL::InvokeTraits<-2>::InvokeDelegates< <lambda_45dc3980e5a5ff53a9eb289d8a61b7e3>, Windows::Foundation::ITypedEventHandler< Windows::ApplicationModel::Core::CoreApplicationView *,Windows::ApplicationModel::Activation::IActivatedEventArgs *> >+0×4f
077d5de8  70353493 twinapi_appcore!Microsoft::WRL::EventSource< Windows::Foundation::ITypedEventHandler< Windows::ApplicationModel::Core::CoreApplicationView *,Windows::ApplicationModel::Activation::IActivatedEventArgs *>,Microsoft::WRL::InvokeModeOptions<-2> >::DoInvoke< <lambda_45dc3980e5a5ff53a9eb289d8a61b7e3> >+0×42
077d5dec  70342a06 twinapi_appcore!Windows::ApplicationModel::Core::CoreApplicationView::Activate+0×296
077d5df0  77425ebc rpcrt4!Invoke+0×34
077d5df4  773f37e3 rpcrt4!NdrStubCall2+0×2e3
077d5df8  74efc1ce combase!CStdStubBuffer_Invoke+0xde
077d5dfc  7742364c rpcrt4!CStdStubBuffer_Invoke+0×2c
077d5e00  74fb659b combase!ObjectMethodExceptionHandlingAction< <lambda_adf5d6ba83bff890864fd80ca2bbf1eb> >+0×7b
077d5e04  74f83091 combase!DefaultStubInvoke+0×211
077d5e08  74f8d59e combase!ServerCall::ContextInvoke+0×38e
077d5e0c  74f8ecc5 combase!AppInvoke+0xb75
077d5e10  74f81c8c combase!ComInvokeWithLockAndIPID+0×62c
077d5e14  74f66d72 combase!CComApartment::ASTAHandleMessage+0×2c2
077d5e18  74f65bfa combase!ASTAWaitContext::Wait+0×47a
077d5e1c  74fbb1b4 combase!ASTAWaitInNewContext+0×81
077d5e20  74fbb0ee combase!ASTAThreadWaitForHandles+0×4e
077d5e24  74fbaf1a combase!CoWaitForMultipleHandles+0xaa
077d5e28  7035772d twinapi_appcore!CTSimpleArray<COSTaskCompletion::TaskContext *,4294967294, CTPolicyCoTaskMem<COSTaskCompletion::TaskContext *>,CSimpleArrayStandardCompareHelper<COSTaskCompletion::TaskContext *>,CSimpleArrayStandardMergeHelper<COSTaskCompletion::TaskContext *> >::RemoveAt+0×9f
077d5e2c  747d3bea SHCore!CTSimpleArray<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener>,4294967294, CTPolicyCoTaskMem<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> >,CSimpleArrayStandardCompareHelper<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> >,CSimpleArrayStandardMergeHelper<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> > >::_Add<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> const &>+0×120
077d5e30  75423744 kernel32!BaseThreadInitThunk+0×24
077d5e34  776d9e54 ntdll!__RtlUserThreadStart+0×2f
077d5e38  776d9e1f ntdll!_RtlUserThreadStart+0×1b

0:008> ub 66f42bbe
Windows_UI_Xaml!DirectUI::MetadataAPI::ImportClassInfo+0×2ed87f:
66f42b9f 8bce            mov     ecx,esi
66f42ba1 e8b96ecfff      call    Windows_UI_Xaml!OnFailure<1354> (66c39a5f)
66f42ba6 e92d2bd1ff      jmp     Windows_UI_Xaml!DirectUI::MetadataAPI::ImportClassInfo+0×3b8 (66c556d8)
66f42bab 8bce            mov     ecx,esi
66f42bad e8ad6ecfff      call    Windows_UI_Xaml!OnFailure<1354> (66c39a5f)
66f42bb2 e9212bd1ff      jmp     Windows_UI_Xaml!DirectUI::MetadataAPI::ImportClassInfo+0×3b8 (66c556d8)
66f42bb7 8bce            mov     ecx,esi
66f42bb9 e8a16ecfff      call    Windows_UI_Xaml!OnFailure<1354> (66c39a5f)

0:008> !error 80004005
Error code: (HRESULT) 0×80004005 (2147500037) - Unspecified error

0:008> dps 0×0772df74 L19
0772df74  66bc3963 Windows_UI_Xaml!CWindowsServices::CreatePalAppChromeProxy+0×23
0772df78  66bc3912 Windows_UI_Xaml!AppChromeProxy::EnsureAppChromeProxy+0×44
0772df7c  66bc38b6 Windows_UI_Xaml!AppChromeProxy::SetThemeOverride+0xf
0772df80  66e51074 Windows_UI_Xaml!DirectUI::DXamlCore::ConfigureCoreWindow+0×95
0772df84  66e50e6f Windows_UI_Xaml!DirectUI::FrameworkView::SetWindow+0×1f
0772df88  70342def twinapi_appcore!Windows::ApplicationModel::Core::CoreApplicationView::SetWindowAndGetDispatcher + 0×20f
0772df8c  77425ebc rpcrt4!Invoke+0×34
0772df90  773f37e3 rpcrt4!NdrStubCall2+0×2e3
0772df94  74efc1ce combase!CStdStubBuffer_Invoke+0xde
0772df98  7742364c rpcrt4!CStdStubBuffer_Invoke+0×2c
0772df9c  74fb659b combase!ObjectMethodExceptionHandlingAction<
<lambda_adf5d6ba83bff890864fd80ca2bbf1eb> >+0×7b
0772dfa0 74f83091 combase!DefaultStubInvoke+0×211
0772dfa4 74f8d59e combase!ServerCall::ContextInvoke+0×38e
0772dfa8 74f8ecc5 combase!AppInvoke+0xb75
0772dfac 74f81c8c combase!ComInvokeWithLockAndIPID+0×62c
0772dfb0 74f66d72 combase!CComApartment::ASTAHandleMessage+0×2c2
0772dfb4 74f65bfa combase!ASTAWaitContext::Wait+0×47a
0772dfb8 74fbb1b4 combase!ASTAWaitInNewContext+0×81
0772dfbc 74fbb0ee combase!ASTAThreadWaitForHandles+0×4e
0772dfc0 74fbaf1a combase!CoWaitForMultipleHandles+0xaa
0772dfc4 7035772d twinapi_appcore!CTSimpleArray<COSTaskCompletion::TaskContext *,4294967294, CTPolicyCoTaskMem<COSTaskCompletion::TaskContext *>,CSimpleArrayStandardCompareHelper<COSTaskCompletion::TaskContext *>,CSimpleArrayStandardMergeHelper<COSTaskCompletion::TaskContext *> >::RemoveAt+0×9f
0772dfc8 747d3bea SHCore!CTSimpleArray<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener>,4294967294, CTPolicyCoTaskMem<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> >,CSimpleArrayStandardCompareHelper<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> >,CSimpleArrayStandardMergeHelper<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> > >::_Add<Microsoft::WRL::ComPtr< IWindowMonitorChangeListener> const &>+0×120
0772dfcc 75423744 kernel32!BaseThreadInitThunk+0×24
0772dfd0 776d9e54 ntdll!__RtlUserThreadStart+0×2f
0772dfd4 776d9e1f ntdll!_RtlUserThreadStart+0×1b

0:008> ub 66bc3963
Windows_UI_Xaml!CWindowsServices::CreatePalAppChromeProxy+0×5:
66bc3945 56              push    esi
66bc3946 8b7508          mov     esi,dword ptr [ebp+8]
66bc3949 832600          and     dword ptr [esi],0
66bc394c e83ebd2d00      call    Windows_UI_Xaml!IsConfigureStatusBarDefaultsPresent (66e9f68f)
66bc3951 84c0            test    al,al
66bc3953 0f856c1a4100    jne     Windows_UI_Xaml!CWindowsServices::CreatePalAppChromeProxy+0×411a85 (66fd53c5)
66bc3959 be05400080      mov     esi,80004005h
66bc395e e80c000000      call    Windows_UI_Xaml!OnFailure<91> (66bc396f)

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

After almost a 6-year break we resume extending the catalog of structural memory patterns as a foundation of pattern-oriented memory forensics, root cause analysis, and software internals. The next pattern we add is borrowed from archaeology and is called Region Strata. When we have several memory snapshots we can analyze Memory Regions for their changes at different times:

For example, we got two memory dumps of the same process saved by WER with approx. one minute difference. We saved the raw stack region of the current thread using .writemem WinDbg command in text files (Strata1.txt and Strata2.txt) and then combined them into one Excel table (Strata.xlsx). In that table we can see memory region changes as shown in this picture (click to expand):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Windows imposes a restriction for the number of GDI handles per process and by default it is approx. 10,000. If this number is reached we can have abnormal software behaviour symptoms such as hangs, glitches in visual appearance, and out-of-memory exceptions resulted in error messages and crashes. We already documented this analysis pattern for kernel and complete memory dumps that we called Handle Limit (GDI). However, one of Software Diagnostics Services’ training customers reported an out-of-memory exception with trace analysis diagnostics pointing to 10,000 leaked GDI objects. The process memory dump was saved and the customer asked whether it was possible to analyze it or similar memory dumps to find out from the dump itself whether we have GDI leak and what GDI objects were involved.

We recreated one of the applications from the kernel pattern variant (the one that leaks fonts) with one modification that it just stops after 10,000 font creation attempts. After launch we tried to open About dialog box but the whole application became unresponsive and no dialog box was visible. We save the process memory dump ad found out that its Main Thread was inside Dialog Box processing:

0:000> kc
# Call Site
00 user32!NtUserWaitMessage
01 user32!DialogBox2
02 user32!InternalDialogBox
03 user32!DialogBoxIndirectParamAorW
04 user32!DialogBoxParamW
05 GUIHandleLeak!WndProc
06 user32!UserCallWinProcCheckWow
07 user32!DispatchMessageWorker
08 GUIHandleLeak!wWinMain
09 GUIHandleLeak!invoke_main
0a GUIHandleLeak!__scrt_common_main_seh
0b kernel32!BaseThreadInitThunk
0c ntdll!RtlUserThreadStart

In order to look at GDI handle table we studied the relevant chapters in Feng Yuan’s book “Windows Graphics Programming” and this post which has all necessary structure definitions: http://stackoverflow.com/questions/13905661/how-to-get-list-of-gdi-handles

We get the current process GDI table address from the disassembly:

0:000> .asm no_code_bytes
Assembly options: no_code_bytes

0:000> uf gdi32!GdiQueryTable
gdi32!GdiQueryTable:
00007ffc`7f172610 sub     rsp,38h
00007ffc`7f172614 or      qword ptr [rsp+20h],0FFFFFFFFFFFFFFFFh
00007ffc`7f17261a lea     rdx,[rsp+20h]
00007ffc`7f17261f mov     ecx,0Eh
00007ffc`7f172624 mov     byte ptr [rsp+28h],0
00007ffc`7f172629 call    qword ptr [gdi32!_imp_NtVdmControl (00007ffc`7f1ba5a8)]
00007ffc`7f17262f test    eax,eax
00007ffc`7f172631 js      gdi32!GdiQueryTable+0x33 (00007ffc`7f172643)  Branch

gdi32!GdiQueryTable+0x23:
00007ffc`7f172633 cmp     byte ptr [rsp+28h],0
00007ffc`7f172638 je      gdi32!GdiQueryTable+0x33 (00007ffc`7f172643)  Branch

gdi32!GdiQueryTable+0x2a:
00007ffc`7f17263a mov     rax,qword ptr [gdi32!pGdiSharedHandleTable (00007ffc`7f2541b8)]
00007ffc`7f172641 jmp     gdi32!GdiQueryTable+0×35 (00007ffc`7f172645)  Branch

gdi32!GdiQueryTable+0x33:
00007ffc`7f172643 xor     eax,eax

gdi32!GdiQueryTable+0x35:
00007ffc`7f172645 add     rsp,38h
00007ffc`7f172649 ret

0:000> dp 00007ffc`7f2541b8 L1
00007ffc`7f2541b8  000000db`56a50000

We dump the first 0×1000 qword values:

0:000> dq 000000db`56a50000 L1000
000000db`56a50000  00000000`00000000 40000000`00000000
000000db`56a50010  00000000`00000000 00000000`00000000
000000db`56a50020  40000000`00000000 00000000`00000000
000000db`56a50030  00000000`00000000 00000000`00000000
000000db`56a50040  00000000`00000000 00000000`00000000
000000db`56a50050  00000000`00000000 00000000`00000000
000000db`56a50060  00000000`00000000 00000000`00000000
000000db`56a50070  00000000`00000000 00000000`00000000
000000db`56a50080  00000000`00000000 00000000`00000000
000000db`56a50090  00000000`00000000 00000000`00000000
000000db`56a500a0  00000000`00000000 00000000`00000000
000000db`56a500b0  00000000`00000000 00000000`00000000
000000db`56a500c0  00000000`00000000 00000000`00000000
000000db`56a500d0  00000000`00000000 00000000`00000000
000000db`56a500e0  00000000`00000000 00000000`00000000
000000db`56a500f0  fffff901`40000e60 40040104`00000000
000000db`56a50100  00000000`00000000 fffff901`40000d60
000000db`56a50110  40080188`00000000 00000000`00000000
000000db`56a50120  fffff901`400008b0 40080108`00000000
000000db`56a50130  00000000`00000000 fffff901`400007c0
000000db`56a50140  40080108`00000000 00000000`00000000
000000db`56a50150  fffff901`400006d0 40080108`00000000
[…]
000000db`56a57ce0 fffff901`429d24f0 400aee0a`00002c30
000000db`56a57cf0 000000db`564f3b20 fffff901`42910570
000000db`56a57d00 400a360a`00002c30 000000db`564e57b0
000000db`56a57d10 fffff901`40700420 40105310`000002b4
000000db`56a57d20 00000089`39410fc0 fffff901`407ec4a0
000000db`56a57d30 400a010a`000002b4 00000089`3900ae70
000000db`56a57d40 fffff901`407036d0 400a010a`000002b4
000000db`56a57d50 00000089`3900ae60 fffff901`440b56e0
000000db`56a57d60 400a030a`00002c30 000000db`564f0e90
000000db`56a57d70 fffff901`43e7fd50 40040104`00000000
000000db`56a57d80 00000000`00000000 fffff901`42c0f010
000000db`56a57d90 400a4b0a`00003190 0000003a`1a30b670
000000db`56a57da0 fffff901`440deaf0 400a6d0a`00002c30
000000db`56a57db0 000000db`564f3b00 fffff901`407f2010
000000db`56a57dc0 40100510`00001704 000000d8`d6230f60
000000db`56a57dd0 fffff901`40714180 40100210`00001704
000000db`56a57de0 000000d8`d6230f48 fffff901`4009d840
000000db`56a57df0 40100210`00001704 000000d8`d6230f78
000000db`56a57e00 fffff901`43e50950 40100230`00001704
000000db`56a57e10 00000000`00000000 fffff901`43e30010
000000db`56a57e20 40100230`00001704 00000000`00000000
000000db`56a57e30 fffff901`44f1d010 4005a105`0000168c
000000db`56a57e40 00000000`00000000 fffff901`440ded80
000000db`56a57e50 400a2e0a`00002c30 000000db`564f3b10
000000db`56a57e60 fffff901`4070b3b0 40050405`00000000
000000db`56a57e70 00000000`00000000 fffff901`42a0a010
000000db`56a57e80 400a870a`00002c30 000000db`564e7160
000000db`56a57e90 fffff901`407a7450 4008cd08`00000000
000000db`56a57ea0 00000000`00000000 fffff901`400c49c0
000000db`56a57eb0 40046904`000002b4 00000089`39410f90
000000db`56a57ec0 fffff901`41fb8010 4005c705`00000000
000000db`56a57ed0 00000000`00000000 fffff901`423dc790
000000db`56a57ee0 40086708`00000000 00000000`00000000
000000db`56a57ef0 fffff901`40699620 40010301`0000168c
000000db`56a57f00 00000000`01100000 fffff901`43e54510
000000db`56a57f10 40050305`0000168c 00000000`00000000
000000db`56a57f20 fffff901`407164c0 40100610`00001448
000000db`56a57f30 000000fe`4d8d0cf0 fffff901`407eee50
000000db`56a57f40 40100410`00001448 000000fe`4d8d0cd8
000000db`56a57f50 fffff901`43e2abb0 40080508`00000000
000000db`56a57f60 00000000`00000000 fffff901`40715010
000000db`56a57f70 40050305`0000168c 00000000`00000000
000000db`56a57f80 fffff901`42872b80 40084e08`00000000
000000db`56a57f90 00000000`00000000 fffff901`407175a0
000000db`56a57fa0 410f080f`00000000 00000000`00000000
000000db`56a57fb0 fffff901`407f4000 40050605`00000000
000000db`56a57fc0 00000000`00000000 fffff901`406d6bb0
000000db`56a57fd0 40080508`00000000 00000000`00000000
000000db`56a57fe0 fffff901`43e3a4c0 40120812`00000000
000000db`56a57ff0 00000000`00000000 fffff901`44fe1010

We see that typical cell value has 3 qwords (8-byte or 4-word sized values for both x64 and Virtualized Process). The non-zeroed data starts from 000000db`56a500f0 address. Clearly some entries have wProcessId equal to the PID from our dump:

0:000> ~
.  0  Id: 2c30.292c Suspend: 0 Teb: 00007ff7`1bf3e000 Unfrozen

Let’s look at one of such entries (the first and the last qword values are pointers):

0:000> dq 000000db`56a57da0 L3
000000db`56a57da0  fffff901`440deaf0 400a6d0a`00002c30
000000db`56a57db0  000000db`564f3b00

0:000> dw 000000db`56a57da0 L3*4
000000db`56a57da0  eaf0 440d f901 ffff 2c30 0000 6d0a 400a
000000db`56a57db0  3b00 564f 00db 0000

Applying 7f to wType word 400a gives us 0xa or 10 which is a font handle:

0:000> ? 400a & 7f
Evaluate expression: 10 = 00000000`0000000a

Since there are entries from other processes from the same session in this table assessing the handle leak visually is difficult so we wrote a WinDbg script that goes from the first non-zero pKernelAddress ($t0) to the first zero entry and for the given wProcessId ($tpid) counts the number of entries ($t1) and the number of entries ($t3) for the given wType ($t2). The script also counts the total entries till the first zero one ($t4):

.while (qwo(@$t0)) { .if (wo(@$t0+8) == @$tpid) {r $t1 = @$t1+1; .if (((qwo(@$t0+8) >> 0n48) & 7f) == @$t2) {r $t3 = @$t3+1} }; r $t0 = @$t0+3*8; r $t4 = @$t4+1}

To execute it we prepare the variables:

0:000> r $t0=000000db`56a500f0

0:000> r $t1=0

0:000> r $t2=a

0:000> r $t3=0

0:000> r $t4=0

0:000> .while (qwo(@$t0)) { .if (wo(@$t0+8) == @$tpid) {r $t1 = @$t1+1; .if (((qwo(@$t0+8) >> 0n48) & 7f) == @$t2) {r $t3 = @$t3+1} }; r $t0 = @$t0+3*8; r $t4 = @$t4+1}

After execution we get the modified variables that show us that the total consecutive non-zero handle table entries is 21464, the total number of entries for the current process is 9990, and the total number of fonts is 9982:

0:000> ? $t0
Evaluate expression: 942052007216 = 000000db`56acdd30

0:000> ? $t1
Evaluate expression: 9990 = 00000000`00002706

0:000> ? $t2
Evaluate expression: 10 = 00000000`0000000a

0:000> ? $t3
Evaluate expression: 9982 = 00000000`000026fe

0:000> ? $t4
Evaluate expression: 21464 = 00000000`000053d8

If we repeat the same script for device contexts (wType is 1) we get only 2 entries for our PID:

0:000> r $t0=000000db`56a500f0

0:000> r $t1=0

0:000> r $t2=1

0:000> r $t3=0

0:000> r $t4=0

0:000> .while (qwo(@$t0)) { .if (wo(@$t0+8) == @$tpid) {r $t1 = @$t1+1; .if (((qwo(@$t0+8) >> 0n48)  & 7f) == @$t2) {r $t3 = @$t3+1} }; r $t0 = @$t0+3*8; r $t4 = @$t4+1}

0:000> ? $t3
Evaluate expression: 2 = 00000000`00000002

Of course, this script may be further improved, for example, to process all possible wType values and print their statistics. It can also be made as a textual WinDbg script procedure with arguments.

We rename the original pattern variant to Handle Limit (GDI, kernel space) and name this pattern variant as Handle Limit (GDI, user space).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -