Software Diagnostics Library » Crash Dump Analysis

Crash Dump Analysis Patterns (Part 241)

Sunday, May 29th, 2016

Most Execution Residue traces in memory dumps are not explicitly temporal (see Special and General Trace and Log Analysis) but may be ordered by some space coordinate, such as memory addresses or page frame numbers. Furthermore, virtual space can be further subdivided into places such as modules and physical space may be restructured into places such as processes. Simple space trace of some data value can be constructed using Value References analysis pattern. These and higher structural space trace constructs can be named as a general Place Trace analysis pattern illustrated in this diagram:

Memory attributes, such as page protection, or derived attributes from memory contents can also be considered as Place Trace data. Sometimes, time ordering can be reconstructed by looking at time information for place containers, for example, elapsed process time or ordering in the process list, or thread order and times for stack region thread owners.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Memory Dump Analysis and History, Memory Forensics, Software Trace Analysis | 1 Comment »

Crash Dump Analysis Patterns (Part 240)

Sunday, May 29th, 2016

Windows processes may contain Execution Residue such as ASCII window class names in mapped memory regions pointing to other running processes (perhaps as a result of Hooksware). For example, calc.exe process memory dump saved on my Windows 10 notebook “knows” about Visio and WinDbg windows that were opened at that time:

0:000> s-a 0 L?FFFFFFFFFFFFFFFF "VISIOA" 00000015`42c6bdd0 56 49 53 49 4f 41 00 00-00 00 00 00 00 00 00 00 VISIOA.............

0:000> s-a 0 L?FFFFFFFFFFFFFFFF "WinDbg" 00000015`42d19720 57 69 6e 44 62 67 46 72-61 6d 65 43 6c 61 73 73 WinDbgFrameClass

This may be useful for some troubleshooting scenarios, for example, pointing to processes which are known for their problematic behavior or Special Processes. Of course, we assume that those windows or classes were genuine, not faked. We call this analysis pattern Window Hint similar to Environment Hint and Module Hint analysis patterns.

Going deeper, we can dump strings from the whole region limiting the output to the strings with length more than 5:

0:000> !address 00000015`42d19720

Usage: <unknown> Base Address: 00000015`42b20000 End Address: 00000015`42d3a000 Region Size: 00000000`0021a000 ( 2.102 MB) State: 00001000 MEM_COMMIT Protect: 00000002 PAGE_READONLY Type: 00040000 MEM_MAPPED Allocation Base: 00000015`42b20000 Allocation Protect: 00000002 PAGE_READONLY

Content source: 1 (target), length: 208e0

0:000> s-[l5]sa 00000015`42b20000 00000015`42d3a000 00000015`42b20a60 “#32769″ 00000015`42b20cc0 “Message” 00000015`42b20f40 “#32774″ 00000015`42b21060 “#32772″ 00000015`42b21510 “Ghost” 00000015`42b215e0 “LivePreview” 00000015`42b216f0 “UserAdapterWindowClass” 00000015`42b21ce0 “MSCTFIME Composition” 00000015`42b222a0 “#32772″ 00000015`42b22390 “#32772″ 00000015`42b22460 “RichEdit20W” 00000015`42b22530 “RichEdit20A” 00000015`42b22600 “ToolbarWindow32″ 00000015`42b226e0 “tooltips_class32″ 00000015`42b227c0 “msctls_statusbar32″ 00000015`42b228a0 “SysListView32″ 00000015`42b22980 “SysHeader32″ 00000015`42b22a50 “SysTabControl32″ 00000015`42b22b30 “SysTreeView32″ 00000015`42b22c10 “msctls_trackbar32″ 00000015`42b22cf0 “msctls_updown32″ 00000015`42b22dd0 “msctls_progress32″ 00000015`42b22eb0 “msctls_hotkey32″ 00000015`42b22f8f “‘SysAnimate32″ 00000015`42b230f0 “SysIPAddress32″ 00000015`42b231d0 “ReBarWindow32″ 00000015`42b232b0 “ComboBoxEx32″ 00000015`42b23390 “SysMonthCal32″ 00000015`42b23470 “SysDateTimePick32″ 00000015`42b23550 “DropDown” 00000015`42b23620 “SysLink” 00000015`42b236f0 “SysPager” 00000015`42b23960 “msctls_netaddress”

[...]

00000015`42d175e0 "OutlookFbThreadWnd" 00000015`42d19720 "WinDbgFrameClass" 00000015`42d19750 "DockClass" 00000015`42d19770 "GhostClass" 00000015`42d19a30 "ATL:00007FF60D792730" 00000015`42d1a0f0 "MSCTFIME Composition" 00000015`42d1a4af "%OleMainThreadWndClass" 00000015`42d1be10 "CicMarshalWndClass" 00000015`42d1c0e0 "VSyncHelper-00000040EC4CA5F0-1f8" 00000015`42d1c100 "8855daf" 00000015`42d1c190 "URL Moniker Notification Window" 00000015`42d1c390 "UserAdapterWindowClass" 00000015`42d1d080 "@>zG#" 00000015`42d1dcaf "!VSyncHelper-00000040D60C5850-1e" 00000015`42d1dccf "ef0477df" 00000015`42d20d50 "VSyncHelper-00000040F39C5650-1f0" 00000015`42d20d70 "313c5a0" 00000015`42d250d0 "#32770" 00000015`42d250f0 "URL Moniker Notification Window" 00000015`42d29270 "VSyncHelper-00000079321C32E0-1f2" 00000015`42d29290 "fb11f8c" 00000015`42d2a1d0 "MSCTFIME Composition" 00000015`42d2a480 "CicMarshalWndClass" 00000015`42d2ac80 "MSCTFIME Composition" 00000015`42d2b8d0 "ShockwaveFlashFullScreen" 00000015`42d2bbb8 "P?U!\" 00000015`42d2c690 "Xaml_WindowedPopupClass" 00000015`42d30a10 "ShockwaveFlashFullScreen" 00000015`42d30b50 "MSCTFIME UI" 00000015`42d30b90 "WinBaseClass" 00000015`42d3441f "!Alternate Owner" 00000015`42d34460 "ShockwaveFlashFullScreen" 00000015`42d344a0 "ATL:00007FF60D792530" 00000015`42d34a50 "SysAnimate32" 00000015`42d34a7f "'ComboBoxEx32" 00000015`42d34ed0 "tooltips_class32" 00000015`42d34f00 "msctls_statusbar32" 00000015`42d35e70 "RawInputClass" 00000015`42d36a10 "SysTabControl32" 00000015`42d38650 "CicMarshalWndClass" 00000015`42d38eb0 "#32772" 00000015`42d3951f "!VSyncHelper-000000C9DA06CD10-1f" 00000015`42d3953f "110e8d16"

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Memory Forensics, Windows 10, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 239)

Monday, May 2nd, 2016

We found out that in Windows 10 (at least on our working system) Notepad is no longer a single threaded application even without opening any common dialogs (like in Evental Dumps analysis pattern example). It has at least 3 additional threads (and other modeling applications we use for our training also have additional threads):

0:000> ~*k

0 Id: 3a64.3b38 Suspend: 1 Teb: 00007ff6`a914d000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`6298f938 00007ffa`e57cf8e5 USER32!NtUserGetMessage+0xa 01 000000e5`6298f940 00007ff6`a9603470 USER32!GetMessageW+0x25 02 000000e5`6298f970 00007ff6`a96141f5 notepad!WinMain+0x178 03 000000e5`6298f9f0 00007ffa`e3b42d92 notepad!WinMainCRTStartup+0x1c5 04 000000e5`6298fab0 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 05 000000e5`6298fae0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

1 Id: 3a64.38b0 Suspend: 1 Teb: 00007ff6`a914b000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`62bffa58 00007ffa`e5bf93a5 ntdll!NtWaitForWorkViaWorkerFactory+0xa 01 000000e5`62bffa60 00007ffa`e3b42d92 ntdll!TppWorkerThread+0x295 02 000000e5`62bffe60 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 03 000000e5`62bffe90 00000000`00000000 ntdll!RtlUserThreadStart+0x34

2 Id: 3a64.3940 Suspend: 1 Teb: 00007ff6`a9149000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`62c7f718 00007ffa`e5bf93a5 ntdll!NtWaitForWorkViaWorkerFactory+0xa 01 000000e5`62c7f720 00007ffa`e3b42d92 ntdll!TppWorkerThread+0x295 02 000000e5`62c7fb20 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 03 000000e5`62c7fb50 00000000`00000000 ntdll!RtlUserThreadStart+0x34

3 Id: 3a64.1030 Suspend: 1 Teb: 00007ff6`a9147000 Unfrozen # Child-SP RetAddr Call Site 00 000000e5`62d1f878 00007ffa`e5bf93a5 ntdll!NtWaitForWorkViaWorkerFactory+0xa 01 000000e5`62d1f880 00007ffa`e3b42d92 ntdll!TppWorkerThread+0x295 02 000000e5`62d1fc80 00007ffa`e5bc9f64 KERNEL32!BaseThreadInitThunk+0x22 03 000000e5`62d1fcb0 00000000`00000000 ntdll!RtlUserThreadStart+0x34

This gave us an idea for the analysis pattern we call Not My Thread since additional threads can be started by any other process DLLs, for example, by Hooksware. However, we need to distinguish between unexpectedly added threads, threads with Special Stack Traces and Special Threads, for example, from .NET support.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Windows 10, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 238)

Thursday, March 17th, 2016

Sometimes developers introduce their own variants of synchronization code instead of using synchronization API provided by language runtime and OS. If we are lucky we can spot it in function and class method names and then use Constant Subtrace analysis pattern:

0: kd> kc *** Stack trace for last set context - .thread/.cxr resets it # Call Site 00 nt!KiSwapContext 01 nt!KiCommitThreadWait 02 nt!KeWaitForSingleObject 03 nt!NtWaitForSingleObject 04 nt!KiSystemServiceCopyEnd 05 ntdll!ZwWaitForSingleObject 06 KERNELBASE!WaitForSingleObjectEx 07 wbemcomn!CWbemCriticalSection::Enter 08 wbemcore!EnsureInitialized 09 wbemcore!InitAndWaitForClient 0a wbemcore!CWbemLevel1Login::ConnectorLogin 0b wbemcore!CWbemLevel1Login::NTLMLogin 0c RPCRT4!Invoke 0d RPCRT4!NdrStubCall2 0e ole32!CStdStubBuffer_Invoke 0f ole32!SyncStubInvoke 10 ole32!StubInvoke 11 ole32!CCtxComChnl::ContextInvoke 12 ole32!AppInvoke 13 ole32!ComInvokeWithLockAndIPID 14 ole32!ThreadInvoke 15 RPCRT4!DispatchToStubInCNoAvrf 16 RPCRT4!RPC_INTERFACE::DispatchToStubWorker 17 RPCRT4!RPC_INTERFACE::DispatchToStub 18 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject 19 RPCRT4!LRPC_SCALL::DispatchRequest 1a RPCRT4!LRPC_SCALL::HandleRequest 1b RPCRT4!LRPC_SASSOCIATION::HandleRequest 1c RPCRT4!LRPC_ADDRESS::HandleRequest 1d RPCRT4!LRPC_ADDRESS::ProcessIO 1e RPCRT4!LrpcIoComplete 1f ntdll!TppAlpcpExecuteCallback 20 ntdll!TppWorkerThread 21 kernel32!BaseThreadInitThunk 22 ntdll!RtlUserThreadStart

0: kd> kc *** Stack trace for last set context - .thread/.cxr resets it # Call Site 00 repdrvfs!SCachePage::operator= 01 repdrvfs!std::vector<scachepage,wbem_allocator<scachepage> >::erase 02 repdrvfs!CPageCache::Read 03 repdrvfs!CPageFile::GetPage 04 repdrvfs!ValidateBTreeAgainstObjHeap 05 repdrvfs!PerformAllValidations 06 repdrvfs!VerifyRepositoryOnline 07 repdrvfs!VerifyRepository 08 repdrvfs!CPageSource::Startup 09 repdrvfs!CPageSource::Init 0a repdrvfs!CFileCache::InnerInitialize 0b repdrvfs!CFileCache::Initialize 0c repdrvfs!CRepository::Initialize 0d repdrvfs!CRepository::Logon 0e wbemcore!CRepository::Init 0f wbemcore!InitSubsystems 10 wbemcore!ConfigMgr::InitSystem 11 wbemcore!EnsureInitialized 12 wbemcore!InitAndWaitForClient 13 wbemcore!CWbemLevel1Login::ConnectorLogin 14 wbemcore!CWbemLevel1Login::NTLMLogin 15 RPCRT4!Invoke 16 RPCRT4!NdrStubCall2 17 ole32!CStdStubBuffer_Invoke 18 ole32!SyncStubInvoke 19 ole32!StubInvoke 1a ole32!CCtxComChnl::ContextInvoke 1b ole32!AppInvoke 1c ole32!ComInvokeWithLockAndIPID 1d ole32!ThreadInvoke 1e RPCRT4!DispatchToStubInCNoAvrf 1f RPCRT4!RPC_INTERFACE::DispatchToStubWorker 20 RPCRT4!RPC_INTERFACE::DispatchToStub 21 RPCRT4!RPC_INTERFACE::DispatchToStubWithObject 22 RPCRT4!LRPC_SCALL::DispatchRequest 23 RPCRT4!LRPC_SCALL::HandleRequest 24 RPCRT4!LRPC_SASSOCIATION::HandleRequest 25 RPCRT4!LRPC_ADDRESS::HandleRequest 26 RPCRT4!LRPC_ADDRESS::ProcessIO 27 RPCRT4!LrpcIoComplete 28 ntdll!TppAlpcpExecuteCallback 29 ntdll!TppWorkerThread 2a kernel32!BaseThreadInitThunk 2b ntdll!RtlUserThreadStart

These two thread stack traces were spotted from a complete memory dump Stack Trace Collection as the part of a larger ALPC Wait Chain. We switched to these threads using .thread /r /p WinDbg command to get the stripped stack trace via kc command for better illustration. We see Constant Subtrace until wbemcore!EnsureInitialized function which serves as a bifurcation stack frame. The first stack trace has “CriticalSection::Enter” after the bifurcation stack frame compared to the second stack trace which looks like Spiking Thread in user space.

There is no hidden critical section associated with that process except the one which is probably related to the spiking Variable Subtrace since it doesn’t have any LockCount:

0: kd> !cs -l -o -s DebugInfo = 0x0000000004a774d0 Critical section = 0x000000000308d690 (+0x308D690) LOCKED LockCount = 0×0 WaiterWoken = No OwningThread = 0×0000000000000928 RecursionCount = 0×1 LockSemaphore = 0×0 SpinCount = 0×0000000000000000 OwningThread = .thread fffffa806ebd8060 ntdll!RtlpStackTraceDataBase is NULL. Probably the stack traces are not enabled

We can also disassemble wbemcomn!CWbemCriticalSection::Enter and find out that it calls WaitForSingleObject once and no other synchronization API indeed:

0: kd> uf wbemcomn!CWbemCriticalSection::Enter [...] wbemcomn!CWbemCriticalSection::Enter+0x41: 000007fe`f78ad1c0 mov rcx,qword ptr [rbx+10h] 000007fe`f78ad1c4 mov edx,r12d 000007fe`f78ad1c7 call qword ptr [wbemcomn!_imp_WaitForSingleObject (000007fe`f78ee4f0)] 000007fe`f78ad1cd cmp eax,esi 000007fe`f78ad1cf je wbemcomn!CWbemCriticalSection::Enter+0x52 (000007fe`f78a62a3) Branch

We add this nonstandard synchronization memory analysis pattern to Wait Chain analysis pattern category.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 237)

Thursday, March 17th, 2016

Variable Subtrace analysis pattern was introduced for inter-correlational analysis of CPU spikes across memory snapshots with just one thread involved. In contrast, we found Constant Subtrace pattern useful in Wait Chain analysis involving several threads in just one memory snapshot (intra-correlational analysis). Here a constant subtrace groups stack traces from Stack Trace Collection with a bifurcation stack trace frame (similar to Bifurcation Point trace analysis pattern) providing some wait chain relationship hint. Such subtraces traces may be initially found by the preceding wait chain analysis or by technology-specific subtraces such as ALPC/RPC server thread frames (as seen in an example stack from COM interface invocation). Here is a minimal stack trace diagram (similar to minimal trace graphs introduced in Accelerated Windows Software Trace Analysis training) illustrating the pattern (it also shows Spiking Thread pattern in user space as seen from a complete memory dump):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Log Analysis, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Crash Dump Analysis Patterns (Part 236)

Saturday, February 13th, 2016

When we have a performance issue we may request a set of consecutive memory dump saved after some interval. In such memory dumps we may see the same thread(s) having similar stack trace(s). In this simple diagnostic scenario we may diagnose several patterns based on the stack traces: Active Threads that can be Spiking Threads with Spike Intervals or stable, not changing, Wait Chains. Here we may easily identify Top active and Blocking modules based on Module Wait Chain.

More complex case arises when we have different Active Threads and/or Wait Chains with different thread IDs at different times. However, if their Top Module is the same we may have found it as a performance root cause component especially in the case of Active Threads since it is statistically probable that such threads were active for considerable time deltas around the snapshot times (since threads are usually waiting). Such hypothesis may also be confirmed by Inter-Correlation analysis with software traces and logs where we can see Thread of Activity Discontinuities and Time Deltas.

We call this analysis pattern Diachronic Module since we see the module component appears in different thread stack traces diachronically (at different times). The typical simplified scenario is illustrated in this diagram:

This analysis pattern is different from synchronous module case (the module component appears in different thread stack traces at the same time) which was named Ubiquitous Component.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Software Trace Analysis, Trace Analysis Patterns | No Comments »

Crash Dump Analysis Patterns (Part 234)

Monday, November 30th, 2015

Sometimes we identify a function parameter that caused abnormal behavior, for example, Invalid Pointer access violation and want to see from which module or function it was probably originated:

0:009> kvL # ChildEBP RetAddr Args to Child 00 0381f048 74e715e9 00000002 0381f098 00000001 ntdll!NtWaitForMultipleObjects+0x15 01 0381f0e4 76ce19fc 0381f098 0381f10c 00000000 KERNELBASE!WaitForMultipleObjectsEx+0x100 02 0381f12c 76ce41d8 00000002 7efde000 00000000 kernel32!WaitForMultipleObjectsExImplementation+0xe0 03 0381f148 76d08074 00000002 0381f17c 00000000 kernel32!WaitForMultipleObjects+0x18 04 0381f1b4 76d07f33 0381f294 00000001 00000001 kernel32!WerpReportFaultInternal+0x186 05 0381f1c8 76d07828 0381f294 00000001 0381f264 kernel32!WerpReportFault+0x70 06 0381f1d8 76d077a7 0381f294 00000001 9a131a15 kernel32!BasepReportFault+0x20 07 0381f264 772574ff 00000000 772573dc 00000000 kernel32!UnhandledExceptionFilter+0x1af 08 0381f26c 772573dc 00000000 0381ffd4 7720c550 ntdll!__RtlUserThreadStart+0x62 09 0381f280 77257281 00000000 00000000 00000000 ntdll!_EH4_CallFilterFunc+0x12 0a 0381f2a8 7723b499 fffffffe 0381ffc4 0381f3e4 ntdll!_except_handler4+0x8e 0b 0381f2cc 7723b46b 0381f394 0381ffc4 0381f3e4 ntdll!ExecuteHandler2+0x26 0c 0381f2f0 7723b40e 0381f394 0381ffc4 0381f3e4 ntdll!ExecuteHandler+0x24 0d 0381f37c 771f0133 0181f394 0381f3e4 0381f394 ntdll!RtlDispatchException+0x127 0e 0381f37c 00010101 0181f394 0381f3e4 0381f394 ntdll!KiUserExceptionDispatcher+0xf (CONTEXT @ 0381f3e4) WARNING: Frame IP not in any known module. Following frames may be wrong. 0f 0381f844 00df3d1e 064bf018 0b925970 06501dd8 0×10101 10 0381f860 00ca0a15 064bf018 0b925970 0c86a949 ModuleA!Bar+0×3e 11 0381f888 00ca07d4 064bf018 0b925970 09584580 ModuleB!Foo2+0×75 12 0381f8e4 00ca0ae3 0b925970 00000000 09584008 ModuleB!Foo1+0xb4 […]

Here we backtrack the invalid function call pointer to a function argument parameter:

0:009> .cxr 0381f3e4 eax=06501dd8 ebx=0381f8c4 ecx=0b925970 edx=0b925527 esi=064bf018 edi=0381f898 eip=00010101 esp=0381f848 ebp=0c86a949 iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 00010101 1f pop ds

0:009> k 1 *** Stack trace for last set context - .thread/.cxr resets it # ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 00 0381f844 00df3d1e 0×10101

0:009> ub 00df3d1e ModuleA!Bar+0x2e: 00df3d0a mov ecx,dword ptr [esp+10h] 00df3d0e mov edi,dword ptr [esp+18h] 00df3d12 push edi 00df3d13 push eax 00df3d14 mov edx,dword ptr [ecx] 00df3d16 push ecx 00df3d17 push esi 00df3d18 mov edx,dword ptr [edx+4] 00df3d1b call dword ptr [edx+10h]

The value of EDX comes from EDX+4 which comes from ECX pointer value which was probably one of function parameters:

0:009> dp ecx L1 0b925970 0b925978

0:009> dp poi(ecx)+4 L1 0b92597c 0b925527

0:009> dc 0b925527 0b925527 20202020 80202020 01010101 01010101 ……… 0b925537 00010101 01010101 01010000 01000101 ……………. […]

Since the data came from ModuleA function parameter it was passed from the previous call:

0:009> kL 4 # ChildEBP RetAddr WARNING: Frame IP not in any known module. Following frames may be wrong. 00 0381f844 00df3d1e 0x10101 01 0381f860 00ca0a15 ModuleA!Bar+0×3e 02 0381f888 00ca07d4 ModuleB!Foo2+0×75 03 0381f8e4 00ca0ae3 ModuleB!Foo1+0xb4

0:009> ub 00ca0a15 ModuleB!Foo2+0×61: 00ca0a01 add esp,10h 00ca0a04 and edi,0FFFFFFF8h 00ca0a07 lea edx,[esp+20h] 00ca0a0b push edx 00ca0a0c push ebp 00ca0a0d push edi 00ca0a0e push esi 00ca0a0f call dword ptr [ModuleA!_imp__Bar (00cc32c0)]

The passed parameter looks like the second argument of Bar and ModuleB!Foo2, and the first argument of ModuleB!Foo1:

0:009> dps 0381f860 0381f860 064bf018 0381f864 00ca0a15 ModuleB!Foo2+0x75 ; return address when calling ModuleA!Bar 0381f868 064bf018 0381f86c 0b925970 0381f870 0c86a949 0381f874 0381f898 0381f878 0b925970 0381f87c 09584580 0381f880 064bf018 0381f884 0381f8c4 0381f888 0b925970 0381f88c 00ca07d4 ModuleB!Foo1+0xb4 ; return address when calling ModuleB!Foo2 0381f890 064bf018 0381f894 0b925970 0381f898 09584580 0381f89c 064bf018 0381f8a0 00ccd7d0 0381f8a4 0381f954 0381f8a8 00000000 0381f8ac 00000001 0381f8b0 00000000 0381f8b4 0b926d18 0381f8b8 00000000 0381f8bc 00000005 0381f8c0 00000000 0381f8c4 0b925970 0381f8c8 00000000 0381f8cc 00000000 0381f8d0 00000000 0381f8d4 00000000 0381f8d8 00000000 0381f8dc 00000000 0381f8e0 00000000 0381f8e4 00000000 0381f8e8 00ca0ae3 ModuleB!iFoo0+0×13 ; return address when calling ModuleB!Foo1 0381f8ec 0b925970 0381f8f0 00000000

We see Parameter Flow analysis pattern:

0:009> kvL 4 # ChildEBP RetAddr Args to Child WARNING: Frame IP not in any known module. Following frames may be wrong. 00 0381f844 00df3d1e 064bf018 0b925970 06501dd8 0×10101 01 0381f860 00ca0a15 064bf018 0b925970 0c86a949 ModuleA!Bar+0×3e 02 0381f888 00ca07d4 064bf018 0b925970 09584580 ModuleB!Foo2+0×75 03 0381f8e4 00ca0ae3 0b925970 00000000 09584008 ModuleB!Foo1+0xb4

The example is from x86 Windows where parameters are passed through stack, and, therefore, are seen in the verbose stack trace output (kv). For x64 Windows systems such parameters may be Hidden and what we see in verbose output can be False Function Parameters. However, we can still track in some cases certain values found in Execution Residue and associate them with particular stack trace frames (including Past Stack Traces).

Parameter Flow memory analysis pattern is similar to trace analysis Data Flow pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 233)

Wednesday, November 18th, 2015

With the possibility of process cloning (reflection) starting from Windows 7 it is possible to get memory snapshots (Clone Dump) from a process clone (similar to fork API in Unix). Procdump tool has -r switch for that purpose. We checked this with x64 Windows 7 notepad.exe. We got two memory dumps: one is a clone with this stack trace:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dbgcfg.dmp] User Mini Dump File with Full Memory: Only application data is available

Comment: ' *** procdump -ma -r Notepad.exe *** Manual dump'

0:000> ~*k

. 0 Id: 25ec.147c Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Child-SP RetAddr Call Site 00 00000000`02c8fd38 00000000`7733aae7 ntdll!NtSuspendThread+0xa 01 00000000`02c8fd40 00000000`77165a4d ntdll!RtlpProcessReflectionStartup+0×2e7 02 00000000`02c8fe30 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd 03 00000000`02c8fe60 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

The process memory has all address space of the original process including module list and heap structure:

0:000> lmn start end module name 00000000`77050000 00000000`7714a000 user32 user32.dll 00000000`77150000 00000000`77270000 kernel32 kernel32.dll 00000000`77270000 00000000`77419000 ntdll ntdll.dll 00000000`ff030000 00000000`ff065000 notepad notepad.exe 000007fe`f57d0000 000007fe`f5841000 winspool winspool.drv 000007fe`fb730000 000007fe`fb786000 uxtheme uxtheme.dll 000007fe`fb910000 000007fe`fbb04000 comctl32 comctl32.dll 000007fe`fbf00000 000007fe`fbf0c000 version version.dll 000007fe`fceb0000 000007fe`fcebf000 CRYPTBASE CRYPTBASE.dll 000007fe`fd310000 000007fe`fd37c000 KERNELBASE KERNELBASE.dll 000007fe`fd3d0000 000007fe`fd499000 usp10 usp10.dll 000007fe`fd4a0000 000007fe`fd511000 shlwapi shlwapi.dll 000007fe`fd520000 000007fe`fd64d000 rpcrt4 rpcrt4.dll 000007fe`fd650000 000007fe`fd66f000 sechost sechost.dll 000007fe`fd680000 000007fe`fd6ae000 imm32 imm32.dll 000007fe`fd6b0000 000007fe`fd78b000 advapi32 advapi32.dll 000007fe`fd810000 000007fe`fd8a7000 comdlg32 comdlg32.dll 000007fe`fdd60000 000007fe`fddff000 msvcrt msvcrt.dll 000007fe`fdfe0000 000007fe`fed69000 shell32 shell32.dll 000007fe`fed70000 000007fe`fedd7000 gdi32 gdi32.dll 000007fe`ff0b0000 000007fe`ff1b9000 msctf msctf.dll 000007fe`ff1c0000 000007fe`ff1ce000 lpk lpk.dll 000007fe`ff1d0000 000007fe`ff2a7000 oleaut32 oleaut32.dll 000007fe`ff2b0000 000007fe`ff349000 clbcatq clbcatq.dll 000007fe`ff350000 000007fe`ff553000 ole32 ole32.dll

0:000> !address -summary

Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 48 7ff`faa06000 ( 8.000 TB) 100.00% Image 129 0`01e79000 ( 30.473 MB) 35.47% 0.00% 21 0`01d10000 ( 29.063 MB) 33.83% 0.00% Other 9 0`016be000 ( 22.742 MB) 26.47% 0.00% Heap 26 0`00320000 ( 3.125 MB) 3.64% 0.00% Stack 3 0`00080000 ( 512.000 kB) 0.58% 0.00% TEB 1 0`00002000 ( 8.000 kB) 0.01% 0.00% PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_IMAGE 130 0`01e7a000 ( 30.477 MB) 35.47% 0.00% MEM_PRIVATE 47 0`01449000 ( 20.285 MB) 23.61% 0.00% MEM_MAPPED 11 0`00c97000 ( 12.590 MB) 14.65% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 50 7ff`fc096000 ( 8.000 TB) 100.00% MEM_COMMIT 176 0`02c10000 ( 44.063 MB) 51.29% 0.00% MEM_RESERVE 12 0`0134a000 ( 19.289 MB) 22.45% 0.00%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READONLY 83 0`01b46000 ( 27.273 MB) 31.75% 0.00% PAGE_EXECUTE_READ 25 0`00f6d000 ( 15.426 MB) 17.95% 0.00% PAGE_WRITECOPY 48 0`00126000 ( 1.148 MB) 1.34% 0.00% PAGE_READWRITE 18 0`00032000 ( 200.000 kB) 0.23% 0.00% PAGE_READWRITE|PAGE_GUARD 2 0`00005000 ( 20.000 kB) 0.02% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 0`ff065000 7fd`f676b000 ( 7.992 TB) Image 7fe`fe4cb000 0`0089e000 ( 8.617 MB) 0`7f0e0000 0`00f00000 ( 15.000 MB) Other 0`00610000 0`01590000 ( 21.563 MB) Heap 0`003b8000 0`000c8000 ( 800.000 kB) Stack 0`02c10000 0`0006c000 ( 432.000 kB) TEB 7ff`fffdb000 0`00002000 ( 8.000 kB) PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

0:000> !heap -s

****************************************************** NT HEAP STATS BELOW ****************************************************** LFH Key : 0x000000381021167d Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ------------------------------------------------------------------------------------- 0000000000280000 00000002 1024 412 1024 14 4 1 0 0 LFH 0000000000250000 00001002 1088 256 1088 5 2 2 0 0 LFH 0000000001cc0000 00001002 64 8 64 3 1 1 0 0 0000000001e60000 00001002 512 120 512 49 3 1 0 0 0000000001dc0000 00001002 512 8 512 2 1 1 0 0 -------------------------------------------------------------------------------------

The other dump saved is a minidump from which we can get thread information for Execution Residue (raw stack data) and reconstruct stack traces in Clone Dump:

Loading Dump File [C:\DebuggingTV\Procdump\notepad.exe_151117_000755.dmp] Comment: ' *** procdump -ma -r Notepad.exe *** Manual dump' User Mini Dump File: Only registers, stack and portions of memory are available

0:000> ~ . 0 Id: 87c.27f4 Suspend: 0 Teb: 000007ff`fffdd000 Unfrozen

0:000> k # Child-SP RetAddr Call Site 00 00000000`0016fac8 00000000`77069e9e 0x77069e6a 01 00000000`0016fad0 00000000`00000000 0x77069e9e

0:000> r rax=0000000000000000 rbx=000000000016fb40 rcx=0000000000280000 rdx=0000000000000000 rsi=0000000000000001 rdi=0000000000000000 rip=0000000077069e6a rsp=000000000016fac8 rbp=00000000ff030000 r8=000000000016f8e8 r9=00000000000a0cdc r10=0000000000000000 r11=0000000000000000 r12=0000000000000000 r13=0000000000000000 r14=0000000000000000 r15=0000000000000000 iopl=0 nv up ei pl zr na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 00000000`77069e6a c3 ret

Now we can see the original stack trace in Clone Dump:

0:000> k =000000000016fac8 ff # Child-SP RetAddr Call Site 00 00000000`0016fac8 00000000`77069e9e ntdll!NtSuspendThread+0xa 01 00000000`0016fad0 00000000`ff031064 user32!GetMessageW+0x34 02 00000000`0016fb00 00000000`ff03133c notepad!WinMain+0x182 03 00000000`0016fb80 00000000`77165a4d notepad!DisplayNonGenuineDlgWorker+0x2da 04 00000000`0016fc40 00000000`7729b831 kernel32!BaseThreadInitThunk+0xd 05 00000000`0016fc70 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Since we know TEB address from the minidump we can get stack region boundaries in Clone Dump:

0:000> dt _NT_TIB 000007fffffdd000 ntdll!_NT_TIB +0x000 ExceptionList : (null) +0x008 StackBase : 0x00000000`00170000 Void +0x010 StackLimit : 0x00000000`0015b000 Void +0x018 SubSystemTib : (null) +0x020 FiberData : 0x00000000`00001e00 Void +0x020 Version : 0x1e00 +0x028 ArbitraryUserPointer : (null) +0x030 Self : 0x000007ff`fffdd000 _NT_TIB

Now we can check Execution Residue (for example, for signs of Hidden Exceptions):

0:000> dpS 0x00000000`0015b000 0x00000000`00170000 [...]

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 232)

Saturday, October 31st, 2015

We already introduced Active Thread pattern variant for Mac OS X. Here we provide an example for Windows. Unless we have Evental Dumps, Active Threads in Windows are usually threads from Busy System or Spiking Threads, and, therefore, represent an abnormal behavior since most threads are waiting or calling some API. For Evental Dumps they may be just normal threads:

0:000> r rax=0000000000000006 rbx=0000000000000003 rcx=0000000000000018 rdx=0000000000000000 rsi=000000000028c601 rdi=0000000002bee25e rip=000007feff1a5a09 rsp=000000000028c380 rbp=0000000000000000 r8=0000000000000000 r9=00000000001653a0 r10=000000000000000e r11=000000000000000a r12=0000000000000006 r13=000000000028ca38 r14=0000000002bec888 r15=0000000000173630 iopl=0 nv up ei pl nz ac po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216 usp10!otlChainingLookup::apply+0×2f9: 000007fe`ff1a5a09 498d0c06 lea rcx,[r14+rax]

0:000> k # Child-SP RetAddr Call Site 00 00000000`0028c380 000007fe`ff19f4f2 usp10!otlChainingLookup::apply+0×2f9 01 00000000`0028c4b0 000007fe`ff19e777 usp10!ApplyLookup+0×592 02 00000000`0028c5a0 000007fe`ff19a634 usp10!ApplyFeatures+0×777 03 00000000`0028c860 000007fe`ff181800 usp10!SubstituteOtlGlyphs+0×224 04 00000000`0028c910 000007fe`ff174cc0 usp10!GenericEngineGetGlyphs+0×1000 05 00000000`0028ccb0 000007fe`ff1389c5 usp10!ShlShape+0×7a0 06 00000000`0028ced0 000007fe`ff147363 usp10!ScriptShape+0×205 07 00000000`0028cf70 000007fe`ff148ac9 usp10!RenderItemNoFallback+0×433 08 00000000`0028d030 000007fe`ff148d86 usp10!RenderItemWithFallback+0×129 09 00000000`0028d080 000007fe`ff14a5f7 usp10!RenderItem+0×36 0a 00000000`0028d0d0 000007fe`ff13b2c9 usp10!ScriptStringAnalyzeGlyphs+0×277 0b 00000000`0028d170 000007fe`fdd616bf usp10!ScriptStringAnalyse+0×399 0c 00000000`0028d1f0 000007fe`fdd614cc lpk!LpkCharsetDraw+0×4eb 0d 00000000`0028d3b0 00000000`774e85c5 lpk!LpkDrawTextEx+0×68 0e 00000000`0028d420 00000000`774e865c user32!DT_DrawStr+0xa6 0f 00000000`0028d4c0 00000000`774e826c user32!DT_DrawJustifiedLine+0xa6 10 00000000`0028d530 00000000`774e6cc8 user32!DrawTextExWorker+0×442 11 00000000`0028d640 000007fe`fbd840d1 user32!DrawTextW+0×57 12 00000000`0028d6b0 000007fe`fbd83e49 comctl32!CLVView::_ComputeLabelSizeWorker+0×1d1 13 00000000`0028da40 000007fe`fbd8cc48 comctl32!CLVView::v_RecomputeLabelSize+0×1f9 14 00000000`0028dd70 000007fe`fbda9d24 comctl32!CLVListView::v_DrawItem+0×284 15 00000000`0028e110 000007fe`fbd9773b comctl32!CLVDrawItemManager::DrawItem+0×4c0 16 00000000`0028e170 000007fe`fbd95f8e comctl32!CLVDrawManager::_PaintItems+0×3df 17 00000000`0028e3b0 000007fe`fbd95e87 comctl32!CLVDrawManager::_PaintWorkArea+0xda 18 00000000`0028e430 000007fe`fbd95cff comctl32!CLVDrawManager::_OnPaintWorkAreas+0×147 19 00000000`0028e4c0 000007fe`fbd06f1b comctl32!CLVDrawManager::_OnPaint+0×14b 1a 00000000`0028e570 000007fe`fbd06011 comctl32!CListView::WndProc+0xebf 1b 00000000`0028e770 00000000`774e9bd1 comctl32!CListView::s_WndProc+0×6cd 1c 00000000`0028e7d0 00000000`774e3bfc user32!UserCallWinProcCheckWow+0×1ad 1d 00000000`0028e890 00000000`774e3b78 user32!CallWindowProcAorW+0xdc 1e 00000000`0028e8e0 000007fe`fbca6215 user32!CallWindowProcW+0×18 1f 00000000`0028e920 000007fe`fbca69a0 comctl32!CallOriginalWndProc+0×1d 20 00000000`0028e960 000007fe`fbca6768 comctl32!CallNextSubclassProc+0×8c 21 00000000`0028e9e0 000007fe`fde1096a comctl32!DefSubclassProc+0×7c 22 00000000`0028ea30 000007fe`fdde9df4 shell32!DefSubclassProc+0×56 23 00000000`0028ea60 000007fe`fbca69a0 shell32!CListViewHost::s_ListViewSubclassWndProc+0×267 24 00000000`0028eb40 000007fe`fbca6877 comctl32!CallNextSubclassProc+0×8c 25 00000000`0028ebc0 00000000`774e9bd1 comctl32!MasterSubclassProc+0xe7 26 00000000`0028ec60 00000000`774e72cb user32!UserCallWinProcCheckWow+0×1ad 27 00000000`0028ed20 00000000`774e6829 user32!DispatchClientMessage+0xc3 28 00000000`0028ed80 00000000`776211f5 user32!_fnDWORD+0×2d 29 00000000`0028ede0 00000000`774e6e5a ntdll!KiUserCallbackDispatcherContinue 2a 00000000`0028ee68 00000000`774e6e6c user32!NtUserDispatchMessage+0xa 2b 00000000`0028ee70 00000000`774e67c2 user32!DispatchMessageWorker+0×55b 2c 00000000`0028eef0 000007fe`fbcc34a4 user32!IsDialogMessageW+0×153 2d 00000000`0028ef80 000007fe`fbcc583f comctl32!Prop_IsDialogMessage+0×1f0 2e 00000000`0028eff0 000007fe`fbcc5c05 comctl32!_RealPropertySheet+0×31b 2f 00000000`0028f0c0 000007fe`ff214e68 comctl32!_PropertySheet+0×55 30 00000000`0028f100 000007fe`ff214bb1 comdlg32!Print_InvokePropertySheets+0×2c6 31 00000000`0028f660 000007fe`ff21499c comdlg32!PrintDlgExX+0×2be 32 00000000`0028f6c0 00000000`ffec250f comdlg32!PrintDlgExW+0×38 33 00000000`0028f730 00000000`ffec242b notepad!GetPrinterDCviaDialog+0xab 34 00000000`0028f7f0 00000000`ffec23e8 notepad!PrintIt+0×46 35 00000000`0028fb70 00000000`ffec14eb notepad!NPCommand+0xdb 36 00000000`0028fca0 00000000`774e9bd1 notepad!NPWndProc+0×540 37 00000000`0028fce0 00000000`774e98da user32!UserCallWinProcCheckWow+0×1ad 38 00000000`0028fda0 00000000`ffec10bc user32!DispatchMessageWorker+0×3b5 39 00000000`0028fe20 00000000`ffec133c notepad!WinMain+0×16f 3a 00000000`0028fea0 00000000`773c59ed notepad!DisplayNonGenuineDlgWorker+0×2da 3b 00000000`0028ff60 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 3c 00000000`0028ff90 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see the thread is active, in the middle of the function. For comparison, the next 2 threads are waiting and calling API respectively:

0:000> ~1k # Child-SP RetAddr Call Site 00 00000000`02edf748 00000000`775eb037 ntdll!NtWaitForMultipleObjects+0xa 01 00000000`02edf750 00000000`773c59ed ntdll!TppWaiterpThread+0×14d 02 00000000`02edf9f0 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 03 00000000`02edfa20 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ~2k # Child-SP RetAddr Call Site 00 00000000`0344e048 000007fe`fd3f403e ntdll!NtUnmapViewOfSection+0xa 01 00000000`0344e050 00000000`774e2edf KERNELBASE!FreeLibrary+0xa4 02 00000000`0344e080 000007fe`fdddaab3 user32!PrivateExtractIconsW+0×34b 03 00000000`0344e5a0 000007fe`fdddac28 shell32!SHPrivateExtractIcons+0×50a 04 00000000`0344e870 000007fe`fdde34b4 shell32!SHDefExtractIconW+0×254 05 00000000`0344eb60 000007fe`fdde3435 shell32!CExtractIcon::_ExtractW+0xcd 06 00000000`0344ebe0 000007fe`fdf0d529 shell32!CExtractIconBase::Extract+0×21 07 00000000`0344ec20 000007fe`fdf0d2da shell32!IExtractIcon_Extract+0×43 08 00000000`0344ec60 000007fe`fdddfff0 shell32!_GetILIndexGivenPXIcon+0×22e 09 00000000`0344f100 000007fe`fdde27a4 shell32!_GetILIndexFromItem+0×87 0a 00000000`0344f1a0 000007fe`fddb6506 shell32!SHGetIconIndexFromPIDL+0×66 0b 00000000`0344f1d0 000007fe`fdedb186 shell32!MapIDListToIconILIndex+0×52 0c 00000000`0344f250 000007fe`fdddc54c shell32!CLoadSystemIconTask::InternalResumeRT+0×110 0d 00000000`0344f2e0 000007fe`fde0efcb shell32!CRunnableTask::Run+0xda 0e 00000000`0344f310 000007fe`fde12b56 shell32!CShellTask::TT_Run+0×124 0f 00000000`0344f340 000007fe`fde12cb2 shell32!CShellTaskThread::ThreadProc+0×1d2 10 00000000`0344f3e0 000007fe`ff2b3843 shell32!CShellTaskThread::s_ThreadProc+0×22 11 00000000`0344f410 00000000`775f15db shlwapi!ExecuteWorkItemThreadProc+0xf 12 00000000`0344f440 00000000`775f0c56 ntdll!RtlpTpWorkCallback+0×16b 13 00000000`0344f520 00000000`773c59ed ntdll!TppWorkerThread+0×5ff 14 00000000`0344f820 00000000`775fc541 kernel32!BaseThreadInitThunk+0xd 15 00000000`0344f850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

0:000> ub ntdll!NtUnmapViewOfSection+0xa ntdll!NtAccessCheckAndAuditAlarm: 00000000`77621540 4c8bd1 mov r10,rcx 00000000`77621543 b826000000 mov eax,26h 00000000`77621548 0f05 syscall 00000000`7762154a c3 ret 00000000`7762154b 0f1f440000 nop dword ptr [rax+rax] ntdll!NtUnmapViewOfSection: 00000000`77621550 4c8bd1 mov r10,rcx 00000000`77621553 b827000000 mov eax,27h 00000000`77621558 0f05 syscall

Our Active Thread is not Spiking Thread since CPU consumption is minimal:

0:000> !runaway f User Mode Time Thread Time 0:1ca4 0 days 0:00:00.171 11:1fb0 0 days 0:00:00.000 10:f98 0 days 0:00:00.000 9:eb8 0 days 0:00:00.000 8:1b80 0 days 0:00:00.000 7:139c 0 days 0:00:00.000 6:1d9c 0 days 0:00:00.000 5:1b44 0 days 0:00:00.000 4:1edc 0 days 0:00:00.000 3:830 0 days 0:00:00.000 2:1638 0 days 0:00:00.000 1:1ab0 0 days 0:00:00.000 Kernel Mode Time Thread Time 0:1ca4 0 days 0:00:00.421 11:1fb0 0 days 0:00:00.000 10:f98 0 days 0:00:00.000 9:eb8 0 days 0:00:00.000 8:1b80 0 days 0:00:00.000 7:139c 0 days 0:00:00.000 6:1d9c 0 days 0:00:00.000 5:1b44 0 days 0:00:00.000 4:1edc 0 days 0:00:00.000 3:830 0 days 0:00:00.000 2:1638 0 days 0:00:00.000 1:1ab0 0 days 0:00:00.000 Elapsed Time Thread Time 11:1fb0 24692 days 13:29:46.335 0:1ca4 0 days 0:02:39.671 1:1ab0 0 days 0:01:48.239 2:1638 0 days 0:01:18.837 4:1edc 0 days 0:01:18.697 3:830 0 days 0:01:18.697 5:1b44 0 days 0:01:18.497 6:1d9c 0 days 0:01:18.387 7:139c 0 days 0:01:15.957 8:1b80 0 days 0:01:14.397 9:eb8 0 days 0:01:01.485 10:f98 0 days 0:00:39.849

However, the huge Elapsed Time for the thread #11 (most likely the value is uninitialized) and its stack trace suggest that the dump was saved on Create Thread debugging event by a debugger:

0:000> ~11k # Child-SP RetAddr Call Site 00 00000000`0666fb08 00000000`00000000 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 231)

Monday, October 26th, 2015

One of the customers of Software Diagnostics Services submitted memory dumps saved by DebugDiag to accompany software logs for the analysis of sudden process exit. We didn’t request such memory dumps and initially dismissed them. However, during software log analysis we decided to look at Adjoint Spaces to see whether there was some additional information in stack traces. We found out that those dumps were saved on each thread exit event. Since other threads were either waiting or Active Threads their analysis gave clues of process behavior before process exit. For example, we found ALPC Wait Chain to Coupled Process. The latter prompted us to analyze Coupled Activities in the software log and diagnose the possible problem there. Since saving memory dumps on thread creation and exit can be useful technique we decided to add Evental Dumps memory analysis pattern to our pattern catalog.

To illustrate this pattern we show Stack Trace Collection from notepad.exe. This process usually has just one thread. But, if we try to open a Print dialog the number of threads increases up to 12.

We attach WinDbg to notepad.exe process and set up debugging event filter (Debug \ Event Filters… menu) for Create Thread event with a command line as shown in this picture:

We then resume execution using g command and switch to Notepad. There we first open File \ Page Setup… dialog. We observe that a memory dump is saved. Then we open File \ Print… dialog and notice the creation 11 more process memory dump:

We now show stack traces from these dumps where we use ~*kc WinDbg command to minimize the amount of inessential for our purposes output:

// 1st dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!NtAlertThread 01 ntdll!TppWaiterEnqueueTransition 02 ntdll!TppWaitpSet 03 ntdll!TpSetWait 04 ntdll!TppTimerpInitTimerQueueQueue 05 ntdll!TppTimerpAllocTimerQueue 06 ntdll!TppTimerpAcquirePoolTimerQueue 07 ntdll!TppTimerAlloc 08 ntdll!TpAllocTimer 09 KERNELBASE!CreateThreadpoolTimer 0a rpcrt4!RPC_THREAD_POOL::CreateTimer 0b rpcrt4!GarbageCollectionNeeded 0c rpcrt4!LRPC_CASSOCIATION::RemoveReference 0d rpcrt4!LRPC_CCALL::`vector deleting destructor' 0e rpcrt4!LRPC_CCALL::FreeBuffer 0f rpcrt4!Ndr64pClientFinally 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 sechost!LsaLookupClose 13 sechost!LookupAccountNameInternal 14 sechost!LookupAccountNameLocalW 15 rpcrt4!RpcpLookupAccountNameDirect 16 rpcrt4!RpcpLookupAccountName 17 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 18 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 19 rpcrt4!RpcBindingSetAuthInfoExW 1a winspool!STRING_HANDLE_bind 1b rpcrt4!GenericHandleMgr 1c rpcrt4!ExplicitBindHandleMgr 1d rpcrt4!Ndr64pClientSetupTransferSyntax 1e rpcrt4!NdrpClientCall3 1f rpcrt4!NdrClientCall3 20 winspool!RpcSplOpenPrinter 21 winspool!OpenPrinterRPC 22 winspool!OpenPrinter2W 23 comdlg32!PrintOpenPrinter 24 comdlg32!PrintDlgX 25 comdlg32!PageSetupDlgX 26 comdlg32!PageSetupDlgW 27 notepad!NPCommand 28 notepad!NPWndProc 29 user32!UserCallWinProcCheckWow 2a user32!DispatchMessageWorker 2b notepad!WinMain 2c notepad!DisplayNonGenuineDlgWorker 2d kernel32!BaseThreadInitThunk 2e ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 2nd dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!ZwOpenKeyEx 01 kernel32!LocalBaseRegOpenKey 02 kernel32!RegOpenKeyExInternalW 03 kernel32!RegOpenKeyExW 04 rpcrt4!Server2003NegotiateDisable 05 rpcrt4!IsBindTimeFeatureNegotiationDisabled 06 rpcrt4!OSF_CCONNECTION::SendBindPacket 07 rpcrt4!OSF_CCONNECTION::ActuallyDoBinding 08 rpcrt4!OSF_CCONNECTION::OpenConnectionAndBind 09 rpcrt4!OSF_CCALL::BindToServer 0a rpcrt4!OSF_BINDING_HANDLE::InitCCallWithAssociation 0b rpcrt4!OSF_BINDING_HANDLE::AllocateCCall 0c rpcrt4!OSF_BINDING_HANDLE::NegotiateTransferSyntax 0d rpcrt4!I_RpcNegotiateTransferSyntax 0e rpcrt4!Ndr64pClientSetupTransferSyntax 0f rpcrt4!NdrpClientCall3 10 rpcrt4!NdrClientCall3 11 srvcli!NetShareEnum 12 ntshrui!CShareCache::RefreshNoCritSec 13 ntshrui!CShareCache::Refresh 14 ntshrui!DllMain 15 ntdll!RtlRunOnceExecuteOnce 16 kernel32!InitOnceExecuteOnce 17 ntshrui!DllGetClassObject 18 ole32!CClassCache::CDllPathEntry::DllGetClassObject 19 ole32!CClassCache::CDllFnPtrMoniker::BindToObjectNoSwitch 1a ole32!CClassCache::GetClassObject 1b ole32!CServerContextActivator::CreateInstance 1c ole32!ActivationPropertiesIn::DelegateCreateInstance 1d ole32!CApartmentActivator::CreateInstance 1e ole32!CProcessActivator::CCICallback 1f ole32!CProcessActivator::AttemptActivation 20 ole32!CProcessActivator::ActivateByContext 21 ole32!CProcessActivator::CreateInstance 22 ole32!ActivationPropertiesIn::DelegateCreateInstance 23 ole32!CClientContextActivator::CreateInstance 24 ole32!ActivationPropertiesIn::DelegateCreateInstance 25 ole32!ICoCreateInstanceEx 26 ole32!CoCreateInstance 27 shell32!_SHCoCreateInstance 28 shell32!SHExtCoCreateInstance 29 shell32!DCA_SHExtCoCreateInstance 2a shell32!CFSIconOverlayManager::_s_LoadIconOverlayIdentifiers 2b shell32!CFSIconOverlayManager::CreateInstance 2c shell32!IconOverlayManagerInit 2d shell32!GetIconOverlayManager 2e shell32!FileIconInit 2f shell32!Shell_GetImageLists 30 comdlg32!CPrintDialog::CPrintDialog 31 comdlg32!Print_GeneralDlgProc 32 user32!UserCallDlgProcCheckWow 33 user32!DefDlgProcWorker 34 user32!InternalCreateDialog 35 user32!CreateDialogIndirectParamAorW 36 user32!CreateDialogIndirectParamW 37 comctl32!_CreatePageDialog 38 comctl32!_CreatePage 39 comctl32!PageChange 3a comctl32!InitPropSheetDlg 3b comctl32!PropSheetDlgProc 3c user32!UserCallDlgProcCheckWow 3d user32!DefDlgProcWorker 3e user32!InternalCreateDialog 3f user32!CreateDialogIndirectParamAorW 40 user32!CreateDialogIndirectParamW 41 comctl32!_RealPropertySheet 42 comctl32!_PropertySheet 43 comdlg32!Print_InvokePropertySheets 44 comdlg32!PrintDlgExX 45 comdlg32!PrintDlgExW 46 notepad!GetPrinterDCviaDialog 47 notepad!PrintIt 48 notepad!NPCommand 49 notepad!NPWndProc 4a user32!UserCallWinProcCheckWow 4b user32!DispatchMessageWorker 4c notepad!WinMain 4d notepad!DisplayNonGenuineDlgWorker 4e kernel32!BaseThreadInitThunk 4f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 3rd dump: we see 2 threads start at the same time

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!RtlCompareMemoryUlong 01 ntdll!RtlpAllocateHeap 02 ntdll!RtlAllocateHeap 03 ntdll!RtlDebugAllocateHeap 04 ntdll! ?? ::FNODOBFM::`string' 05 ntdll!RtlAllocateHeap 06 ole32!CRpcResolver::GetThreadWinstaDesktop 07 ole32!CRpcResolver::GetConnection 08 ole32!CoInitializeSecurity 09 ole32!InitializeSecurity 0a ole32!ChannelProcessInitialize 0b ole32!CComApartment::InitRemoting 0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0d ole32!CGIPTable::RegisterInterfaceInGlobal 0e shell32!MarshalToGIT 0f shell32!CBrowserProgressAggregator::BeginSession 10 shell32!IUnknown_BeginBrowserProgressSession 11 shell32!CDefView::CreateViewWindow3 12 shell32!CExplorerBrowser::_CreateViewWindow 13 shell32!CExplorerBrowser::_SwitchView 14 shell32!CExplorerBrowser::_BrowseToView 15 shell32!CExplorerBrowser::_BrowseObjectInternal 16 shell32!CExplorerBrowser::_OnBrowseObject 17 shell32!CExplorerBrowser::BrowseObject 18 comdlg32!CPrintDialog::CreatePrintBrowser 19 comdlg32!CPrintDialog::OnInitDialog 1a comdlg32!Print_GeneralDlgProc 1b user32!UserCallDlgProcCheckWow 1c user32!DefDlgProcWorker 1d user32!InternalCreateDialog 1e user32!CreateDialogIndirectParamAorW 1f user32!CreateDialogIndirectParamW 20 comctl32!_CreatePageDialog 21 comctl32!_CreatePage 22 comctl32!PageChange 23 comctl32!InitPropSheetDlg 24 comctl32!PropSheetDlgProc 25 user32!UserCallDlgProcCheckWow 26 user32!DefDlgProcWorker 27 user32!InternalCreateDialog 28 user32!CreateDialogIndirectParamAorW 29 user32!CreateDialogIndirectParamW 2a comctl32!_RealPropertySheet 2b comctl32!_PropertySheet 2c comdlg32!Print_InvokePropertySheets 2d comdlg32!PrintDlgExX 2e comdlg32!PrintDlgExW 2f notepad!GetPrinterDCviaDialog 30 notepad!PrintIt 31 notepad!NPCommand 32 notepad!NPWndProc 33 user32!UserCallWinProcCheckWow 34 user32!DispatchMessageWorker 35 notepad!WinMain 36 notepad!DisplayNonGenuineDlgWorker 37 kernel32!BaseThreadInitThunk 38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupOpenLocalPolicy 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!RpcBindingSetAuthInfoExW 0c winspool!STRING_HANDLE_bind 0d rpcrt4!GenericHandleMgr 0e rpcrt4!ExplicitBindHandleMgr 0f rpcrt4!Ndr64pClientSetupTransferSyntax 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 winspool!RpcSplOpenPrinter 13 winspool!OpenPrinterRPC 14 winspool!OpenPrinter2W 15 prncache!PrintCache::Listeners::Listener::Start 16 prncache!PrintCache::Listeners::Listener::StartCB 17 ntdll!TppWorkpExecuteCallback 18 ntdll!TppWorkerThread 19 kernel32!BaseThreadInitThunk 1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 0 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 4th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!RtlCompareMemoryUlong 01 ntdll!RtlpAllocateHeap 02 ntdll!RtlAllocateHeap 03 ntdll!RtlDebugAllocateHeap 04 ntdll! ?? ::FNODOBFM::`string' 05 ntdll!RtlAllocateHeap 06 ole32!CRpcResolver::GetThreadWinstaDesktop 07 ole32!CRpcResolver::GetConnection 08 ole32!CoInitializeSecurity 09 ole32!InitializeSecurity 0a ole32!ChannelProcessInitialize 0b ole32!CComApartment::InitRemoting 0c ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0d ole32!CGIPTable::RegisterInterfaceInGlobal 0e shell32!MarshalToGIT 0f shell32!CBrowserProgressAggregator::BeginSession 10 shell32!IUnknown_BeginBrowserProgressSession 11 shell32!CDefView::CreateViewWindow3 12 shell32!CExplorerBrowser::_CreateViewWindow 13 shell32!CExplorerBrowser::_SwitchView 14 shell32!CExplorerBrowser::_BrowseToView 15 shell32!CExplorerBrowser::_BrowseObjectInternal 16 shell32!CExplorerBrowser::_OnBrowseObject 17 shell32!CExplorerBrowser::BrowseObject 18 comdlg32!CPrintDialog::CreatePrintBrowser 19 comdlg32!CPrintDialog::OnInitDialog 1a comdlg32!Print_GeneralDlgProc 1b user32!UserCallDlgProcCheckWow 1c user32!DefDlgProcWorker 1d user32!InternalCreateDialog 1e user32!CreateDialogIndirectParamAorW 1f user32!CreateDialogIndirectParamW 20 comctl32!_CreatePageDialog 21 comctl32!_CreatePage 22 comctl32!PageChange 23 comctl32!InitPropSheetDlg 24 comctl32!PropSheetDlgProc 25 user32!UserCallDlgProcCheckWow 26 user32!DefDlgProcWorker 27 user32!InternalCreateDialog 28 user32!CreateDialogIndirectParamAorW 29 user32!CreateDialogIndirectParamW 2a comctl32!_RealPropertySheet 2b comctl32!_PropertySheet 2c comdlg32!Print_InvokePropertySheets 2d comdlg32!PrintDlgExX 2e comdlg32!PrintDlgExW 2f notepad!GetPrinterDCviaDialog 30 notepad!PrintIt 31 notepad!NPCommand 32 notepad!NPWndProc 33 user32!UserCallWinProcCheckWow 34 user32!DispatchMessageWorker 35 notepad!WinMain 36 notepad!DisplayNonGenuineDlgWorker 37 kernel32!BaseThreadInitThunk 38 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupOpenLocalPolicy 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!RpcBindingSetAuthInfoExW 0c winspool!STRING_HANDLE_bind 0d rpcrt4!GenericHandleMgr 0e rpcrt4!ExplicitBindHandleMgr 0f rpcrt4!Ndr64pClientSetupTransferSyntax 10 rpcrt4!NdrpClientCall3 11 rpcrt4!NdrClientCall3 12 winspool!RpcSplOpenPrinter 13 winspool!OpenPrinterRPC 14 winspool!OpenPrinter2W 15 prncache!PrintCache::Listeners::Listener::Start 16 prncache!PrintCache::Listeners::Listener::StartCB 17 ntdll!TppWorkpExecuteCallback 18 ntdll!TppWorkerThread 19 kernel32!BaseThreadInitThunk 1a ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!LdrpInitializeThread 01 ntdll!LdrpInitialize 02 ntdll!LdrInitializeThunk

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 5th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 sechost!LsaLookupClose 05 sechost!LookupAccountNameInternal 06 sechost!LookupAccountNameLocalW 07 rpcrt4!RpcpLookupAccountNameDirect 08 rpcrt4!RpcpLookupAccountName 09 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 0a rpcrt4!LRPC_FAST_BINDING_HANDLE::SetAuthInformation 0b rpcrt4!LRPC_FAST_BINDING_HANDLE::LRPC_FAST_BINDING_HANDLE 0c rpcrt4!LrpcCreateFastBindingHandle 0d rpcrt4!RpcBindingCreateW 0e ole32!CFastBH::CreateFromBindingString 0f ole32!CFastBH::GetOrCreate 10 ole32!CRpcResolver::GetConnection 11 ole32!CoInitializeSecurity 12 ole32!InitializeSecurity 13 ole32!ChannelProcessInitialize 14 ole32!CComApartment::InitRemoting 15 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 16 ole32!CGIPTable::RegisterInterfaceInGlobal 17 shell32!MarshalToGIT 18 shell32!CBrowserProgressAggregator::BeginSession 19 shell32!IUnknown_BeginBrowserProgressSession 1a shell32!CDefView::CreateViewWindow3 1b shell32!CExplorerBrowser::_CreateViewWindow 1c shell32!CExplorerBrowser::_SwitchView 1d shell32!CExplorerBrowser::_BrowseToView 1e shell32!CExplorerBrowser::_BrowseObjectInternal 1f shell32!CExplorerBrowser::_OnBrowseObject 20 shell32!CExplorerBrowser::BrowseObject 21 comdlg32!CPrintDialog::CreatePrintBrowser 22 comdlg32!CPrintDialog::OnInitDialog 23 comdlg32!Print_GeneralDlgProc 24 user32!UserCallDlgProcCheckWow 25 user32!DefDlgProcWorker 26 user32!InternalCreateDialog 27 user32!CreateDialogIndirectParamAorW 28 user32!CreateDialogIndirectParamW 29 comctl32!_CreatePageDialog 2a comctl32!_CreatePage 2b comctl32!PageChange 2c comctl32!InitPropSheetDlg 2d comctl32!PropSheetDlgProc 2e user32!UserCallDlgProcCheckWow 2f user32!DefDlgProcWorker 30 user32!InternalCreateDialog 31 user32!CreateDialogIndirectParamAorW 32 user32!CreateDialogIndirectParamW 33 comctl32!_RealPropertySheet 34 comctl32!_PropertySheet 35 comdlg32!Print_InvokePropertySheets 36 comdlg32!PrintDlgExX 37 comdlg32!PrintDlgExW 38 notepad!GetPrinterDCviaDialog 39 notepad!PrintIt 3a notepad!NPCommand 3b notepad!NPWndProc 3c user32!UserCallWinProcCheckWow 3d user32!DispatchMessageWorker 3e notepad!WinMain 3f notepad!DisplayNonGenuineDlgWorker 40 kernel32!BaseThreadInitThunk 41 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcSplOpenPrinter 05 winspool!OpenPrinterRPC 06 winspool!OpenPrinter2W 07 prncache!PrintCache::Listeners::Listener::Start 08 prncache!PrintCache::Listeners::Listener::StartCB 09 ntdll!TppWorkpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcEnumPrinters 05 winspool!EnumPrintersW 06 prncache!PrintCache::Listeners::ConnectionListener::EnumConnectionsAndRegister 07 prncache!PrintCache::Listeners::ConnectionListener::UpdateCB 08 ntdll!TppWorkpExecuteCallback 09 ntdll!TppWorkerThread 0a kernel32!BaseThreadInitThunk 0b ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 6th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 ntdll!NtAlpcConnectPort 01 rpcrt4!LRPC_CASSOCIATION::AlpcConnect 02 rpcrt4!LRPC_CASSOCIATION::Connect 03 rpcrt4!LRPC_BASE_BINDING_HANDLE::DriveStateForward 04 rpcrt4!LRPC_FAST_BINDING_HANDLE::Bind 05 rpcrt4!RpcBindingBind 06 ole32!CFastBH::CreateFromBindingString 07 ole32!CFastBH::GetOrCreate 08 ole32!CRpcResolver::GetConnection 09 ole32!CoInitializeSecurity 0a ole32!InitializeSecurity 0b ole32!ChannelProcessInitialize 0c ole32!CComApartment::InitRemoting 0d ole32!CGIPTable::RegisterInterfaceInGlobalHlp 0e ole32!CGIPTable::RegisterInterfaceInGlobal 0f shell32!MarshalToGIT 10 shell32!CBrowserProgressAggregator::BeginSession 11 shell32!IUnknown_BeginBrowserProgressSession 12 shell32!CDefView::CreateViewWindow3 13 shell32!CExplorerBrowser::_CreateViewWindow 14 shell32!CExplorerBrowser::_SwitchView 15 shell32!CExplorerBrowser::_BrowseToView 16 shell32!CExplorerBrowser::_BrowseObjectInternal 17 shell32!CExplorerBrowser::_OnBrowseObject 18 shell32!CExplorerBrowser::BrowseObject 19 comdlg32!CPrintDialog::CreatePrintBrowser 1a comdlg32!CPrintDialog::OnInitDialog 1b comdlg32!Print_GeneralDlgProc 1c user32!UserCallDlgProcCheckWow 1d user32!DefDlgProcWorker 1e user32!InternalCreateDialog 1f user32!CreateDialogIndirectParamAorW 20 user32!CreateDialogIndirectParamW 21 comctl32!_CreatePageDialog 22 comctl32!_CreatePage 23 comctl32!PageChange 24 comctl32!InitPropSheetDlg 25 comctl32!PropSheetDlgProc 26 user32!UserCallDlgProcCheckWow 27 user32!DefDlgProcWorker 28 user32!InternalCreateDialog 29 user32!CreateDialogIndirectParamAorW 2a user32!CreateDialogIndirectParamW 2b comctl32!_RealPropertySheet 2c comctl32!_PropertySheet 2d comdlg32!Print_InvokePropertySheets 2e comdlg32!PrintDlgExX 2f comdlg32!PrintDlgExW 30 notepad!GetPrinterDCviaDialog 31 notepad!PrintIt 32 notepad!NPCommand 33 notepad!NPWndProc 34 user32!UserCallWinProcCheckWow 35 user32!DispatchMessageWorker 36 notepad!WinMain 37 notepad!DisplayNonGenuineDlgWorker 38 kernel32!BaseThreadInitThunk 39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!RtlEnterCriticalSection 01 ntdll!RtlDebugAllocateHeap 02 ntdll! ?? ::FNODOBFM::`string' 03 ntdll!RtlAllocateHeap 04 rpcrt4!AllocWrapper 05 rpcrt4!SID_CACHE::Query 06 rpcrt4!RpcpLookupAccountName 07 rpcrt4!LRPC_BASE_BINDING_HANDLE::SetAuthInformation 08 rpcrt4!LRPC_BINDING_HANDLE::SetAuthInformation 09 rpcrt4!RpcBindingSetAuthInfoExW 0a winspool!STRING_HANDLE_bind 0b rpcrt4!GenericHandleMgr 0c rpcrt4!ExplicitBindHandleMgr 0d rpcrt4!Ndr64pClientSetupTransferSyntax 0e rpcrt4!NdrpClientCall3 0f rpcrt4!NdrClientCall3 10 winspool!RpcSplOpenPrinter 11 winspool!OpenPrinterRPC 12 winspool!OpenPrinter2W 13 prncache!PrintCache::Listeners::Listener::Start 14 prncache!PrintCache::Listeners::Listener::StartCB 15 ntdll!TppWorkpExecuteCallback 16 ntdll!TppWorkerThread 17 kernel32!BaseThreadInitThunk 18 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 user32!NtUserAttachThreadInput 01 shell32!CWaitTask::s_WaitBeforeCursing 02 ntdll!RtlpTpWaitCallback 03 ntdll!TppWaitpExecuteCallback 04 ntdll!TppWorkerThread 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 7th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 rpcrt4!RpcStringFreeW 01 ole32!CFastBH::CreateFromBindingString 02 ole32!CFastBH::GetOrCreate 03 ole32!CRpcResolver::GetConnection 04 ole32!CoInitializeSecurity 05 ole32!InitializeSecurity 06 ole32!ChannelProcessInitialize 07 ole32!CComApartment::InitRemoting 08 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 09 ole32!CGIPTable::RegisterInterfaceInGlobal 0a shell32!MarshalToGIT 0b shell32!CBrowserProgressAggregator::BeginSession 0c shell32!IUnknown_BeginBrowserProgressSession 0d shell32!CDefView::CreateViewWindow3 0e shell32!CExplorerBrowser::_CreateViewWindow 0f shell32!CExplorerBrowser::_SwitchView 10 shell32!CExplorerBrowser::_BrowseToView 11 shell32!CExplorerBrowser::_BrowseObjectInternal 12 shell32!CExplorerBrowser::_OnBrowseObject 13 shell32!CExplorerBrowser::BrowseObject 14 comdlg32!CPrintDialog::CreatePrintBrowser 15 comdlg32!CPrintDialog::OnInitDialog 16 comdlg32!Print_GeneralDlgProc 17 user32!UserCallDlgProcCheckWow 18 user32!DefDlgProcWorker 19 user32!InternalCreateDialog 1a user32!CreateDialogIndirectParamAorW 1b user32!CreateDialogIndirectParamW 1c comctl32!_CreatePageDialog 1d comctl32!_CreatePage 1e comctl32!PageChange 1f comctl32!InitPropSheetDlg 20 comctl32!PropSheetDlgProc 21 user32!UserCallDlgProcCheckWow 22 user32!DefDlgProcWorker 23 user32!InternalCreateDialog 24 user32!CreateDialogIndirectParamAorW 25 user32!CreateDialogIndirectParamW 26 comctl32!_RealPropertySheet 27 comctl32!_PropertySheet 28 comdlg32!Print_InvokePropertySheets 29 comdlg32!PrintDlgExX 2a comdlg32!PrintDlgExW 2b notepad!GetPrinterDCviaDialog 2c notepad!PrintIt 2d notepad!NPCommand 2e notepad!NPWndProc 2f user32!UserCallWinProcCheckWow 30 user32!DispatchMessageWorker 31 notepad!WinMain 32 notepad!DisplayNonGenuineDlgWorker 33 kernel32!BaseThreadInitThunk 34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcSplOpenPrinter 05 winspool!OpenPrinterRPC 06 winspool!OpenPrinter2W 07 prncache!PrintCache::Listeners::Listener::Start 08 prncache!PrintCache::Listeners::Listener::StartCB 09 ntdll!TppWorkpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 8th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 user32!NtUserSetWindowLongPtr 01 user32!SetWindowLongPtr 02 ole32!OXIDEntry::StartServer 03 ole32!CGIPTable::RegisterInterfaceInGlobalHlp 04 ole32!CGIPTable::RegisterInterfaceInGlobal 05 shell32!MarshalToGIT 06 shell32!CBrowserProgressAggregator::BeginSession 07 shell32!IUnknown_BeginBrowserProgressSession 08 shell32!CDefView::CreateViewWindow3 09 shell32!CExplorerBrowser::_CreateViewWindow 0a shell32!CExplorerBrowser::_SwitchView 0b shell32!CExplorerBrowser::_BrowseToView 0c shell32!CExplorerBrowser::_BrowseObjectInternal 0d shell32!CExplorerBrowser::_OnBrowseObject 0e shell32!CExplorerBrowser::BrowseObject 0f comdlg32!CPrintDialog::CreatePrintBrowser 10 comdlg32!CPrintDialog::OnInitDialog 11 comdlg32!Print_GeneralDlgProc 12 user32!UserCallDlgProcCheckWow 13 user32!DefDlgProcWorker 14 user32!InternalCreateDialog 15 user32!CreateDialogIndirectParamAorW 16 user32!CreateDialogIndirectParamW 17 comctl32!_CreatePageDialog 18 comctl32!_CreatePage 19 comctl32!PageChange 1a comctl32!InitPropSheetDlg 1b comctl32!PropSheetDlgProc 1c user32!UserCallDlgProcCheckWow 1d user32!DefDlgProcWorker 1e user32!InternalCreateDialog 1f user32!CreateDialogIndirectParamAorW 20 user32!CreateDialogIndirectParamW 21 comctl32!_RealPropertySheet 22 comctl32!_PropertySheet 23 comdlg32!Print_InvokePropertySheets 24 comdlg32!PrintDlgExX 25 comdlg32!PrintDlgExW 26 notepad!GetPrinterDCviaDialog 27 notepad!PrintIt 28 notepad!NPCommand 29 notepad!NPWndProc 2a user32!UserCallWinProcCheckWow 2b user32!DispatchMessageWorker 2c notepad!WinMain 2d notepad!DisplayNonGenuineDlgWorker 2e kernel32!BaseThreadInitThunk 2f ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 9th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 user32!NtUserPeekMessage 01 user32!PeekMessageW 02 shell32!PeekMessageWithWakeMask 03 shell32!SHProcessMessagesUntilEventsEx 04 shell32!CDefView::_SetItemCollection 05 shell32!CDefView::_CreateNewCollection 06 shell32!CDefView::CreateViewWindow3 07 shell32!CExplorerBrowser::_CreateViewWindow 08 shell32!CExplorerBrowser::_SwitchView 09 shell32!CExplorerBrowser::_BrowseToView 0a shell32!CExplorerBrowser::_BrowseObjectInternal 0b shell32!CExplorerBrowser::_OnBrowseObject 0c shell32!CExplorerBrowser::BrowseObject 0d comdlg32!CPrintDialog::CreatePrintBrowser 0e comdlg32!CPrintDialog::OnInitDialog 0f comdlg32!Print_GeneralDlgProc 10 user32!UserCallDlgProcCheckWow 11 user32!DefDlgProcWorker 12 user32!InternalCreateDialog 13 user32!CreateDialogIndirectParamAorW 14 user32!CreateDialogIndirectParamW 15 comctl32!_CreatePageDialog 16 comctl32!_CreatePage 17 comctl32!PageChange 18 comctl32!InitPropSheetDlg 19 comctl32!PropSheetDlgProc 1a user32!UserCallDlgProcCheckWow 1b user32!DefDlgProcWorker 1c user32!InternalCreateDialog 1d user32!CreateDialogIndirectParamAorW 1e user32!CreateDialogIndirectParamW 1f comctl32!_RealPropertySheet 20 comctl32!_PropertySheet 21 comdlg32!Print_InvokePropertySheets 22 comdlg32!PrintDlgExX 23 comdlg32!PrintDlgExW 24 notepad!GetPrinterDCviaDialog 25 notepad!PrintIt 26 notepad!NPCommand 27 notepad!NPWndProc 28 user32!UserCallWinProcCheckWow 29 user32!DispatchMessageWorker 2a notepad!WinMain 2b notepad!DisplayNonGenuineDlgWorker 2c kernel32!BaseThreadInitThunk 2d ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 KERNELBASE!WaitForSingleObjectEx 02 shlwapi!CreateThreadWorker 03 shlwapi!SHCreateThread 04 shell32!CEnumThread::Run 05 shell32!CEnumTask::_StartEnumThread 06 shell32!CEnumTask::_IncrEnumFolder 07 shell32!CEnumTask::InternalResumeRT 08 shell32!CRunnableTask::Run 09 shell32!CShellTask::TT_Run 0a shell32!CShellTaskThread::ThreadProc 0b shell32!CShellTaskThread::s_ThreadProc 0c shlwapi!ExecuteWorkItemThreadProc 0d ntdll!RtlpTpWorkCallback 0e ntdll!TppWorkerThread 0f kernel32!BaseThreadInitThunk 10 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!SHProcessMessagesUntilEventsEx 07 shell32!CWaitTask::s_WaitBeforeCursing 08 ntdll!RtlpTpWaitCallback 09 ntdll!TppWaitpExecuteCallback 0a ntdll!TppWorkerThread 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.1310 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 10th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 kernel32!TlsGetValue 01 usp10!UspFreeMem 02 usp10!ScriptApplyDigitSubstitution 03 lpk!LpkCharsetDraw 04 lpk!LpkDrawTextEx 05 user32!DT_GetExtentMinusPrefixes 06 user32!NeedsEndEllipsis 07 user32!AddEllipsisAndDrawLine 08 user32!DrawTextExWorker 09 user32!DrawTextW 0a comctl32!CLVView::_ComputeLabelSizeWorker 0b comctl32!CLVView::v_RecomputeLabelSize 0c comctl32!CLVListView::v_DrawItem 0d comctl32!CLVDrawItemManager::DrawItem 0e comctl32!CLVDrawManager::_PaintItems 0f comctl32!CLVDrawManager::_PaintWorkArea 10 comctl32!CLVDrawManager::_OnPaintWorkAreas 11 comctl32!CLVDrawManager::_OnPaint 12 comctl32!CListView::WndProc 13 comctl32!CListView::s_WndProc 14 user32!UserCallWinProcCheckWow 15 user32!CallWindowProcAorW 16 user32!CallWindowProcW 17 comctl32!CallOriginalWndProc 18 comctl32!CallNextSubclassProc 19 comctl32!DefSubclassProc 1a shell32!DefSubclassProc 1b shell32!CListViewHost::s_ListViewSubclassWndProc 1c comctl32!CallNextSubclassProc 1d comctl32!MasterSubclassProc 1e user32!UserCallWinProcCheckWow 1f user32!DispatchClientMessage 20 user32!_fnDWORD 21 ntdll!KiUserCallbackDispatcherContinue 22 user32!NtUserDispatchMessage 23 user32!DispatchMessageWorker 24 user32!IsDialogMessageW 25 comctl32!Prop_IsDialogMessage 26 comctl32!_RealPropertySheet 27 comctl32!_PropertySheet 28 comdlg32!Print_InvokePropertySheets 29 comdlg32!PrintDlgExX 2a comdlg32!PrintDlgExW 2b notepad!GetPrinterDCviaDialog 2c notepad!PrintIt 2d notepad!NPCommand 2e notepad!NPWndProc 2f user32!UserCallWinProcCheckWow 30 user32!DispatchMessageWorker 31 notepad!WinMain 32 notepad!DisplayNonGenuineDlgWorker 33 kernel32!BaseThreadInitThunk 34 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 ntdll!ZwCreateSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtQueryKey 01 kernel32!BaseRegGetKeySemantics 02 kernel32!BaseRegGetUserAndMachineClass 03 kernel32!LocalBaseRegQueryValue 04 kernel32!RegQueryValueExW 05 shlwapi!SHRegQueryValueW 06 shlwapi!SHRegGetValueW 07 shlwapi!SHQueryValueExW 08 shell32!_GetServerInfo 09 shell32!_SHCoCreateInstance 0a shell32!CRegFolder::_CreateCachedRegFolder 0b shell32!CRegFolder::_BindToItem 0c shell32!CRegFolder::BindToObject 0d shell32!CRegFolder::_BindToItem 0e shell32!CRegFolder::BindToObject 0f shell32!SHBindToObject 10 shell32!CIconOverlayTask::InternalResumeRT 11 shell32!CRunnableTask::Run 12 shell32!CShellTask::TT_Run 13 shell32!CShellTaskThread::ThreadProc 14 shell32!CShellTaskThread::s_ThreadProc 15 shlwapi!ExecuteWorkItemThreadProc 16 ntdll!RtlpTpWorkCallback 17 ntdll!TppWorkerThread 18 kernel32!BaseThreadInitThunk 19 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 11th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 usp10!GenericEngineGetGlyphs 01 usp10!ShlShape 02 usp10!ScriptShape 03 usp10!RenderItemNoFallback 04 usp10!RenderItemWithFallback 05 usp10!RenderItem 06 usp10!ScriptStringAnalyzeGlyphs 07 usp10!ScriptStringAnalyse 08 lpk!LpkCharsetDraw 09 lpk!LpkDrawTextEx 0a user32!DT_GetExtentMinusPrefixes 0b user32!NeedsEndEllipsis 0c user32!AddEllipsisAndDrawLine 0d user32!DrawTextExWorker 0e user32!DrawTextW 0f comctl32!CLVView::_ComputeLabelSizeWorker 10 comctl32!CLVView::v_RecomputeLabelSize 11 comctl32!CLVListView::v_DrawItem 12 comctl32!CLVDrawItemManager::DrawItem 13 comctl32!CLVDrawManager::_PaintItems 14 comctl32!CLVDrawManager::_PaintWorkArea 15 comctl32!CLVDrawManager::_OnPaintWorkAreas 16 comctl32!CLVDrawManager::_OnPaint 17 comctl32!CListView::WndProc 18 comctl32!CListView::s_WndProc 19 user32!UserCallWinProcCheckWow 1a user32!CallWindowProcAorW 1b user32!CallWindowProcW 1c comctl32!CallOriginalWndProc 1d comctl32!CallNextSubclassProc 1e comctl32!DefSubclassProc 1f shell32!DefSubclassProc 20 shell32!CListViewHost::s_ListViewSubclassWndProc 21 comctl32!CallNextSubclassProc 22 comctl32!MasterSubclassProc 23 user32!UserCallWinProcCheckWow 24 user32!DispatchClientMessage 25 user32!_fnDWORD 26 ntdll!KiUserCallbackDispatcherContinue 27 user32!NtUserDispatchMessage 28 user32!DispatchMessageWorker 29 user32!IsDialogMessageW 2a comctl32!Prop_IsDialogMessage 2b comctl32!_RealPropertySheet 2c comctl32!_PropertySheet 2d comdlg32!Print_InvokePropertySheets 2e comdlg32!PrintDlgExX 2f comdlg32!PrintDlgExW 30 notepad!GetPrinterDCviaDialog 31 notepad!PrintIt 32 notepad!NPCommand 33 notepad!NPWndProc 34 user32!UserCallWinProcCheckWow 35 user32!DispatchMessageWorker 36 notepad!WinMain 37 notepad!DisplayNonGenuineDlgWorker 38 kernel32!BaseThreadInitThunk 39 ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 ntdll!RtlpWaitOnCriticalSection 02 ntdll!RtlEnterCriticalSection 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForSingleObject 01 ntdll!RtlpWaitOnCriticalSection 02 ntdll!RtlEnterCriticalSection 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 user32!NtUserGetIconInfo 01 user32!CopyIcoCur 02 user32!InternalCopyImage 03 user32!CopyImage 04 comctl32!CImageList::_ReplaceIcon 05 comctl32!CImageList::ReplaceIcon 06 comctl32!CSparseImageList::ReplaceIcon 07 shell32!CIconCache::AddToBackIconTable 08 shell32!AddToBackIconTable 09 shell32!SHAddIconsToCache 0a shell32!_GetILIndexGivenPXIcon 0b shell32!_GetILIndexFromItem 0c shell32!SHGetIconIndexFromPIDL 0d shell32!MapIDListToIconILIndex 0e shell32!CLoadSystemIconTask::InternalResumeRT 0f shell32!CRunnableTask::Run 10 shell32!CShellTask::TT_Run 11 shell32!CShellTaskThread::ThreadProc 12 shell32!CShellTaskThread::s_ThreadProc 13 shlwapi!ExecuteWorkItemThreadProc 14 ntdll!RtlpTpWorkCallback 15 ntdll!TppWorkerThread 16 kernel32!BaseThreadInitThunk 17 ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 gdi32!NtUserSelectPalette 01 gdi32!SelectPalette 02 gdi32!SetDIBits 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwDelayExecution 01 KERNELBASE!SleepEx 02 ole32!CROIDTable::WorkerThreadLoop 03 ole32!CRpcThread::WorkerLoop 04 ole32!CRpcThreadCache::RpcWorkerThreadEntry 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CCALL::SendReceive 02 rpcrt4!NdrpClientCall3 03 rpcrt4!NdrClientCall3 04 winspool!RpcFindNextPrinterChangeNotification 05 winspool!FindNextPrinterChangeNotification 06 prncache!PrintCache::Listeners::Listener::ProcessWait 07 prncache!PrintCache::Listeners::Listener::ProcessWaitCB 08 ntdll!TppWaitpExecuteCallback 09 ntdll!TppWorkerThread 0a kernel32!BaseThreadInitThunk 0b ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

// 12th dump

. 0 Id: 1344.1ca4 Suspend: 1 Teb: 000007ff`fffdd000 Unfrozen # Call Site 00 usp10!otlChainingLookup::apply 01 usp10!ApplyLookup 02 usp10!ApplyFeatures 03 usp10!SubstituteOtlGlyphs 04 usp10!GenericEngineGetGlyphs 05 usp10!ShlShape 06 usp10!ScriptShape 07 usp10!RenderItemNoFallback 08 usp10!RenderItemWithFallback 09 usp10!RenderItem 0a usp10!ScriptStringAnalyzeGlyphs 0b usp10!ScriptStringAnalyse 0c lpk!LpkCharsetDraw 0d lpk!LpkDrawTextEx 0e user32!DT_DrawStr 0f user32!DT_DrawJustifiedLine 10 user32!DrawTextExWorker 11 user32!DrawTextW 12 comctl32!CLVView::_ComputeLabelSizeWorker 13 comctl32!CLVView::v_RecomputeLabelSize 14 comctl32!CLVListView::v_DrawItem 15 comctl32!CLVDrawItemManager::DrawItem 16 comctl32!CLVDrawManager::_PaintItems 17 comctl32!CLVDrawManager::_PaintWorkArea 18 comctl32!CLVDrawManager::_OnPaintWorkAreas 19 comctl32!CLVDrawManager::_OnPaint 1a comctl32!CListView::WndProc 1b comctl32!CListView::s_WndProc 1c user32!UserCallWinProcCheckWow 1d user32!CallWindowProcAorW 1e user32!CallWindowProcW 1f comctl32!CallOriginalWndProc 20 comctl32!CallNextSubclassProc 21 comctl32!DefSubclassProc 22 shell32!DefSubclassProc 23 shell32!CListViewHost::s_ListViewSubclassWndProc 24 comctl32!CallNextSubclassProc 25 comctl32!MasterSubclassProc 26 user32!UserCallWinProcCheckWow 27 user32!DispatchClientMessage 28 user32!_fnDWORD 29 ntdll!KiUserCallbackDispatcherContinue 2a user32!NtUserDispatchMessage 2b user32!DispatchMessageWorker 2c user32!IsDialogMessageW 2d comctl32!Prop_IsDialogMessage 2e comctl32!_RealPropertySheet 2f comctl32!_PropertySheet 30 comdlg32!Print_InvokePropertySheets 31 comdlg32!PrintDlgExX 32 comdlg32!PrintDlgExW 33 notepad!GetPrinterDCviaDialog 34 notepad!PrintIt 35 notepad!NPCommand 36 notepad!NPWndProc 37 user32!UserCallWinProcCheckWow 38 user32!DispatchMessageWorker 39 notepad!WinMain 3a notepad!DisplayNonGenuineDlgWorker 3b kernel32!BaseThreadInitThunk 3c ntdll!RtlUserThreadStart

1 Id: 1344.1ab0 Suspend: 1 Teb: 000007ff`fffdb000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 ntdll!TppWaiterpThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

2 Id: 1344.1638 Suspend: 1 Teb: 000007ff`fffd9000 Unfrozen # Call Site 00 ntdll!NtUnmapViewOfSection 01 KERNELBASE!FreeLibrary 02 user32!PrivateExtractIconsW 03 shell32!SHPrivateExtractIcons 04 shell32!SHDefExtractIconW 05 shell32!CExtractIcon::_ExtractW 06 shell32!CExtractIconBase::Extract 07 shell32!IExtractIcon_Extract 08 shell32!_GetILIndexGivenPXIcon 09 shell32!_GetILIndexFromItem 0a shell32!SHGetIconIndexFromPIDL 0b shell32!MapIDListToIconILIndex 0c shell32!CLoadSystemIconTask::InternalResumeRT 0d shell32!CRunnableTask::Run 0e shell32!CShellTask::TT_Run 0f shell32!CShellTaskThread::ThreadProc 10 shell32!CShellTaskThread::s_ThreadProc 11 shlwapi!ExecuteWorkItemThreadProc 12 ntdll!RtlpTpWorkCallback 13 ntdll!TppWorkerThread 14 kernel32!BaseThreadInitThunk 15 ntdll!RtlUserThreadStart

3 Id: 1344.830 Suspend: 1 Teb: 000007ff`fffd7000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjects 03 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChanges 04 prncache!PrintCache::Store::CacheStore::RegistryMonitor::MonitorRegistryChangesThreadProc 05 kernel32!BaseThreadInitThunk 06 ntdll!RtlUserThreadStart

4 Id: 1344.1edc Suspend: 1 Teb: 000007ff`fffd5000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

5 Id: 1344.1b44 Suspend: 1 Teb: 000007ff`fffd3000 Unfrozen # Call Site 00 ntdll!NtWaitForMultipleObjects 01 KERNELBASE!WaitForMultipleObjectsEx 02 kernel32!WaitForMultipleObjectsExImplementation 03 user32!RealMsgWaitForMultipleObjectsEx 04 user32!MsgWaitForMultipleObjectsEx 05 user32!MsgWaitForMultipleObjects 06 shell32!CShellTaskScheduler::_TT_MsgWaitForMultipleObjects 07 shell32!CShellTaskScheduler::TT_TransitionThreadToRunningOrTerminating 08 shell32!CShellTaskThread::ThreadProc 09 shell32!CShellTaskThread::s_ThreadProc 0a shlwapi!ExecuteWorkItemThreadProc 0b ntdll!RtlpTpWorkCallback 0c ntdll!TppWorkerThread 0d kernel32!BaseThreadInitThunk 0e ntdll!RtlUserThreadStart

6 Id: 1344.1d9c Suspend: 1 Teb: 000007ff`fffae000 Unfrozen # Call Site 00 gdi32!ZwGdiSetDIBitsToDeviceInternal 01 gdi32!SetDIBitsToDevice 02 gdi32!SetDIBits 03 user32!BitmapFromDIB 04 user32!ConvertDIBBitmap 05 user32!ConvertDIBIcon 06 user32!CreateIconFromResourceEx 07 user32!PrivateEnumProc 08 kernel32!EnumResourceNamesInternal 09 kernel32!EnumResourceNamesExW 0a user32!PrivateExtractIconsW 0b shell32!SHPrivateExtractIcons 0c shell32!SHDefExtractIconW 0d shell32!CExtractIcon::_ExtractW 0e shell32!CExtractIconBase::Extract 0f shell32!IExtractIcon_Extract 10 shell32!_GetILIndexGivenPXIcon 11 shell32!_GetILIndexFromItem 12 shell32!SHGetIconIndexFromPIDL 13 shell32!MapIDListToIconILIndex 14 shell32!CLoadSystemIconTask::InternalResumeRT 15 shell32!CRunnableTask::Run 16 shell32!CShellTask::TT_Run 17 shell32!CShellTaskThread::ThreadProc 18 shell32!CShellTaskThread::s_ThreadProc 19 shlwapi!ExecuteWorkItemThreadProc 1a ntdll!RtlpTpWorkCallback 1b ntdll!TppWorkerThread 1c kernel32!BaseThreadInitThunk 1d ntdll!RtlUserThreadStart

7 Id: 1344.139c Suspend: 1 Teb: 000007ff`fffac000 Unfrozen # Call Site 00 ntdll!ZwMapViewOfSection 01 KERNELBASE!BasepLoadLibraryAsDataFileInternal 02 KERNELBASE!LoadLibraryExW 03 user32!PrivateExtractIconsW 04 shell32!SHPrivateExtractIcons 05 shell32!SHDefExtractIconW 06 shell32!CExtractIcon::_ExtractW 07 shell32!CExtractIconBase::Extract 08 shell32!IExtractIcon_Extract 09 shell32!_GetILIndexGivenPXIcon 0a shell32!_GetILIndexFromItem 0b shell32!SHGetIconIndexFromPIDL 0c shell32!MapIDListToIconILIndex 0d shell32!CLoadSystemIconTask::InternalResumeRT 0e shell32!CRunnableTask::Run 0f shell32!CShellTask::TT_Run 10 shell32!CShellTaskThread::ThreadProc 11 shell32!CShellTaskThread::s_ThreadProc 12 shlwapi!ExecuteWorkItemThreadProc 13 ntdll!RtlpTpWorkCallback 14 ntdll!TppWorkerThread 15 kernel32!BaseThreadInitThunk 16 ntdll!RtlUserThreadStart

8 Id: 1344.1b80 Suspend: 1 Teb: 000007ff`fffaa000 Unfrozen # Call Site 00 ntdll!ZwAlpcSendWaitReceivePort 01 rpcrt4!LRPC_CASSOCIATION::AlpcSendWaitReceivePort 02 rpcrt4!LRPC_BASE_CCALL::DoSendReceive 03 rpcrt4!LRPC_BASE_CCALL::SendReceive 04 rpcrt4!NdrpClientCall2 05 rpcrt4!NdrClientCall2 06 ole32!CRpcResolver::BulkUpdateOIDs 07 ole32!CROIDTable::ClientBulkUpdateOIDWithPingServer 08 ole32!CROIDTable::WorkerThreadLoop 09 ole32!CRpcThread::WorkerLoop 0a ole32!CRpcThreadCache::RpcWorkerThreadEntry 0b kernel32!BaseThreadInitThunk 0c ntdll!RtlUserThreadStart

9 Id: 1344.eb8 Suspend: 1 Teb: 000007ff`fffa8000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

10 Id: 1344.f98 Suspend: 1 Teb: 000007ff`fffa6000 Unfrozen # Call Site 00 ntdll!NtWaitForWorkViaWorkerFactory 01 ntdll!TppWorkerThread 02 kernel32!BaseThreadInitThunk 03 ntdll!RtlUserThreadStart

11 Id: 1344.1fb0 Suspend: 1 Teb: 000007ff`fffa4000 Unfrozen # Call Site 00 ntdll!RtlUserThreadStart

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, Software Trace Analysis, Trace Analysis Patterns, x64 Windows | No Comments »

Spiking Thread, Top Module, Module Hint, and Memory Fluctuation: pattern cooperation

Saturday, October 24th, 2015

We noticed that after restarting Windows 7 system on one of our notebooks it becomes very sluggish. Task Manager showed 25% CPU usage in one of svchost.exe processes and very high usage of physical memory. So we immediately dumped it using procdump. The resulted process memory dump was almost 1.5 GB. Although the analysis case is very simple and straightforward we decided to publish to show the value of crash and hang dump analysis in understanding abnormal software behavior in “user” context.

When we open the memory dump and run !runaway WinDbg command we immediately recognize Spiking Thread pattern:

0:000> !runaway f User Mode Time Thread Time 38:14a0 0 days 0:04:59.911 36:a88 0 days 0:00:00.187 2:41c 0 days 0:00:00.187 39:19f8 0 days 0:00:00.046 34:fa8 0 days 0:00:00.046 21:12b4 0 days 0:00:00.031 43:8f0 0 days 0:00:00.015 42:1504 0 days 0:00:00.015 41:1a20 0 days 0:00:00.015 40:978 0 days 0:00:00.015 33:e0c 0 days 0:00:00.015 32:ff8 0 days 0:00:00.015 22:1304 0 days 0:00:00.015 19:f68 0 days 0:00:00.015 9:664 0 days 0:00:00.015 8:660 0 days 0:00:00.015 6:518 0 days 0:00:00.015 4:4a4 0 days 0:00:00.015 51:160c 0 days 0:00:00.000 50:1590 0 days 0:00:00.000 49:15d8 0 days 0:00:00.000 48:ac8 0 days 0:00:00.000 47:14d0 0 days 0:00:00.000 46:1bfc 0 days 0:00:00.000 45:18e8 0 days 0:00:00.000 44:1448 0 days 0:00:00.000 37:1910 0 days 0:00:00.000 35:558 0 days 0:00:00.000 31:14b8 0 days 0:00:00.000 30:14b4 0 days 0:00:00.000 29:14ac 0 days 0:00:00.000 28:13d0 0 days 0:00:00.000 27:13c8 0 days 0:00:00.000 26:13b4 0 days 0:00:00.000 25:13b0 0 days 0:00:00.000 24:13a8 0 days 0:00:00.000 23:1328 0 days 0:00:00.000 20:12b0 0 days 0:00:00.000 18:e20 0 days 0:00:00.000 17:e10 0 days 0:00:00.000 16:dd0 0 days 0:00:00.000 15:ce0 0 days 0:00:00.000 14:754 0 days 0:00:00.000 13:718 0 days 0:00:00.000 12:678 0 days 0:00:00.000 11:674 0 days 0:00:00.000 10:668 0 days 0:00:00.000 7:548 0 days 0:00:00.000 5:4ac 0 days 0:00:00.000 3:4a0 0 days 0:00:00.000 1:418 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Kernel Mode Time Thread Time 38:14a0 0 days 0:00:55.707 36:a88 0 days 0:00:01.778 2:41c 0 days 0:00:00.405 34:fa8 0 days 0:00:00.109 9:664 0 days 0:00:00.062 43:8f0 0 days 0:00:00.046 42:1504 0 days 0:00:00.046 21:12b4 0 days 0:00:00.046 32:ff8 0 days 0:00:00.031 22:1304 0 days 0:00:00.031 18:e20 0 days 0:00:00.031 39:19f8 0 days 0:00:00.015 19:f68 0 days 0:00:00.015 6:518 0 days 0:00:00.015 3:4a0 0 days 0:00:00.015 1:418 0 days 0:00:00.015 51:160c 0 days 0:00:00.000 50:1590 0 days 0:00:00.000 49:15d8 0 days 0:00:00.000 48:ac8 0 days 0:00:00.000 47:14d0 0 days 0:00:00.000 46:1bfc 0 days 0:00:00.000 45:18e8 0 days 0:00:00.000 44:1448 0 days 0:00:00.000 41:1a20 0 days 0:00:00.000 40:978 0 days 0:00:00.000 37:1910 0 days 0:00:00.000 35:558 0 days 0:00:00.000 33:e0c 0 days 0:00:00.000 31:14b8 0 days 0:00:00.000 30:14b4 0 days 0:00:00.000 29:14ac 0 days 0:00:00.000 28:13d0 0 days 0:00:00.000 27:13c8 0 days 0:00:00.000 26:13b4 0 days 0:00:00.000 25:13b0 0 days 0:00:00.000 24:13a8 0 days 0:00:00.000 23:1328 0 days 0:00:00.000 20:12b0 0 days 0:00:00.000 17:e10 0 days 0:00:00.000 16:dd0 0 days 0:00:00.000 15:ce0 0 days 0:00:00.000 14:754 0 days 0:00:00.000 13:718 0 days 0:00:00.000 12:678 0 days 0:00:00.000 11:674 0 days 0:00:00.000 10:668 0 days 0:00:00.000 8:660 0 days 0:00:00.000 7:548 0 days 0:00:00.000 5:4ac 0 days 0:00:00.000 4:4a4 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Elapsed Time Thread Time 0:410 0 days 0:10:24.550 2:41c 0 days 0:10:24.534 1:418 0 days 0:10:24.534 4:4a4 0 days 0:10:24.331 3:4a0 0 days 0:10:24.331 5:4ac 0 days 0:10:24.269 6:518 0 days 0:10:23.957 7:548 0 days 0:10:23.817 8:660 0 days 0:10:22.176 9:664 0 days 0:10:22.156 10:668 0 days 0:10:22.126 11:674 0 days 0:10:22.026 12:678 0 days 0:10:21.986 13:718 0 days 0:10:20.066 14:754 0 days 0:10:20.056 15:ce0 0 days 0:10:15.131 16:dd0 0 days 0:10:14.641 17:e10 0 days 0:10:14.551 18:e20 0 days 0:10:14.531 19:f68 0 days 0:10:13.611 21:12b4 0 days 0:10:10.647 20:12b0 0 days 0:10:10.647 22:1304 0 days 0:10:10.553 23:1328 0 days 0:10:10.381 24:13a8 0 days 0:10:09.024 26:13b4 0 days 0:10:08.931 25:13b0 0 days 0:10:08.931 28:13d0 0 days 0:10:08.899 27:13c8 0 days 0:10:08.899 31:14b8 0 days 0:10:07.932 30:14b4 0 days 0:10:07.932 29:14ac 0 days 0:10:07.932 32:ff8 0 days 0:08:11.785 33:e0c 0 days 0:08:11.644 34:fa8 0 days 0:08:06.750 35:558 0 days 0:08:05.765 36:a88 0 days 0:08:05.127 37:1910 0 days 0:08:02.608 38:14a0 0 days 0:07:19.276 42:1504 0 days 0:04:55.634 41:1a20 0 days 0:04:55.634 40:978 0 days 0:04:55.634 39:19f8 0 days 0:04:55.634 43:8f0 0 days 0:04:55.618 44:1448 0 days 0:04:42.634 46:1bfc 0 days 0:04:20.945 45:18e8 0 days 0:04:20.945 47:14d0 0 days 0:02:42.515 48:ac8 0 days 0:01:27.434 50:1590 0 days 0:00:04.917 49:15d8 0 days 0:00:04.917 51:160c 0 days 0:00:03.387

We see that most of the time the thread was consuming CPU in user mode and less in kernel mode.

We then examine that thread stack trace:

0:000> ~38k # Child-SP RetAddr Call Site 00 00000000`0414ddd8 00000000`7769cf66 ntdll!ZwQueryPerformanceCounter+0xa *** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll - 01 00000000`0414dde0 000007fe`ee7ee7aa ntdll!RtlQueryPerformanceFrequency+0x16 02 00000000`0414de10 000007fe`ee7ee53f wuaueng!DllInstall+0×153da 03 00000000`0414de40 000007fe`ee7e0d99 wuaueng!DllInstall+0×1516f 04 00000000`0414e190 000007fe`ee7df542 wuaueng!DllInstall+0×79c9 05 00000000`0414e1f0 000007fe`ee7df57c wuaueng!DllInstall+0×6172 06 00000000`0414e2b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 07 00000000`0414e370 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 08 00000000`0414e430 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 09 00000000`0414e4f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0a 00000000`0414e5b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0b 00000000`0414e670 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0c 00000000`0414e730 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0d 00000000`0414e7f0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0e 00000000`0414e8b0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 0f 00000000`0414e970 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 10 00000000`0414ea30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 11 00000000`0414eaf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 12 00000000`0414ebb0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 13 00000000`0414ec70 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 14 00000000`0414ed30 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 15 00000000`0414edf0 000007fe`ee7df57c wuaueng!DllInstall+0×61ac 16 00000000`0414eeb0 000007fe`ee7df2d7 wuaueng!DllInstall+0×61ac 17 00000000`0414ef70 000007fe`ee7e9b4f wuaueng!DllInstall+0×5f07 18 00000000`0414f020 000007fe`ee7eb7e8 wuaueng!DllInstall+0×1077f 19 00000000`0414f370 000007fe`ee8010b2 wuaueng!DllInstall+0×12418 1a 00000000`0414f5b0 000007fe`ee7fe53e wuaueng!DllInstall+0×27ce2 1b 00000000`0414f610 000007fe`ee7fccac wuaueng!DllInstall+0×2516e 1c 00000000`0414f660 000007fe`ee7dec19 wuaueng!DllInstall+0×238dc 1d 00000000`0414f690 000007fe`ee7de30f wuaueng!DllInstall+0×5849 1e 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0×4f3f 1f 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd 20 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see that Top Module is from Windows Update Agent:

0:000> lmv m wuaueng Browse full module list start end module name 000007fe`ee740000 000007fe`ee9a0000 wuaueng (export symbols) wuaueng.dll Loaded symbol image file: wuaueng.dll Image path: c:\Windows\System32\wuaueng.dll Image name: wuaueng.dll Browse all global symbols functions data Timestamp: Wed May 14 17:21:24 2014 (53739804) CheckSum: 00265DEA ImageSize: 00260000 File version: 7.6.7600.320 Product version: 7.6.7600.320 File flags: 8 (Mask 3F) Private File OS: 40004 NT Win32 File type: 2.0 Dll File date: 00000000.00000000 Translations: 0409.04b0 CompanyName: Microsoft Corporation ProductName: Microsoft® Windows® Operating System InternalName: wuaueng.dll OriginalFilename: wuaueng.dll ProductVersion: 7.6.7600.320 FileVersion: 7.6.7600.320 (winmain_wtr_wsus3sp2(oobla).140514-0916) FileDescription: Windows Update Agent LegalCopyright: © Microsoft Corporation. All rights reserved.

We also examine process address space:

0:000> !address -summary

Mapping file section regions... Mapping module regions... Mapping PEB regions... Mapping TEB and stack regions... Mapping heap regions... Mapping page heap regions... Mapping other regions... Mapping stack trace database regions... Mapping activation context regions...

--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 507 7ff`9373b000 ( 7.998 TB) 99.98% Heap 574 0`3e2ad000 ( 994.676 MB) 57.27% 0.01% 9786 0`2803e000 ( 640.242 MB) 36.87% 0.01% Image 874 0`049a6000 ( 73.648 MB) 4.24% 0.00% Stack 156 0`01a00000 ( 26.000 MB) 1.50% 0.00% Other 13 0`001bb000 ( 1.730 MB) 0.10% 0.00% TEB 52 0`00068000 ( 416.000 kB) 0.02% 0.00% PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00%

--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_PRIVATE 834 0`5345e000 ( 1.301 GB) 76.72% 0.02% MEM_MAPPED 9747 0`14ab0000 ( 330.688 MB) 19.04% 0.00% MEM_IMAGE 875 0`049a7000 ( 73.652 MB) 4.24% 0.00%

--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal MEM_FREE 507 7ff`9373b000 ( 7.998 TB) 99.98% MEM_COMMIT 11164 0`513d8000 ( 1.269 GB) 74.85% 0.02% MEM_RESERVE 292 0`1b4dd000 ( 436.863 MB) 25.15% 0.01%

--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal PAGE_READWRITE 5533 0`41d6c000 ( 1.029 GB) 60.66% 0.01% PAGE_WRITECOPY 4911 0`098b9000 ( 152.723 MB) 8.79% 0.00% PAGE_READONLY 504 0`03204000 ( 50.016 MB) 2.88% 0.00% PAGE_EXECUTE_READ 164 0`02b29000 ( 43.160 MB) 2.49% 0.00% PAGE_READWRITE|PAGE_GUARD 52 0`00086000 ( 536.000 kB) 0.03% 0.00%

--- Largest Region by Usage ----------- Base Address -------- Region Size ---------- Free 0`ff84b000 7fd`eed65000 ( 7.992 TB) Heap 0`83eb0000 0`01c17000 ( 28.090 MB) 0`93070000 0`0ffbd000 ( 255.738 MB) Image 7fe`fe18a000 0`0089e000 ( 8.617 MB) Stack 0`07140000 0`0007b000 ( 492.000 kB) Other 0`00760000 0`00181000 ( 1.504 MB) TEB 7ff`fff0a000 0`00002000 ( 8.000 kB) PEB 7ff`fffdf000 0`00001000 ( 4.000 kB)

We see that process heap occupies almost 1 GB. Let’s see its statistics:

0:000> !heap -s

****************************** NT HEAP STATS BELOW ****************************** LFH Key : 0x000000a57ddeb5ed Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ------------------------------------------------------------------------------------- Virtual block: 0000000051eb0000 - 0000000051eb0000 (size 0000000000000000) Virtual block: 0000000052930000 - 0000000052930000 (size 0000000000000000) Virtual block: 0000000026ba0000 - 0000000026ba0000 (size 0000000000000000) Virtual block: 0000000053e50000 - 0000000053e50000 (size 0000000000000000) Virtual block: 0000000054490000 - 0000000054490000 (size 0000000000000000) Virtual block: 00000000547a0000 - 00000000547a0000 (size 0000000000000000) Virtual block: 000000003bda0000 - 000000003bda0000 (size 0000000000000000) Virtual block: 0000000056030000 - 0000000056030000 (size 0000000000000000) Virtual block: 00000000567c0000 - 00000000567c0000 (size 0000000000000000) Virtual block: 00000000572a0000 - 00000000572a0000 (size 0000000000000000) Virtual block: 0000000057870000 - 0000000057870000 (size 0000000000000000) Virtual block: 0000000045a10000 - 0000000045a10000 (size 0000000000000000) Virtual block: 0000000058fb0000 - 0000000058fb0000 (size 0000000000000000) Virtual block: 0000000045c10000 - 0000000045c10000 (size 0000000000000000) Virtual block: 00000000599c0000 - 00000000599c0000 (size 0000000000000000) Virtual block: 0000000059ff0000 - 0000000059ff0000 (size 0000000000000000) Virtual block: 000000005ae20000 - 000000005ae20000 (size 0000000000000000) Virtual block: 000000005c5d0000 - 000000005c5d0000 (size 0000000000000000) Virtual block: 0000000054b90000 - 0000000054b90000 (size 0000000000000000) Virtual block: 000000005d070000 - 000000005d070000 (size 0000000000000000) Virtual block: 000000006e370000 - 000000006e370000 (size 0000000000000000) Virtual block: 000000006f8e0000 - 000000006f8e0000 (size 0000000000000000) Virtual block: 000000006ed20000 - 000000006ed20000 (size 0000000000000000) Virtual block: 0000000070890000 - 0000000070890000 (size 0000000000000000) Virtual block: 000000005e370000 - 000000005e370000 (size 0000000000000000) Virtual block: 000000005f5a0000 - 000000005f5a0000 (size 0000000000000000) Virtual block: 000000005fa60000 - 000000005fa60000 (size 0000000000000000) Virtual block: 000000005ffe0000 - 000000005ffe0000 (size 0000000000000000) Virtual block: 0000000060770000 - 0000000060770000 (size 0000000000000000) Virtual block: 0000000060aa0000 - 0000000060aa0000 (size 0000000000000000) Virtual block: 0000000061810000 - 0000000061810000 (size 0000000000000000) Virtual block: 0000000061a30000 - 0000000061a30000 (size 0000000000000000) Virtual block: 0000000061f00000 - 0000000061f00000 (size 0000000000000000) Virtual block: 0000000064470000 - 0000000064470000 (size 0000000000000000) Virtual block: 0000000064c00000 - 0000000064c00000 (size 0000000000000000) Virtual block: 00000000656d0000 - 00000000656d0000 (size 0000000000000000) Virtual block: 00000000660f0000 - 00000000660f0000 (size 0000000000000000) Virtual block: 0000000066530000 - 0000000066530000 (size 0000000000000000) Virtual block: 00000000669d0000 - 00000000669d0000 (size 0000000000000000) Virtual block: 00000000676a0000 - 00000000676a0000 (size 0000000000000000) Virtual block: 0000000067a70000 - 0000000067a70000 (size 0000000000000000) Virtual block: 0000000068a20000 - 0000000068a20000 (size 0000000000000000) Virtual block: 0000000069f10000 - 0000000069f10000 (size 0000000000000000) Virtual block: 000000006a6c0000 - 000000006a6c0000 (size 0000000000000000) Virtual block: 000000006ad80000 - 000000006ad80000 (size 0000000000000000) Virtual block: 000000006b9a0000 - 000000006b9a0000 (size 0000000000000000) Virtual block: 000000006bb40000 - 000000006bb40000 (size 0000000000000000) Virtual block: 000000006c4f0000 - 000000006c4f0000 (size 0000000000000000) Virtual block: 000000006dc30000 - 000000006dc30000 (size 0000000000000000) Virtual block: 000000006de10000 - 000000006de10000 (size 0000000000000000) Virtual block: 000000006ef80000 - 000000006ef80000 (size 0000000000000000) Virtual block: 00000000728f0000 - 00000000728f0000 (size 0000000000000000) Virtual block: 0000000071270000 - 0000000071270000 (size 0000000000000000) Virtual block: 0000000074030000 - 0000000074030000 (size 0000000000000000) Virtual block: 00000000746c0000 - 00000000746c0000 (size 0000000000000000) Virtual block: 00000000749d0000 - 00000000749d0000 (size 0000000000000000) Virtual block: 0000000074e50000 - 0000000074e50000 (size 0000000000000000) Virtual block: 0000000071730000 - 0000000071730000 (size 0000000000000000) Virtual block: 0000000075c30000 - 0000000075c30000 (size 0000000000000000) Virtual block: 0000000075e00000 - 0000000075e00000 (size 0000000000000000) Virtual block: 0000000075fe0000 - 0000000075fe0000 (size 0000000000000000) Virtual block: 00000000761a0000 - 00000000761a0000 (size 0000000000000000) Virtual block: 0000000076460000 - 0000000076460000 (size 0000000000000000) Virtual block: 0000000076620000 - 0000000076620000 (size 0000000000000000) Virtual block: 0000000077850000 - 0000000077850000 (size 0000000000000000) Virtual block: 0000000076c60000 - 0000000076c60000 (size 0000000000000000) Virtual block: 0000000078ec0000 - 0000000078ec0000 (size 0000000000000000) Virtual block: 0000000062b30000 - 0000000062b30000 (size 0000000000000000) Virtual block: 000000007a520000 - 000000007a520000 (size 0000000000000000) Virtual block: 000000007ab50000 - 000000007ab50000 (size 0000000000000000) Virtual block: 00000000770e0000 - 00000000770e0000 (size 0000000000000000) Virtual block: 000000007af00000 - 000000007af00000 (size 0000000000000000) Virtual block: 000000007c6b0000 - 000000007c6b0000 (size 0000000000000000) Virtual block: 000000007de50000 - 000000007de50000 (size 0000000000000000) Virtual block: 000000007e2d0000 - 000000007e2d0000 (size 0000000000000000) Virtual block: 000000007fff0000 - 000000007fff0000 (size 0000000000000000) Virtual block: 00000000628c0000 - 00000000628c0000 (size 0000000000000000) Virtual block: 00000000809b0000 - 00000000809b0000 (size 0000000000000000) Virtual block: 0000000080d80000 - 0000000080d80000 (size 0000000000000000) Virtual block: 0000000081310000 - 0000000081310000 (size 0000000000000000) Virtual block: 00000000631a0000 - 00000000631a0000 (size 0000000000000000) Virtual block: 0000000062c70000 - 0000000062c70000 (size 0000000000000000) Virtual block: 0000000092ca0000 - 0000000092ca0000 (size 0000000000000000) Virtual block: 0000000003940000 - 0000000003940000 (size 0000000000000000) Virtual block: 0000000081cd0000 - 0000000081cd0000 (size 0000000000000000) Virtual block: 0000000082260000 - 0000000082260000 (size 0000000000000000) Virtual block: 0000000077200000 - 0000000077200000 (size 0000000000000000) Virtual block: 000000007ec90000 - 000000007ec90000 (size 0000000000000000) Virtual block: 000000007edb0000 - 000000007edb0000 (size 0000000000000000) Virtual block: 000000007eec0000 - 000000007eec0000 (size 0000000000000000) Virtual block: 00000000829f0000 - 00000000829f0000 (size 0000000000000000) Virtual block: 0000000003770000 - 0000000003770000 (size 0000000000000000) Virtual block: 0000000082fc0000 - 0000000082fc0000 (size 0000000000000000) Virtual block: 0000000083210000 - 0000000083210000 (size 0000000000000000) Virtual block: 0000000083320000 - 0000000083320000 (size 0000000000000000) Virtual block: 0000000083760000 - 0000000083760000 (size 0000000000000000) Virtual block: 0000000083870000 - 0000000083870000 (size 0000000000000000) Virtual block: 0000000083eb0000 - 0000000083eb0000 (size 0000000000000000) Virtual block: 0000000085ad0000 - 0000000085ad0000 (size 0000000000000000) Virtual block: 0000000086290000 - 0000000086290000 (size 0000000000000000) 0000000000340000 00000002 210688 174796 210688 60711 775 45 100 1 LFH External fragmentation 34 % (775 free blocks) 0000000000010000 00008000 64 4 64 1 1 1 0 0 00000000005c0000 00001002 3136 1256 3136 278 39 3 0 0 LFH External fragmentation 22 % (39 free blocks) 0000000000960000 00001002 512 8 512 2 1 1 0 0 0000000001520000 00001002 512 276 512 2 10 1 0 0 LFH 00000000013f0000 00001002 512 8 512 2 1 1 0 0 0000000001790000 00001002 512 248 512 0 6 1 0 0 LFH 00000000010b0000 00001002 512 8 512 2 1 1 0 0 00000000019b0000 00001002 512 308 512 5 8 1 0 0 LFH 0000000001220000 00001002 512 380 512 63 21 1 0 0 LFH 00000000021e0000 00001002 512 256 512 11 3 1 0 0 LFH 0000000001900000 00001002 512 156 512 133 1 1 0 0 00000000024b0000 00001002 1536 792 1536 27 4 2 0 0 LFH 0000000002e00000 00001002 48256 16656 48256 9836 98 13 0 1 LFH External fragmentation 59 % (98 free blocks) 0000000002f10000 00001002 3584 3280 3584 1773 79 3 0 5 LFH External fragmentation 54 % (79 free blocks) 00000000035c0000 00001002 64 8 64 3 1 1 0 0 0000000003200000 00001002 512 388 512 2 3 1 0 0 LFH 00000000038c0000 00001002 512 8 512 3 1 1 0 0 0000000003ad0000 00001002 512 8 512 3 1 1 0 0 0000000003d40000 00001002 512 8 512 3 1 1 0 0 00000000005b0000 00001002 64 8 64 3 1 1 0 0 0000000004280000 00011002 512 64 512 59 2 1 0 0 00000000040c0000 00001002 1088 296 1088 14 6 2 0 0 LFH 0000000004270000 00001002 1088 304 1088 21 5 2 0 0 LFH 0000000006650000 00000002 1088 80 1088 10 2 2 0 0 0000000007050000 00001002 512 8 512 3 1 1 0 0 0000000007cc0000 00001002 512 8 512 3 1 1 0 0 0000000000be0000 00001002 64 24 64 17 2 1 0 0 0000000006540000 00001002 1536 544 1536 259 4 2 0 0 LFH 0000000000b40000 00001002 64 8 64 3 1 1 0 0 0000000004950000 00001002 512 8 512 3 1 1 0 0 ————————————————————————————-

We see many large virtual blocks allocated with less than 200 KB of normal blocks committed. We suspect possible Memory Leak. Let’s check a few such virtual blocks:

0:000> !address 0000000062b30000

Usage: Heap Base Address: 00000000`62b30000 End Address: 00000000`62c65000 Region Size: 00000000`00135000 ( 1.207 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`62b30000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×62b30000

Content source: 1 (target), length: 135000

0:000> !address 0000000003770000

Usage: Heap Base Address: 00000000`03770000 End Address: 00000000`03870000 Region Size: 00000000`00100000 ( 1.000 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`03770000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×3770000

Content source: 1 (target), length: 100000

0:000> !address 0000000085ad0000

Usage: Heap Base Address: 00000000`85ad0000 End Address: 00000000`8628d000 Region Size: 00000000`007bd000 ( 7.738 MB) State: 00001000 MEM_COMMIT Protect: 00000004 PAGE_READWRITE Type: 00020000 MEM_PRIVATE Allocation Base: 00000000`85ad0000 Allocation Protect: 00000004 PAGE_READWRITE More info: heap owning the address: !heap 0×340000 More info: heap large/virtual block More info: heap entry containing the address: !heap -x 0×85ad0000

Content source: 1 (target), length: 7bd000

We see they all contain assembly packages for KB articles for various updates (Module Hints):

0:000> dc 00000000`85ad0000 00000000`85ad0000 86290000 00000000 83eb0000 00000000 ..)…………. 00000000`85ad0010 00000000 00000000 00000000 00000000 ……………. 00000000`85ad0020 007bd000 00000000 007bd000 00000000 ..{…….{….. 00000000`85ad0030 00000000 00000000 f333297f 04000000 ………)3….. 00000000`85ad0040 00000001 007bc51a 00000004 00000000 ……{……… 00000000`85ad0050 0061003c 00730073 006d0065 006c0062 <.a.s.s.e.m.b.l. 00000000`85ad0060 00200079 006d0078 006e006c 003d0073 y. .x.m.l.n.s.=. 00000000`85ad0070 00750022 006e0072 0073003a 00680063 “.u.r.n.:.s.c.h.

0:000> du 00000000`85ad0050 00000000`85ad0050 “<assembly xmlns=”urn:schemas-mic” 00000000`85ad0090 “rosoft-com:asm.v3″ manifestVersi” 00000000`85ad00d0 “on=”1.0″ description=”Fix for KB” 00000000`85ad0110 “2798162″ displayName=”default” c” 00000000`85ad0150 “ompany=”Microsoft Corporation” c” 00000000`85ad0190 “opyright=”Microsoft Corporation”" 00000000`85ad01d0 ” supportInformation=”http://supp” 00000000`85ad0210 “ort.microsoft.com/?kbid=2798162″” 00000000`85ad0250 ” creationTimeStamp=”2013-04-15T0″ 00000000`85ad0290 “4:10:39Z” lastUpdateTimeStamp=”2″ 00000000`85ad02d0 “013-04-15T04:10:39Z”><assemblyid” 00000000`85ad0310 “entity name=”Package_for_KB27981″

However, in less than 20 minutes CPU and memory consumption normalized and the new saved process memory dump was less than 150 KB. We open it and see that CPU consumption happened for about 15 minutes (in the past) with the consuming thread now #32 instead of #38:

0:000> !runaway f User Mode Time Thread Time 32:14a0 0 days 0:13:24.029 33:b6c 0 days 0:00:00.078 41:174c 0 days 0:00:00.062 40:1bd8 0 days 0:00:00.062 30:fa8 0 days 0:00:00.046 45:650 0 days 0:00:00.015 42:4ac 0 days 0:00:00.015 39:990 0 days 0:00:00.015 29:e0c 0 days 0:00:00.015 28:ff8 0 days 0:00:00.015 18:1304 0 days 0:00:00.015 17:f68 0 days 0:00:00.015 7:664 0 days 0:00:00.015 6:660 0 days 0:00:00.015 4:518 0 days 0:00:00.015 3:4a4 0 days 0:00:00.015 47:9c8 0 days 0:00:00.000 46:f20 0 days 0:00:00.000 44:1440 0 days 0:00:00.000 43:11d0 0 days 0:00:00.000 38:db4 0 days 0:00:00.000 37:6ac 0 days 0:00:00.000 36:c4c 0 days 0:00:00.000 35:ff4 0 days 0:00:00.000 34:1950 0 days 0:00:00.000 31:1910 0 days 0:00:00.000 27:14b8 0 days 0:00:00.000 26:14b4 0 days 0:00:00.000 25:14ac 0 days 0:00:00.000 24:13d0 0 days 0:00:00.000 23:13c8 0 days 0:00:00.000 22:13b4 0 days 0:00:00.000 21:13b0 0 days 0:00:00.000 20:13a8 0 days 0:00:00.000 19:1328 0 days 0:00:00.000 16:e20 0 days 0:00:00.000 15:e10 0 days 0:00:00.000 14:dd0 0 days 0:00:00.000 13:ce0 0 days 0:00:00.000 12:754 0 days 0:00:00.000 11:718 0 days 0:00:00.000 10:678 0 days 0:00:00.000 9:674 0 days 0:00:00.000 8:668 0 days 0:00:00.000 5:548 0 days 0:00:00.000 2:4a0 0 days 0:00:00.000 1:418 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Kernel Mode Time Thread Time 32:14a0 0 days 0:02:24.878 33:b6c 0 days 0:00:00.171 30:fa8 0 days 0:00:00.109 40:1bd8 0 days 0:00:00.093 43:11d0 0 days 0:00:00.078 42:4ac 0 days 0:00:00.062 34:1950 0 days 0:00:00.062 7:664 0 days 0:00:00.062 41:174c 0 days 0:00:00.046 28:ff8 0 days 0:00:00.031 18:1304 0 days 0:00:00.031 16:e20 0 days 0:00:00.031 1:418 0 days 0:00:00.031 45:650 0 days 0:00:00.015 17:f68 0 days 0:00:00.015 6:660 0 days 0:00:00.015 4:518 0 days 0:00:00.015 2:4a0 0 days 0:00:00.015 47:9c8 0 days 0:00:00.000 46:f20 0 days 0:00:00.000 44:1440 0 days 0:00:00.000 39:990 0 days 0:00:00.000 38:db4 0 days 0:00:00.000 37:6ac 0 days 0:00:00.000 36:c4c 0 days 0:00:00.000 35:ff4 0 days 0:00:00.000 31:1910 0 days 0:00:00.000 29:e0c 0 days 0:00:00.000 27:14b8 0 days 0:00:00.000 26:14b4 0 days 0:00:00.000 25:14ac 0 days 0:00:00.000 24:13d0 0 days 0:00:00.000 23:13c8 0 days 0:00:00.000 22:13b4 0 days 0:00:00.000 21:13b0 0 days 0:00:00.000 20:13a8 0 days 0:00:00.000 19:1328 0 days 0:00:00.000 15:e10 0 days 0:00:00.000 14:dd0 0 days 0:00:00.000 13:ce0 0 days 0:00:00.000 12:754 0 days 0:00:00.000 11:718 0 days 0:00:00.000 10:678 0 days 0:00:00.000 9:674 0 days 0:00:00.000 8:668 0 days 0:00:00.000 5:548 0 days 0:00:00.000 3:4a4 0 days 0:00:00.000 0:410 0 days 0:00:00.000 Elapsed Time Thread Time 0:410 0 days 0:35:17.550 1:418 0 days 0:35:17.534 3:4a4 0 days 0:35:17.331 2:4a0 0 days 0:35:17.331 4:518 0 days 0:35:16.957 5:548 0 days 0:35:16.817 6:660 0 days 0:35:15.176 7:664 0 days 0:35:15.156 8:668 0 days 0:35:15.126 9:674 0 days 0:35:15.026 10:678 0 days 0:35:14.986 11:718 0 days 0:35:13.066 12:754 0 days 0:35:13.056 13:ce0 0 days 0:35:08.131 14:dd0 0 days 0:35:07.641 15:e10 0 days 0:35:07.551 16:e20 0 days 0:35:07.531 17:f68 0 days 0:35:06.611 18:1304 0 days 0:35:03.553 19:1328 0 days 0:35:03.381 20:13a8 0 days 0:35:02.024 22:13b4 0 days 0:35:01.931 21:13b0 0 days 0:35:01.931 24:13d0 0 days 0:35:01.899 23:13c8 0 days 0:35:01.899 27:14b8 0 days 0:35:00.932 26:14b4 0 days 0:35:00.932 25:14ac 0 days 0:35:00.932 28:ff8 0 days 0:33:04.785 29:e0c 0 days 0:33:04.644 30:fa8 0 days 0:32:59.750 31:1910 0 days 0:32:55.608 32:14a0 0 days 0:32:12.276 34:1950 0 days 0:18:39.607 33:b6c 0 days 0:18:39.607 35:ff4 0 days 0:17:43.530 36:c4c 0 days 0:08:45.458 38:db4 0 days 0:07:41.551 37:6ac 0 days 0:07:41.551 39:990 0 days 0:06:54.877 40:1bd8 0 days 0:06:54.867 41:174c 0 days 0:05:38.282 42:4ac 0 days 0:03:54.627 43:11d0 0 days 0:03:53.122 44:1440 0 days 0:03:51.627 45:650 0 days 0:02:15.536 47:9c8 0 days 0:00:11.100 46:f20 0 days 0:00:11.100

0:000> ~32k # Child-SP RetAddr Call Site 00 00000000`0414f558 000007fe`fd4f1430 ntdll!NtWaitForMultipleObjects+0xa 01 00000000`0414f560 00000000`775706e0 KERNELBASE!WaitForMultipleObjectsEx+0xe8 *** ERROR: Symbol file could not be found. Defaulted to export symbols for wuaueng.dll - 02 00000000`0414f660 000007fe`ee7de250 kernel32!WaitForMultipleObjects+0xb0 03 00000000`0414f6f0 00000000`775759ed wuaueng!DllInstall+0x4e80 04 00000000`0414f770 00000000`776ac541 kernel32!BaseThreadInitThunk+0xd 05 00000000`0414f7a0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

Process heap doesn’t have any large allocated virtual block with only almost 100 KB committed normal blocks:

0:000> !heap -s

**** NT HEAP STATS BELOW ****************************** LFH Key Termination on corruption Heap Flags (k) (k) (k) (k) length ------------------------- 0000000000340000 External fragmentation 0000000000010000 00008000 00000000005c0000 00001002 External fragmentation 0000000000960000 00001002 512 0000000001520000 00001002 00000000013f0000 00001002 512 0000000001790000 00001002 00000000010b0000 00001002 512 00000000019b0000 00001002 0000000001220000 00001002 00000000021e0000 00001002 0000000001900000 00001002 00000000024b0000 00001002 0000000002e00000 00001002 External fragmentation 0000000002f10000 00001002 External fragmentation 00000000035c0000 00001002 0000000003200000 00001002 00000000038c0000 00001002 512 0000000003ad0000 00001002 512 0000000003d40000 00001002 512 00000000005b0000 00001002 0000000004280000 00011002 00000000040c0000 00001002 0000000004270000 00001002 0000000007050000 00001002 512 0000000007cc0000 00001002 512 0000000000be0000 00001002 Virtual block: 0000000009b90000 0000000006540000 00001002 External fragmentation 0000000000b40000 00001002 0000000004950000 00001002 ———R ************************** : 0x000000a57ddeb5ed : ENABLED Reserv Commit Virt Free List UCR Virt Lock Fast blocks cont. heap ------------------------------------------------------------ 00000002 210688 94444 210688 85646 752 86 0 1 LFH 90 % (752 free blocks) 64 4 64 1 1 1 0 0 3136 1256 3136 282 39 3 0 0 LFH 22 % (39 free blocks) 8 512 2 1 1 0 0 512 280 512 3 10 1 0 0 LFH 8 512 2 1 1 0 0 512 248 512 0 6 1 0 0 LFH 8 512 2 1 1 0 0 512 320 512 6 12 1 0 0 LFH 512 380 512 63 22 1 0 0 LFH 512 256 512 11 3 1 0 0 LFH 512 156 512 133 1 1 0 0 1536 792 1536 27 4 2 0 0 LFH 48256 16656 48256 9870 101 13 0 1 LFH 59 % (101 free blocks) 3584 3280 3584 1805 73 3 0 5 LFH 55 % (73 free blocks) 64 8 64 3 1 1 0 0 512 388 512 2 3 1 0 0 LFH 8 512 3 1 1 0 0 8 512 3 1 1 0 0 8 512 3 1 1 0 0 64 8 64 3 1 1 0 0 512 64 512 59 2 1 0 0 1088 296 1088 14 6 2 0 0 LFH 1088 304 1088 22 6 2 0 0 LFH 8 512 3 1 1 0 0 8 512 3 1 1 0 0 64 24 64 17 2 1 0 0 - 0000000009b90000 (size 0000000000000000) 1536 1064 1536 775 11 2 1 0 LFH 72 % (11 free blocks) 64 8 64 3 1 1 0 0 512 312 512 55 12 1 0 0 LFH 12;————————————————————————-

So we consider the dump set as an instance of Memory Fluctuation.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns, x64 Windows | No Comments »

Crash Dump Analysis Patterns (Part 230)

Wednesday, October 14th, 2015

Sometimes, a stack trace from Stack Trace Collection may look well-formed at the first sight like having an expected start frames, for example:

0:000> ~*k

[...]

# 19 Id: 16a4.21f4 Suspend: 0 Teb: 7e95b000 Unfrozen ChildEBP RetAddr 0c2de6b0 74eb112f ntdll!NtWaitForMultipleObjects+0xc 0c2de83c 76ca7b89 KERNELBASE!WaitForMultipleObjectsEx+0xcc 0c2de858 76d007bf kernel32!WaitForMultipleObjects+0x19 0c2dec98 76d00295 kernel32!WerpReportFaultInternal+0x50b 0c2deca8 76ce1709 kernel32!WerpReportFault+0x74 0c2decb0 74f5f705 kernel32!BasepReportFault+0x19 0c2ded3c 76fb4f84 KERNELBASE!UnhandledExceptionFilter+0x1f4 0c2ded54 76fb5728 ntdll!TppExceptionFilter+0x30 0c2ded64 76f5c95a ntdll!TppWorkerpInnerExceptionFilter+0xe 0c2df914 76ca7c04 ntdll!TppWorkerThread+0×87f5a 0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0×24 0c2df970 76f1acea ntdll!__RtlUserThreadStart+0×2f 0c2df980 00000000 ntdll!_RtlUserThreadStart+0×1b

[...]

So, we may think something wrong happened in ntdll!TppWorkerThread code (although 0×87f5a offset looks suspicious). However, in reality, in this case due to exception filter logic (or some other reason in different cases) we have Hidden Stack Trace. When looking at UnhandledExceptionFilter parameters (or raw stack as in the case of Hidden Exceptions) we find an exception context:

0:019> kv ChildEBP RetAddr Args to Child 0c2de6b0 74eb112f 00000003 0c2de880 00000001 ntdll!NtWaitForMultipleObjects+0xc 0c2de83c 76ca7b89 00000003 0c2de880 00000000 KERNELBASE!WaitForMultipleObjectsEx+0xcc 0c2de858 76d007bf 00000003 0c2de880 00000000 kernel32!WaitForMultipleObjects+0x19 0c2dec98 76d00295 00000000 00000001 00000000 kernel32!WerpReportFaultInternal+0x50b 0c2deca8 76ce1709 0c2ded3c 74f5f705 0c2ded94 kernel32!WerpReportFault+0x74 0c2decb0 74f5f705 0c2ded94 00000001 a79b7895 kernel32!BasepReportFault+0x19 0c2ded3c 76fb4f84 0c2ded94 0c2ded94 00000000 KERNELBASE!UnhandledExceptionFilter+0×1f4 0c2ded54 76fb5728 00000000 00000000 0c2df914 ntdll!TppExceptionFilter+0×30 0c2ded64 76f5c95a 0c2df8d0 76f00a70 0c2df914 ntdll!TppWorkerpInnerExceptionFilter+0xe 0c2df914 76ca7c04 0f79e380 76ca7be0 a5b45024 ntdll!TppWorkerThread+0×87f5a 0c2df928 76f1ad1f 0f79e380 a59141d7 00000000 kernel32!BaseThreadInitThunk+0×24 0c2df970 76f1acea ffffffff 76f00233 00000000 ntdll!__RtlUserThreadStart+0×2f 0c2df980 00000000 76ed4a00 0f79e380 00000000 ntdll!_RtlUserThreadStart+0×1b

0:019> dd 0c2ded94 L2 0c2ded94 0c2deef8 0c2def48

0:019> .cxr 0c2def48 eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10 eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202 ntdll!RtlEnterCriticalSection+0×12: 76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

0:019> k *** Stack trace for last set context - .thread/.cxr resets it ChildEBP RetAddr 0c2df3ac 7407999c ntdll!RtlEnterCriticalSection+0x12 0c2df3cc 7407acd6 ModuleA!DoWork+0x1b [...] 0c2df73c 76ee3aa7 ModuleA!ThreadPoolWorkCallback+0xa9 0c2df77c 76ee1291 ntdll!TppWorkpExecuteCallback+0x137 0c2df914 76ca7c04 ntdll!TppWorkerThread+0x48e 0c2df928 76f1ad1f kernel32!BaseThreadInitThunk+0x24 0c2df970 76f1acea ntdll!__RtlUserThreadStart+0x2f 0c2df980 00000000 ntdll!_RtlUserThreadStart+0x1b

We consider this a different pattern than Hidden Call because an entire stack (sub)trace is missing between UnhandledExceptionFilter and thread start frames:

ntdll!NtWaitForMultipleObjects KERNELBASE!WaitForMultipleObjectsEx kernel32!WaitForMultipleObjects kernel32!WerpReportFaultInternal kernel32!WerpReportFault kernel32!BasepReportFault KERNELBASE!UnhandledExceptionFilter [...] ntdll!TppWorkerThread kernel32!BaseThreadInitThunk ntdll!__RtlUserThreadStart ntdll!_RtlUserThreadStart

This pattern is also different from Past Stack Trace pattern because Hidden Stack Trace belongs to PRESENT time zone. Our example is also different from Hidden Exception analysis pattern and its recovered stack trace because exception processing is not hidden and shows Exception Stack Trace albeit with a hidden part.

We were also fortunate to have Stored Exception (accessible by !analyze -v command):

0:019> .exr -1 ExceptionAddress: 76f00fb2 (ntdll!RtlEnterCriticalSection+0x00000012) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000001 Parameter[1]: 15f237e5 Attempt to write to address 15f237e5

0:019> .ecxr eax=15f237e5 ebx=15f235e9 ecx=15f237e1 edx=7e95b000 esi=15f237e1 edi=09724b10 eip=76f00fb2 esp=0c2df3ac ebp=0c2df3ac iopl=0 nv up ei pl nz na po nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010202 ntdll!RtlEnterCriticalSection+0x12: 76f00fb2 f00fba3000 lock btr dword ptr [eax],0 ds:002b:15f237e5=????????

So this Hidden Stack Trace is detected straightforwardly. But in other cases, such as when we have Multiple Exceptions in a process dump or Stack Trace Collection from a complete memory dump, we need to pay attention to such a possibility.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Workaround Patterns (Part 5)

Saturday, September 26th, 2015

We resume our workaround patterns for common reusable solutions to common software execution problems. In the past we proposed a general pattern of Axed Code for removing problem software behavior. The Shadow File blog post about fixing free() crashes by introducing heap metadata header with a data length set to zero led me to generalize and introduce a complementary Axed Data pattern. Such a pattern suggests cutting the data size specified in metadata memory plane (which may be separate from the data plane). In some cases, it may avoid buffer overwrites including Local and Shared in addition to Dynamic Memory Corruption (process heap and kernel pool). The following picture illustrates this general pattern approach.

Sometimes, it may even be possible to provide a workaround by cutting real file data, for example, by changing its file or database record size. But this is also done conceptually by changing file or database system metadata.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Debugging, Workaround Patterns | No Comments »

Crash Dump Analysis Patterns (Part 229)

Sunday, September 13th, 2015

The advent of virtual machines, the possibility of saving complete memory snapshots without interruption, and the ability to quickly convert such snapshots into a debugger readable memory dump format such as in the case of VMware allows to study how Stack Trace Collections and Wait Chains change over time in complex problem scenarios. Such Stack Trace Surface may also show service restarts if PID changes for processes of interest. We call this pattern by analogy with a memory dump surface where each line corresponds to an individual memory snapshot with coordinates from 0 to the highest address:

In case of orbifold memory space we have a case of a 3D volume (we may call 3D orbifold).

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Mathematics of Debugging, Memory Visualization, Software Trace Analysis | No Comments »

Trace Analysis Patterns (Part 113)

Saturday, September 12th, 2015

Recently we analyzed a few logs which ended with a specialized Activity Region from a subsystem that sets operational parameters. The problem description stated that the system became unresponsive after changing parameters in a certain sequence. Usually, for that system, when we stop logging (even after setting parameters) we end up with messages from some Background Components since some time passes between the end of setting parameters activity and the time the operator sends stop logging request:

However, in the problem case we see message flow stops right in the middle of parameter setting activity:

So we advised to check for any crashes or hangs, and, indeed, it was found that the system was actually experiencing system crashes, and we got memory dumps for analysis where we found Top Module from a 3rd-party vendor related to parameter setting activity.

Please also note an analogy here between normal thread stack traces from threads that are waiting most of the time and Spiking Thread stack trace caught up in the middle of some function.

We call this pattern Ruptured Trace after a ruptured computation.

Note, that if it is possible to restart the system and resume the same tracing we may get an instance of Blackout analysis pattern.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Core Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Log Analysis, Software Trace Analysis, Software Trace Analysis Tips and Tricks, Trace Analysis Patterns | No Comments »

Crash Dump Analysis Patterns (Part 35b)

Saturday, September 12th, 2015

Sometimes we notice the anomalies in object distribution in heaps and pools. Memory consumption may be high in case of big objects. Such anomalies may point to possible memory, handle, and object leaks. But it may also be a temporary condition (Memory Fluctuation) due to the large amount of queued or postponed work that can be solved by proper software configuration. Diagnosed anomalies may also direction troubleshooting efforts if they cluster around certain component(s) or specific functionality. The distribution can be assessed by both the total memory consumption and the total number of objects of a particular class.

Here’s an example of Object Distribution Anomaly analysis pattern from .NET heap. The output of !DumpHeap -stat WinDbg SOS extension command shows the abnormal distribution of objects related to SQL data queries:

Count TotalSize […] 2342 281040 System.Reflection.RuntimeParameterInfo 3868 309440 System.Data.Metadata.Edm.TypeUsage 13218 317232 System.Object 3484 390208 System.Reflection.RuntimeMethodInfo 6092 508044 System.Int32[] 2756 617344 System.Data.SqlClient._SqlMetaData 2770 822870 System.Char[] 24560 1375360 System.RuntimeType 18 4195296 System.Int64[] 449691 10792584 System.Data.SqlClient.SqlGen.SqlBuilder 449961 10799064 System.Int64 449691 14390112 System.Data.Query.InternalTrees.ComparisonOp 449695 14390240 System.Data.Query.InternalTrees.ConditionalOp 6360 15509435 System.Byte[] 449690 17987600 System.Data.Query.InternalTrees.ConstantOp 449938 17997520 System.Data.Query.InternalTrees.VarRefOp 450898 21643104 System.Data.Common.CommandTrees.DbPropertyExpression […]

The anomalous character of the distribution is also illustrated in the following diagrams:

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 27e)

Monday, August 24th, 2015

This is another variant of Stack Trace Collection pattern that shows stack traces from threads currently execution on all CPUs. Although we can see the non-idle running threads from the stack traces corresponding to all processes and their threads we may also want to see idle thread stack traces too. Also, the corresponding WinDbg command (!running -t -i) is faster if we want to double check the output of !analyze -v command in case of BSOD. The latter command may show the stack trace from the current CPU instead of the stack trace from the thread running on a different CPU that caused a bugcheck. Here’s an example from one of the memory dumps for which !analyze -v command shows an incorrect stack trace in the output when we open the dump file. It reports the stack trace from CPU 0 but the bugcheck happened on CPU 1:

0: kd> !running -t -i

System Processors: (00000000000000ff) Idle Processors: (00000000000000fd)

Prcbs Current (pri) Next (pri) Idle 0 fffff801e5d85180 fffff801e5ddea00 ( 0) fffff801e5ddea00 ................

Child-SP RetAddr Call Site fffff801`e8c9eb60 fffff801`e5c69b74 hal!KeQueryPerformanceCounter+0x75 fffff801`e8c9eba0 fffff801`e5c69e01 nt!KiCheckStall+0x2c fffff801`e8c9ebd0 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 fffff801`e8c9ece0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b fffff801`e8c9ed30 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 fffff801`e8c9ee70 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 fffff801`e8c8c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf fffff801`e8c8c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa fffff801`e8c8c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 fffff801`e8c8cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 fffff801`e8c8cc60 00000000`00000000 nt!KiIdleLoop+0x2c

1 ffffd000f0975180 ffffe0000d726880 (12) ffffd000f09813c0 …………….

Child-SP RetAddr Call Site ffffd000`202cb618 fffff801`e5a1cc3c hal!HalpAcpiPmRegisterReadPort+0x1b ffffd000`202cb620 fffff801`e5a417e7 hal!HalpAcpiPmRegisterRead+0x30 ffffd000`202cb650 fffff801`e5c66af5 hal!HaliHaltSystem+0x53 ffffd000`202cb690 fffff801`e5c66741 nt!KiBugCheckDebugBreak+0×99 ffffd000`202cb6f0 fffff801`e5bd2aa4 nt!KeBugCheck2+0xc6d ffffd000`202cbe00 fffff801`e5bde4e9 nt!KeBugCheckEx+0×104 ffffd000`202cbe40 fffff801`e5bdcd3a nt!KiBugCheckDispatch+0×69 ffffd000`202cbf80 fffff800`913601da nt!KiPageFault+0×23a ffffd000`202cc118 fffff800`91363710 DriverA!memcpy+0×21a […]

2 ffffd000f09ee180 ffffd000f09fa3c0 ( 0) ffffd000f09fa3c0 ................

Child-SP RetAddr Call Site ffffd000`f09f9f88 fffff801`e5c69e01 nt!KiCheckStall+0xa ffffd000`f09f9f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`f09fa0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`f09fa0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`f09fa230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb5938e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb5938f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb593920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb593b10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb593c60 00000000`00000000 nt!KiIdleLoop+0x2c

3 ffffd000eb5e5180 ffffd000eb5f13c0 ( 0) ffffd000eb5f13c0 ................

Child-SP RetAddr Call Site ffffd000`eb5f0f60 fffff801`e5c69e01 nt!KiCheckStall+0x5f ffffd000`eb5f0f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb5f10a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb5f10f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb5f1230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb5fa8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb5fa8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb5fa920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb5fab10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb5fac60 00000000`00000000 nt!KiIdleLoop+0x2c

4 ffffd000f08d1180 ffffd000f08dd3c0 ( 0) ffffd000f08dd3c0 ................

Child-SP RetAddr Call Site ffffd000`f08dcf90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x227 ffffd000`f08dd0a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`f08dd0f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`f08dd230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb85b8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb85b8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb85b920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb85bb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb85bc60 00000000`00000000 nt!KiIdleLoop+0x2c

5 ffffd000eb8ad180 ffffd000eb8b93c0 ( 0) ffffd000eb8b93c0 ................

Child-SP RetAddr Call Site ffffd000`eb8b8f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb8b8f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb8b90a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb8b90f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb8b9230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb8db8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb8db8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb8db920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb8dbb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb8dbc60 00000000`00000000 nt!KiIdleLoop+0x2c

6 ffffd000eb92a180 ffffd000eb9363c0 ( 0) ffffd000eb9363c0 ................

Child-SP RetAddr Call Site ffffd000`eb935f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb935f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb9360a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb9360f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb936230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb93f8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb93f8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb93f920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb93fb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb93fc60 00000000`00000000 nt!KiIdleLoop+0x2c

7 ffffd000eb967180 ffffd000eb9733c0 ( 0) ffffd000eb9733c0 ................

Child-SP RetAddr Call Site ffffd000`eb972f60 fffff801`e5c69e01 nt!KiCheckStall+0x75 ffffd000`eb972f90 fffff801`e5c6aa8f nt!KiFreezeTargetExecution+0x231 ffffd000`eb9730a0 fffff801`e5bdbec2 nt!KiProcessNMI+0x3b ffffd000`eb9730f0 fffff801`e5bdbd36 nt!KxNmiInterrupt+0x82 ffffd000`eb973230 fffff801`e5a2d82f nt!KiNmiInterrupt+0x176 ffffd000`eb97c8e8 fffff801`e5bb91a2 hal!HalProcessorIdle+0xf ffffd000`eb97c8f0 fffff801`e5ad7848 nt!PpmIdleDefaultExecute+0xa ffffd000`eb97c920 fffff801`e5ad72a6 nt!PpmIdleExecuteTransition+0x3e8 ffffd000`eb97cb10 fffff801`e5bd64bc nt!PoIdle+0x2f6 ffffd000`eb97cc60 00000000`00000000 nt!KiIdleLoop+0x2c

This command is obviously faster than repeatedly switching to subsequent CPUs using ~s command and then checking the corresponding stack trace (k). It also helps in diagnosing Spiking Threads in kernel and complete memory dumps.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Complete Memory Dump Analysis, Crash Dump Analysis, Crash Dump Patterns, Kernel Memory Dump Analysis, x64 Windows | 1 Comment »

Crash Dump Analysis Patterns (Part 228)

Sunday, June 28th, 2015

Predicate Stack Trace Collections allow you to get a subset of stack traces, for example, by showing only stack traces where a specific module is used (for example, !stacks 2 module WinDbg command). From diagnostic analysis perspective, the order in which threads from the subset appear is also important, especially when the output is sorted by thread creation time or simply the order is given by a global thread linked list. We call this analysis pattern Thread Poset by analogy with a mathematical concept of poset (partially ordered set):

Such an analysis pattern is mostly useful when we compare stack traces for differences or when we don’t have symbols for some problem version and want to map threads to some other previous normal run where symbol files are available. Any discrepancies may point in the direction of further diagnostic analysis. For example, we got this fragment of Stack Trace Collection:

4.000188 fffffa800d3d3b50 ffd0780f Blocked ModuleA+0x1ac1 4.00018c fffffa800d3f9950 ffd07b53 Blocked ModuleA+0xd802 4.000190 fffffa800d4161b0 fffffda6 Blocked ModuleA+0x9ce4 4.000194 fffffa800d418b50 fffffda6 Blocked ModuleA+0x9ce4 4.000198 fffffa800d418660 fffffda6 Blocked ModuleA+0x9ce4 4.0001ac fffffa800d41eb50 ffd078d2 Blocked ModuleA+0xa7cf 4.0001b0 fffffa800d41e660 ffd0780f Blocked ModuleA+0x9ce4 4.0001c0 fffffa800d48f300 ffd0e5c0 Blocked ModuleA+0x7ee5

We didn’t have symbols, and, therefore, didn’t know whether there was anything wrong with those threads. Fortunately, we had Thread Poset from an earlier 32-bit version with available symbol files:

4.0000ec 85d8dc58 000068c Blocked ModuleA!FuncA+0x9b 4.0000f0 85d9fc78 001375a Blocked ModuleA!FuncB+0x67 4.0000fc 85db8a58 000068c Blocked ModuleA!WorkerThread+0xa2 4.000104 85cdbd48 000ff44 Blocked ModuleA!WorkerThread+0xa2 4.000108 85da2788 000ff47 Blocked ModuleA!WorkerThread+0xa2 4.000110 857862e0 0013758 Blocked ModuleA!FuncC+0xe4 4.000114 85dda250 000ff44 Blocked ModuleA!FuncD+0xf2

If we map worker threads to the middle section of x64 version we see just one more worker thread but the overall order is the same:

4.000188 fffffa800d3d3b50 ffd0780f Blocked ModuleA+0x1ac1 4.00018c fffffa800d3f9950 ffd07b53 Blocked ModuleA+0xd802 4.000190 fffffa800d4161b0 fffffda6 Blocked ModuleA+0×9ce4 4.000194 fffffa800d418b50 fffffda6 Blocked ModuleA+0×9ce4 4.000198 fffffa800d418660 fffffda6 Blocked ModuleA+0×9ce4 4.0001ac fffffa800d41eb50 ffd078d2 Blocked ModuleA+0xa7cf 4.0001b0 fffffa800d41e660 ffd0780f Blocked ModuleA+0×9ce4 4.0001c0 fffffa800d48f300 ffd0e5c0 Blocked ModuleA+0×7ee5

So we may think of x64 Thread Poset as normal if x86 Thread Poset is normal too. Of course, only initially, then to continue looking for other patterns of abnormal behavior. If necessary, we may need to inspect stack traces deeper, because individual threads from two Thread Posets may differ in their stack trace depth, subtraces, and in usage of other components. Despite the same order, some threads may actually be abnormal.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 227)

Sunday, May 17th, 2015

Managed code Nested Exceptions give us process virtual space bound stack traces. However, exception objects may be marshaled across processes and even computers. The remote stack trace return addresses don’t have the same validity in different process contexts. Fortunately, there is _remoteStackTraceString field in exception objects that contains the original stack trace. Default analysis command sometimes uses it:

0:013> !analyze -v

[...]

EXCEPTION_OBJECT: !pe 25203b0 Exception object: 00000000025203b0 Exception type: System.Reflection.TargetInvocationException Message: Exception has been thrown by the target of an invocation. InnerException: System.Management.Instrumentation.WmiProviderInstallationException, Use !PrintException 0000000002522cf0 to see more. StackTrace (generated): SP IP Function 000000001D39E720 0000000000000001 Component!Proxy.Start()+0x20 000000001D39E720 000007FEF503D0B6 mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0x286 000000001D39E880 000007FEF503CE1A mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+0xa 000000001D39E8B0 000007FEF503CDD8 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x58 000000001D39E900 000007FEF4FB0302 mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x52

[...]

MANAGED_STACK_COMMAND: ** Check field _remoteStackTraceString **;!do 2522cf0;!do 2521900

[...]

0:013> !DumpObj 2522cf0 [...] 000007fef51b77f0 4000054 2c System.String 0 instance 2521900 _remoteStackTraceString […]

0:013> !DumpObj 2521900 Name: System.String […] String: at System.Management.Instrumentation.InstrumentationManager.RegisterType(Type managementType) at Component.Provider..ctor() at Component.Start()

Checking this field may also be necessary for exceptions of interest from managed space Execution Residue. We call this pattern Distributed Exception. The basic idea is illustrated in the following diagram using the borrowed UML notation (not limited to just two computers):

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in .NET Debugging, Crash Dump Analysis, Crash Dump Patterns | No Comments »

Crash Dump Analysis Patterns (Part 226)

Wednesday, May 13th, 2015

Occasionally, we look at Stack Trace Collection and notice Internal Stack Trace. This is a stack trace that is shouldn’t be seen in a normal crash dump because statistically it is rare (we planned to name this pattern Rare Stack Trace initially). This stack trace is also not Special Stack Trace because it is not associated with the special system events or problems. It is also not a stack trace that belongs to various Wait Chains or Spiking Threads. This is also a real stack trace and not a reconstructed or hypothetical stack trace such as Rough Stack Trace or Past Stack Trace. This is simply a thread stack trace that shows some internal operation, for example, where it suggests that message hooking was involved:

THREAD fffffa8123702b00 Cid 11cc.0448 Teb: 000007fffffda000 Win32Thread: fffff900c1e6ec20 WAIT: (WrUserRequest) UserMode Non-Alertable fffffa81230cf4e0 SynchronizationEvent Not impersonating DeviceMap fffff8a0058745e0 Owning Process fffffa81237a8b30 Image: ProcessA.exe Attached Process N/A Image: N/A Wait Start TickCount 1258266 Ticks: 18 (0:00:00:00.280) Context Switch Count 13752 IdealProcessor: 1 NoStackSwap LargeStack UserTime 00:00:00.468 KernelTime 00:00:00.187 Win32 Start Address ProcessA!ThreadProc (0×000007feff17c608) Stack Init fffff8800878c700 Current fffff8800878ba10 Base fffff8800878d000 Limit fffff88008781000 Call fffff8800878c750 Priority 12 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5 Child-SP RetAddr Call Site fffff880`0878ba50 fffff800`01a6c8f2 nt!KiSwapContext+0×7a fffff880`0878bb90 fffff800`01a7dc9f nt!KiCommitThreadWait+0×1d2 fffff880`0878bc20 fffff960`0010dbd7 nt!KeWaitForSingleObject+0×19f fffff880`0878bcc0 fffff960`0010dc71 win32k!xxxRealSleepThread+0×257 fffff880`0878bd60 fffff960`000c4bf7 win32k!xxxSleepThread+0×59 fffff880`0878bd90 fffff960`000d07a5 win32k!xxxInterSendMsgEx+0×112a fffff880`0878bea0 fffff960`00151bf8 win32k!xxxCallHook2+0×62d fffff880`0878c010 fffff960`000d2454 win32k!xxxCallMouseHook+0×40 fffff880`0878c050 fffff960`0010bf23 win32k!xxxScanSysQueue+0×1828 fffff880`0878c390 fffff960`00118fae win32k!xxxRealInternalGetMessage+0×453 fffff880`0878c470 fffff800`01a76113 win32k!NtUserRealInternalGetMessage+0×7e fffff880`0878c500 00000000`771b913a nt!KiSystemServiceCopyEnd+0×13 (TrapFrame @ fffff880`0878c570) 00000000`053ff258 000007fe`fac910f4 USER32!NtUserRealInternalGetMessage+0xa 00000000`053ff260 000007fe`fac911fa DUser!CoreSC::xwProcessNL+0×173 00000000`053ff2d0 00000000`771b9181 DUser!MphProcessMessage+0xbd 00000000`053ff330 00000000`774111f5 USER32!_ClientGetMessageMPH+0×3d 00000000`053ff3c0 00000000`771b908a ntdll!KiUserCallbackDispatcherContinue (TrapFrame @ 00000000`053ff288) 00000000`053ff438 00000000`771b9055 USER32!NtUserPeekMessage+0xa 00000000`053ff440 000007fe`ebae03fa USER32!PeekMessageW+0×105 00000000`053ff490 000007fe`ebae4925 ProcessA+0×5a […] 00000000`053ff820 00000000`773ec541 kernel32!BaseThreadInitThunk+0xd 00000000`053ff850 00000000`00000000 ntdll!RtlUserThreadStart+0×1d

We see that this thread was neither waiting for significant time nor consuming CPU. It was reported that ProcessA.exe was very slow responding. So perhaps this was slowly punctuated thread execution with periodic small waits. In fact, Execution Residue analysis revealed Non-Coincidental Symbolic Information of the 3rd-party Message Hook and its Module Product Process was identified. Its removal resolved the problem.

- Dmitry Vostokov @ DumpAnalysis.org + TraceAnalysis.org -

Posted in Crash Dump Analysis, Crash Dump Patterns | No Comments »

May 2026
M	T	W	T	F	S	S
« Mar
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Archive for the ‘Crash Dump Analysis’ Category

Pages

Recent Comments

Categories

Archives

ARM64

Automated Analysis

Blogroll

Debugging Channels

Forensics

Hardware

Linux

Mac OS X

Magazines and Newspapers

Malware Analysis

Medical Diagnostics

Narratology

Related Links

Reversing

Scripting Languages

Source Code

Tracing Tools

Meta